Using redirection within the script produces a unicode output. How to emit single-byte ASCII text?

Question

I am using Sandcastle Helpfile Builder to produce a helpfile (.chm). The project is a .shfbproj file, which is XML format, works with msbuild.

I want to automatically update the Footer text that appears in the generated .chm file. I use this snippet:

$newFooter = "<FooterText>MyProduct v1.2.3.4</FooterText>";

get-content  -Encoding ASCII $projFile.FullName | 
    %{$_ -replace '<FooterText>(.+)</FooterText>', $newFooter } > $TmpFile

move-item $TmpFile $projFile.FullName -force

The output directed to the $TmpFile is always a multi-byte string. But I don't want that. How do I set the encoding of the output to ASCII?

Reproduce

Thanks Powershell for defaulting to an obscure output format (UCS-2 which was replaced in 1996!!??). utf8 would have been fine ;) – Decollate

See also: #40099271 – Radiocarbon

It's possible to mix encodings with ">>" too. I would stick with set-content and add-content. – Flowerage

@LimitedAtonement: Yes, that decision was unfortunate, but fortunately the problem was rectified in PowerShell Core (v6+), where BOM-less UTF-8 is now the default. As an aside: what Windows PowerShell by default uses with > / Out-File (but not with Set-Content) is UTF-16LE (which is part of the Unicode standard and won't go away), not UCS-2 (which is indeed obsolete, because it couldn't represent all Unicode characters). – Streetcar

Answer 1

You could change the $OutputEncoding variable before writing to the file. The other option is not to use the > operator, but instead pipe directly to Out-File and use the -Encoding parameter.

Sivia

the > operator is actually an alias to out-file, which defaults to utf. tomasr is correct that you should use out-file specifically to set the encoding. – Phototype

@JamesPogran What about 2> ? I'm not sure how, in PowerShell to run a command like cmd > out 2> err through the pipes to out-file (explicitly specifying an encoding). Also, "which defaults to utf"; what do you mean by utf? utf8? utf16? utf32? I thought the default is UCS-2 anyway, not UTF (UCS Transfer Format); did you see it documented otherwise somewhere? – Decollate

The Out-File -Encoding recommendation is helpful, but $OutputEncoding does not apply here, because it is unrelated to > / >>. Its sole purpose is to define the encoding to use when sending data from PowerShell to an external program. – Streetcar

e.g. git diff HEAD~1 HEAD | Out-File -Encoding ascii patch.txt – Wayside

Your link is dead – Roller

Answer 2

The > redirection operator is a "shortcut" to Out-File. Out-File's default encoding is Unicode but you can change it to ASCII, so pipe to Out-File instead:

Get-Content -Encoding ASCII $projFile.FullName |
    % { $_ -replace '<FooterText>(.+)</FooterText>', $newFooter } |
    Out-File $tmpfile -Encoding ASCII

Allare

Zomfg what a pain in the ass, https://mcmap.net/q/41742/-perlio-in-windows-powershell-and-cmd-exe/124486 So, essentially generate_bat_file.pl > run_all.bat is totally useless because executing a bat file -- even in PowerShell -- will result in an error if that .bat file is encoded in UTF16. – Wabash

Answer 3

| sc filename does the trick (sc being an alias for Set-Content)

for >> filename use | ac filename does the trick (ac being an alias for Add-Content)

Athapaskan

>> can mix encodings in the same file too. – Flowerage

Answer 4

I found I had to use the following:

write-output "First line" | out-file -encoding ascii OutputFileName
write-output "Next line" | out-file -encoding ascii -append OutputFileName
....

Changing the output encoding using:

$OutputEncoding = New-Object -typename System.Text.ASCIIEncoding

did not work

Snatchy

Indeed, $OutputEncoding does not apply here, because it is unrelated to > / >> / Out-File (or any cmdlet, for that matter). Its sole purpose is to define the encoding to use when sending data from PowerShell to an external program. However, that observation would have gained more exposure as a comment on the highly up-voted, but partially incorrect, accepted answer (I've since posted a comment - let's hope the answer gets fixed). You then wouldn't have needed to post an answer of your own (the rest of which duplicates Shay Levy's older answer). – Streetcar

Answer 5

You can set the default encoding of out-file to be ascii:

$PSDefaultParameterValues=@{'out-file:encoding'='ascii'}

Then something like this will result in an ascii file:

echo hi > out

In powershell 6 and 7, the default encoding of out-file was changed to utf8 no bom.

Flowerage

Indeed, but note that in Windows PowerShell this only works in the latest - and final - version, v5.1. Also worth mentioning that (BOM-less) UTF-8 is PowerShell Core's (v6+) default encoding across all cmdlets. – Streetcar

Answer 6

Just a little example using streams, although I realize this wasn't the original question.

C:\temp\ConfirmWrapper.ps1 -Force -Verbose 4>&1 6>&1 | Out-File -Encoding default -FilePath C:\temp\confirmLog.txt -Append

Will output the information(6) and verbose(4) streams to the output(1) stream and redirect all that to the out-file with ANSI(default) encoding.

Selfsown