UNIX format files with Powershell

Question

How do you create a unix file format in Powershell? I am using the following to create a file, but it always creates it in the windows format.

"hello world" | out-file -filepath test.txt -append

As I understand, the new line characters CRLF make it to be a Windows format file whereas the unix format needs only a LF at the end of the line. I tried replacing the CRLF with the following, but it didn't work

"hello world" | %{ $_.Replace("`r`n","`n") } | out-file -filepath test.txt -append

Nook

Your test doesn't work because there is no CRLF in "hello world" – Kirk

@Kirk when you write it out to a file, then the cr/lf is appended to the line – Nook

that's the point I was trying to make. clearly your replace operation is before the write (out-file) so if out-file adds the CRLF, it happens after your attempt to replace it. – Kirk

I know this is not answer to your question, but you can check if it is affordable to call dos2unix on the target. – Born

Answer 1

One ugly-looking answer is (taking input from dos.txt outputting to unix.txt):

[string]::Join( "`n", (gc dos.txt)) | sc unix.txt

but I would really like to be able to make Set-Content do this by itself and this solution does not stream and therefore does not work well on large files...

And this solution will end the file with a DOS line ending as well... so it is not 100%

Disarticulate

Extending your solution, there is in fact a way to write the bytes and avoids the DOS line ending: sc unix.txt ([byte[]][char[]] "$contenttext") -Encoding Byte – Nook

Good thinking, thanx for that! – Disarticulate

@Kirk this also ends the file with a Windows-style line ending (\r\n) – Boettcher

Indeed, this invariably creates a trailing CRLF on Windows. With the benefit of hindsight: in PowerShell v5+, you can use Set-Content -NoNewLine to avoid this problem. – Styria

Answer 2

There is a Cmdlet in the PowerShell Community Extensions called ConvertTo-UnixLineEnding

Colb

Answer 3

Two more examples on how you can replace CRLF by LF:

1. Example:

   (Get-Content -Raw test.txt) -replace "`r`n","`n" | Set-Content test.txt -NoNewline
   

2. Example:

   [IO.File]::WriteAllText('C:\test.txt', ([IO.File]::ReadAllText('C:\test.txt') -replace "`r`n","`n"))
   

Be aware, this does really just replace CRLF by LF. You might need to add a trailing LF if your Windows file does not contain a trailing CRLF.

Solstice

best answer! elegant one-liners & verified both work in Win PowerShell 5.1 on Win11 – Entropy

Answer 4

I've found that solution:

sc unix.txt ([byte[]][char[]] "$contenttext") -Encoding Byte

posted above, fails on encoding convertions in some cases.

So, here is yet another solution (a bit more verbose, but it works directly with bytes):

function ConvertTo-LinuxLineEndings($path) {
    $oldBytes = [io.file]::ReadAllBytes($path)
    if (!$oldBytes.Length) {
        return;
    }
    [byte[]]$newBytes = @()
    [byte[]]::Resize([ref]$newBytes, $oldBytes.Length)
    $newLength = 0
    for ($i = 0; $i -lt $oldBytes.Length - 1; $i++) {
        if (($oldBytes[$i] -eq [byte][char]"`r") -and ($oldBytes[$i + 1] -eq [byte][char]"`n")) {
            continue;
        }
        $newBytes[$newLength++] = $oldBytes[$i]
    }
    $newBytes[$newLength++] = $oldBytes[$oldBytes.Length - 1]
    [byte[]]::Resize([ref]$newBytes, $newLength)
    [io.file]::WriteAllBytes($path, $newBytes)
}

Slat

Answer 5

make your file in the Windows CRLF format. then convert all lines to Unix format in new file:

$streamWriter = New-Object System.IO.StreamWriter("\\wsl.localhost\Ubuntu\home\user1\.bashrc2")
$streamWriter.NewLine = "`n"
gc "\\wsl.localhost\Ubuntu\home\user1\.bashrc" | % {$streamWriter.WriteLine($_)}
$streamWriter.Flush()
$streamWriter.Close()

not a one-liner, but works for all lines, including EOF. new file now shows as Unix format in Notepad on Win11.

delete original file & rename new file to original, if you like:

ri "\\wsl.localhost\Ubuntu\home\user1\.bashrc" -Force
rni "\\wsl.localhost\Ubuntu\home\user1\.bashrc2" "\\wsl.localhost\Ubuntu\home\user1\.bashrc"

Entropy

Nice - good to know that the newline sequence can be controlled via the System.IO.StreamWriter type's .NewLine property. As an inconsequential aside: you don't need to call .Flush() before .Close(). – Styria

Flush was need for me, in Windows PowerShell 5.1 on Win11 – Entropy

As for flushing: at least the .NET (Core) source code shows that flushing is automatically performed on .Close(). If that weren't the case, it would amount to a serious bug/design flaw. Even in W11 / WinPS v5.1 I don't see a problem with not calling .Flush(), so if there is a problem, after all, it would be good to know what specific OS / PowerShell versions are affected. – Styria

Answer 6

The following makes sure that the output file has Unix-style line endings (LF) no matter what line endings the input has:

Get-Content input.txt | % { $_ + "`n"} | Set-Content output.txt -NoNewline

Conversely the following makes that the output file has Windows-style line endings (CR LF) no matter what line endings the input has:

Get-Content input.txt| % { $_ + "`r`n"} | Set-Content output.txt -NoNewline

Bus