Windows Batch: How to keep empty lines with loop for /f

Question

I'm searching how to keep empty lines when I browse a file with a for loop.

for /f "tokens=1* delims=[" %%i in ('type "test1.txt" ^| find /v /n ""') do (
SET tmp=%%i
echo !tmp! >> test2.txt
)

Actually it works for everybody, but as far as I'm concerned it does not work. For instance if test1.txt content is:

Hello I come from France
I live in Paris

I'm sorry I don't know English, could we speak French please?
If it doesn't bother you

Thank you

Result in test2.txt will be:

[1 
[2 
[3 
[4 
[5 
[6 
[7 

If I put off the "1" near the star "*", result is:

[1]Hello I come from France 
[2]I live in Paris 
[3] 
[4]I'm sorry I don't know English, could we speak French please? 
[5]If it doesn't bother you 
[6] 
[7]Thank you 

Desired output is:

Hello I come from France
I live in Paris

I'm sorry I don't know English, could we speak French please?
If it doesn't bother you

Thank you

Can you please help me to solve this trouble?

Danaedanaher

Answer 1

This could be done as

@echo off
    setlocal enableextensions disabledelayedexpansion

    for /f "tokens=1,* delims=]" %%a in ('
        find /n /v "" ^< "file1.txt"
    ') do (
        >> "file2.txt" echo(%%b
    )

The output from the inner find command is like

[123]texttexttext

The code uses the closing bracket as delimiter, so the tokens (we are requesting two tokens: 1,* or 1*) are

[123 texttexttext
^    ^
1    2
%%a  %%b

But, as repeated delimiters are handled as only one delimiter, if one line begins with a closing bracket it will be removed. This can be prevented as

@echo off
    setlocal enableextensions disabledelayedexpansion

    for /f "tokens=1,* delims=0123456789" %%a in ('
        find /n /v "" ^< "file1.txt"
    ') do (
        set "line=%%b"
        setlocal enabledelayedexpansion
        >>"file2.txt" echo(!line:~1!
        endlocal
    )

Here the numbers are used as delimiters and the line is tokenized as

[   ]texttexttext
^   ^
%%a %%b

Then the value of the second token is stored inside a variable, with delayed expansion disabled to avoid problems with exclamations inside the data (that will be handled/replaced by the parser if delayed expansion is active)

Once the data is inside the variable, delayed expansion is activated (something needed as we want to retrieve the contents from a variable changed inside a block of code) to output the line from the second position (first character in string is 0) to remove the closing bracket. Once done, delayed expansion is disabled again.

edited as the OP has to incorporate it to a larger/complex script, this code should face the most usual problems

@echo off
    rem For this test we will have delayed expansion from the start
    setlocal enableextensions enabledelayedexpansion

    rem External code block that will make delayed expansion necessary
    if 1==1 ( 
        rem Variables changed inside block
        set "input_file=file1.txt"
        set "output_file=file2.txt"

        rem Grab a reference to the content of the file variables
        for %%i in ("!input_file!") do for %%o in ("!output_file!") do (

            rem Prepare the environment for file work
            setlocal disabledelayedexpansion

            rem Prepare output file
            type nul > "%%~fo"

            rem Process input file and write to output file
            for /f "tokens=1,* delims=0123456789" %%a in ('
                find /n /v "" ^< "%%~fi"
            ') do (
                set "line=%%b"
                setlocal enabledelayedexpansion
                >>"%%~fo" echo(!line:~1!
                endlocal
            )

            rem Restore the previous environment
            endlocal
        )
    )

Tristan

Thank you. Your answer works. But there something I didn't expect. I need to do it on a bigger file, and it has special caracters. I noticed that it didn't copy exclamation mark "!" and "^". Maybe I should have precised since the begining that I needed to use it with special caracters. For instance, this line: !(^bytes=[^,]+(,[^,]+){0,4}$|^$) is copied like this : (bytes=[,]+(,[,]+){0,4}$|$) – Danaedanaher

@Cainzer, This will only happen if you have delayed expansion enabled before entering the for /f loop. That is the reason for the setlocal enableextensions disabledelayedexpansion in the posted code. As indicated in the answer delayed expansion needs to be disabled. – Tristan

Ok. Does it delete other variables that I used previously with delayed expansion enabled ? I mean variables which were not in the loop, but before in my code... – Danaedanaher

@Cainzer, no, place a setlocal disabledelayedexpansion before the for command and an endlocal after the for. Only changes made inside the for loop will be discarded. Or you can wrap the code inside the do clause between the setlocal disabledelayedexpansion/endlocal – Tristan

All right, I'll test and valide your answer if it's ok. I did it with a test loop and it seemed ok. Let's do it in real batch now. Thank you very much – Danaedanaher

So yes I'm facing to something new. How to do if "file1.txt" is a variable ? For example : !FILE_NAME! – Danaedanaher

@Cainzer, I've included in the answer a sample showing a wait to deal with it. – Tristan

Your answer is an interesting variant that I have never seen before. I'm used to seeing the loop asfor /f "delims=" ..., and the line number prefix is stripped by using echo(!line:*]=!. I feel like your variant is a bit more complex, but both techniques yield the same result. – Spring

Answer 2

Here is a slightly different variant using the findstr command rather than find and doing the redirection to the output file file2.txt only once rather than per for /F loop iteration:

@echo off
setlocal EnableExtensions DisableDelayedExpansion
>> "file2.txt" (
    for /F "delims=" %%a in ('findstr /N "^" "file1.txt"') do (
        set "line=%%a"
        setlocal EnableDelayedExpansion
        echo(!line:*:=!
        endlocal
    )
)
endlocal

The findstr command precedes every line by a line number and a colon, like this:

1:Hello I come from France

The sub-string substitution portion !line:*:=! replaces everything up to the first colon (due to *) by nothing, thus removing this.

Replace the >> operator by > in case you want to overwrite an already existing file rather than to append to it.

Academia