findstr or grep that autodetects chararacter encoding (UTF-16)

Question

I want to do this:

 findstr /s /c:some-symbol *

or the grep equivalent

 grep -R some-symbol *

but I need the utility to autodetect files encoded in UTF-16 (and friends) and search them appropriately. My files even have the byte-ordering mark FFEE in them so I'm not even looking for heroic autodetection.

Any suggestions?

---

I'm referring to Windows Vista and XP.

Obligate

Are some of your files in UTF-16 and some in ASCII, or what? – Sallust

Yep, some ASCII, some UTF-16. – Obligate

Answer 1

A workaround is to convert your UTF-16 to ASCII or ANSI

TYPE UTF-16.txt > ASCII.txt

Then you can use FINDSTR.

FINDSTR object ASCII.txt

Hubblebubble

...pardon, what? – Eagre

or type UTF-16.txt | findstr if you don't need filename (OP needs filename because searching multiple files, but some may find this useful) – Mangan

Answer 2

Thanks for the suggestions. I was referring to Windows Vista and XP.

I also discovered this workaround, using free Sysinternals strings.exe:

C:\> strings -s -b dir_tree_to_search | grep regexp 

Strings.exe extracts all of the strings it finds (from binaries, but works fine with text files too) and prepends each result with a filename and colon, so take that into account in the regexp (or use cut or another step in the pipeline). The -s makes it do a recursive extraction and -b just suppresses the banner message.

Ultimately I'm still kind of surprised that the flagship searching utilities Gnu grep and findstr don't handle Unicode character encodings natively.

Obligate

On their home unix environments, UTF-16 is much less common, and files are generally in UTF-8, which they handle just fine. – Douce

Maybe not so great for extracting the whole line, but perfect for trying to find all files containing a string (which I'm trying to do). Thanks. – Fen

Answer 3

On Windows, you can also use find.exe.

find /i /n "YourSearchString" *.*

The only problem is this prints file names followed by matches. You may filter them by piping to findstr

find /i /n "YourSearchString" *.* | findstr /i "YourSearchString"

Woosley

Unfortunately find command doesn't support matching patterns like findstr (wildcards / regular expressions). – Adela

Answer 4

findstr /s /c:some-symbol *

can be replaced with the following character encoding aware command:

for /r %f in (*) do @find /i /n "some-symbol" "%f"

Hegel

If add Venkateshwar's answer below, you get: for /r %f in (*) do @find /i /n "some-symbol" "%f" | findstr /i "some-symbol" which will filter out the filenames. I found this useful when searching a set of files looking for "Fail". I didn't care what file it appeared in, I just wanted to see if any file had "Fail" in it. – Dalpe

Answer 5

According to this blog article by Damon Cortesi grep doesn't work with UTF-16 files, as you found out. However, it presents this work-around:

for f in `find . -type f | xargs -I {} file {} | grep UTF-16 | cut -f1 -d\:`
        do iconv -f UTF-16 -t UTF-8 $f | grep -iH --label=$f ${GREP_FOR}
done

This is obviously for Unix, not sure what the equivalent on Windows would be. The author of that article also provides a shell-script to do the above that you can find on github here.

This only greps files that are UTF-16. You'd also grep your ASCII files the normal way.

Klansman

Answer 6

In higher versions of Windows, UTF-16 is supported out-of-box. If not, try changing active code page by chcp command.

In my case when using findstr alone was failing for UTF-16 files, however it worked with type:

type *.* | findstr /s /c:some-symbol

Erectile

Answer 7

You didn't say which platform you want to do this on.

On Windows, you could use PowerGREP, which automatically detects Unicode files that start with a byte order mark. (There's also an option to auto-detect files without a BOM. The auto-detection is very reliable for UTF-8, but limited for UTF-16.)

Dawdle