Identifying and removing null characters in UNIX

Question

I have a text file containing unwanted null characters (ASCII NUL, \0). When I try to view it in vi I see ^@ symbols, interleaved in normal text. How can I:

1. Identify which lines in the file contain null characters? I have tried grepping for \0 and \x0, but this did not work.

2. Remove the null characters? Running strings on the file cleaned it up, but I'm just wondering if this is the best way?

Flack

This kind of question probably belongs to SuperUser.com – Peddling

In fact, this question is on superuser.com: superuser.com/questions/75130/how-to-remove-ths-symbol-with-vim – Musical

Answer 1

I’d use tr:

tr < file-with-nulls -d '\000' > file-without-nulls

If you are wondering if input redirection in the middle of the command arguments works, it does. Most shells will recognize and deal with I/O redirection (<, >, …) anywhere in the command line, actually.

Illaffected

and a "diff file-with-nulls file-without-nulls" should show me which lines had null characters? It brings back a lot more than expected. – Flack

Actually, I believe it should be tr -d '\000' < file-with-nulls > file-without-nulls since < is part of the shell pipe functionality and not tr. – Bobettebobina

Most shells will recognize & deal with < or > anywhere in the argument string, actually. Surprised me too. – Lethargy

+1 For usage of input redirection instead of cat |. A fine, clean solution and it solved my problem. – Rezzani

This is an order of magnitude slower than sed for me – Yager

@Yager that's pretty interesting. I wonder what's behind that; buffering? – Illaffected

@Illaffected I have no idea how the internals of either tool work, so I can't hazard a guess – Yager

@Illaffected Is there a reason you use '\000' instead of '\0'? On the surface they seem to have the same effect – Larrikin

@HaroldFischer I don't recall why I wrote it that way 8 years ago I'm afraid :) – Illaffected

@Illaffected '\000' is used in lieu of '\0' in the POSIX opengroup specification for tr. That is a good reason to prefer it – Larrikin

@HaroldFischer well I'm not sure what you're trying to do; if you want to see if file has any nulls in it you could use wc to compare the size of the file pre-filtering and post-filtering. In a Unicode world it's generally a better idea to be prepared for non-ASCII characters than to worry about them. – Illaffected

@Illaffected I do apologize, that question was meant for someone else – Larrikin

I manage to detect nulls using grep -Poa '\000' and using wc. Seems easier and more direct / less error-prone. – Rearmost

@Rearmost I'm glad that works for you, but I'm not sure what's less "error-prone" about it. – Illaffected

I can identify nulls more exactly matched to the content rather than aggregate file size. I guess if we are replacing all nulls then maybe that's less of a problem. Grepping just gives a more focused approach that I was trying to mention. – Rearmost

This is an extra-good answer for the description of the file redirects. That has the potential to make things so much clearer! – Nonoccurrence

How can we automate the process, the moment multiple files arrived, remove its null characters ? – Principate

Answer 2

Use the following sed command for removing the null characters in a file.

sed -i 's/\x0//g' null.txt

this solution edits the file in place, important if the file is still being used. passing -i'ext' creates a backup of the original file with 'ext' suffix added.

Sonora

Note: In FreeBSD (and I believe also Mac OS X), sed -i requires an extension in the next argument, but it may be empty. In those systems, add a '', as in: sed -i '' 's/\x0//g "$FILE". – Henry

This is an order of magnitude faster than tr for me – Yager

For me, using Git for Windows and $ sed --version -> sed (GNU sed) 4.7, I had to use the following invocation to get a backup file called example.csv.bak: sed -i.bak 's/\x0//g' example.csv – Sair

@TimČas you did it great, just missed one ' so it should be sed -i '' 's/\x0//g' some_file.xml – Atrophied

On mac this only did the first null character and not all of them. gsed did work to do all of them. – Sansone

Is sed -i '/\x0/d' null.txt a valid alternative? Maybe more elegant – Newsboy

Answer 3

A large number of unwanted NUL characters, say one every other byte, indicates that the file is encoded in UTF-16 and that you should use iconv to convert it to UTF-8.

Predatory

I ran out of disk space while my application was logging. This resulting in these characters. – Flack

For example, it works using this command: iconv -f UTF-16 -t UTF-8 file. – Sidewalk

Answer 4

I discovered the following, which prints out which lines, if any, have null characters:

perl -ne '/\000/ and print;' file-with-nulls

Also, an octal dump can tell you if there are nulls:

od file-with-nulls | grep ' 000'

Flack

for me only 'od -to1 ..' had the desired behaviout otherwise 2 bytes were put out like '001002'. Seems somehow like default settings or implementation of od have changed over time. – Rheology

Answer 5

If the lines in the file end with \r\n\000 then what works is to delete the \n\000 then replace the \r with \n.

tr -d '\n\000' <infile | tr '\r' '\n' >outfile

Stoltz

PS. If you find yourself in a Windows DOS shell, you can get the GNU/win32 versions of Unix commands from Sourceforge.net. I use them all the time. Check out "od" the octal dump command for analysing what's in a file... – Stoltz

Answer 6

Here is example how to remove NULL characters using ex (in-place):

ex -s +"%s/\%x00//g" -cwq nulls.txt

and for multiple files:

ex -s +'bufdo!%s/\%x00//g' -cxa *.txt

^{For recursivity, you may use globbing option **/*.txt (if it is supported by your shell).}

Useful for scripting since sed and its -i parameter is a non-standard BSD extension.

See also: How to check if the file is a binary file and read all the files which are not?

Bennink

Answer 7

I used:

recode UTF-16..UTF-8 <filename>

to get rid of zeroes in file.

Brandybrandyn

Answer 8

I faced the same error with:

import codecs as cd
f=cd.open(filePath,'r','ISO-8859-1')

I solved the problem by changing the encoding to utf-16

f=cd.open(filePath,'r','utf-16')

Deanadeanda

Answer 9

Remove trailing null character at the end of a PDF file using PHP, . This is independent of OS

This script uses PHP to remove a trailing NULL value at the end of a binary file, solving a crashing issue that was triggered by the NULL value. You can edit this script to remove all NULL characters, but seeing it done once will help you understand how this works.

Backstory
We were receiving PDF's from a 3rd party that we needed to upload to our system using a PDF library. In the files being sent to us, there was a null value that was sometimes being appended to the PDF file. When our system processed these files, files that had the trailing NULL value caused the system to crash.

Originally we were using sed but sed behaves differently on Macs and Linux machines. We needed a platform independent method to extract the trailing null value. Php was the best option. Also, it was a PHP application so it made sense :)

This script performs the following operation:

Take the binary file, convert it to HEX (binary files don't like exploding by new lines or carriage returns), explode the string using carriage return as the delimiter, pop the last member of the array if the value is null, implode the array using carriage return, process the file.

//In this case we are getting the file as a string from another application. 
// We use this line to get a sample bad file.
$fd = file_get_contents($filename);

//We trim leading and tailing whitespace and convert the string into hex
$bin2hex = trim(bin2hex($fd));

//We create an array using carriage return as the delminiter
$bin2hex_ex = explode('0d0a', $bin2hex);

//look at the last element.  if the last element is equal to 00 we pop it off
$end = end($bin2hex_ex);
if($end === '00') {
   array_pop($bin2hex_ex);
}

//we implode the array using carriage return as the glue
$bin2hex = implode('0d0a', $bin2hex_ex);

//the new string no longer has the null character at the EOF
$fd = hex2bin($bin2hex);

Bickering