Remove ^@ Characters in a Unix File

Question

I have a question about removing invisible characters which can be only be seen when we try to view the file using "vi" command. We have a file that's being generated by Datastage Application (Source is a DB2 Table -> Target is a .txt File). File has data with different data types. I'm having an issue with just 3 columns which have their datatypes defined as CHAR.

If you open the file in a Textpad you'd see spaces. But when you view the same file on Unix via vi command, we see ^@ characters in blue color. My file is a delimiter file with the delimiter as ^@^ (I know it's kinda sounds weird) .

I have tried:

1. tr -d [:cntrl:] <Filename >NewFileName — Still no luck — [Delimiters are removed but the spaces remain]
2. tr -s "^@" <Filename >NewFilename — Still no luck — I see file reduce in file size but the invisible characters still stay.
3. Tried changing the delimiter — but still see the same invisible characters.
4. Used sed "s/^@/g/" (and other sed commands) <Filename — still no luck.

Any suggestions are really appreciated. I have researched the posts on this website but I couldn't find one. If it's a simple please excuse me and share your thoughts.

Deil

Answer 1

In vi, NUL characters are represented as ^@. To get rid of them:

tr

Using tr, you should be able to remove the NUL characters as follows:

tr -d '\000' < file-name > new-file-name

Unintelligent

This is tagged "unix", not "linux" - only the example given for tr is useful with the "unix" tag. – End

@ThomasDickey Care to clarify? And does your statement cover all versions of unix? – Unintelligent

Try reading the POSIX descriptions. In "vi" you are describing "vim", while in "sed", you are describing the GNU implementation. Neither follows POSIX for these cases. You may as well toss in Perl. – End

@ThomasDickey I've made some updates. Am I still far off the mark? Actually, for vi, I found this URL describing a similar solution to get rid of line feeds: tech-recipes.com/rx/150/… – Unintelligent

POSIX sed doesn't know about '\x'. If you are going to use that, you should change the tag to "linux" (no "unix"). The plain (no control modifier) @ can't be POSIX. Most terminals (not all) honor control-@ (^@) for a null. – End

OK, thanks for the feedback. I'll remove the vim and sed sections. – Unintelligent

Thanks Robby.. This worked and my problem solved. I'd really appreciate your time. Thomas your inputs as well, I learned something new today. – Deil

Answer 2

open the file with vim and then type ':' without the single quote and paste this:

%s/control + 2//g (on regular PCs)

%s/control + shift + 2 //g (on Mac PCs)

of course, replace with keys from your keyboard

Refreshing