How to create a hex dump of file containing only the hex characters without spaces in bash?

Question

How do I create an unmodified hex dump of a binary file in Linux using bash? The od and hexdump commands both insert spaces in the dump and this is not ideal.

Is there a way to simply write a long string with all the hex characters, minus spaces or newlines in the output?

Japheth

See also: How to print only the hex values from hexdump without line numbers or ASCII table? – Ayacucho

Answer 1

xxd -p file

Or if you want it all on a single line:

xxd -p file | tr -d '\n'

Execrate

fyi To reverse the process: xxd -r -ps hexascii.txt file (it is ok with or without newlines) – Oppen

Answer 2

Format strings can make hexdump behave exactly as you want it to (no whitespace at all, byte by byte):

hexdump -ve '1/1 "%.2x"'

1/1 means "each format is applied once and takes one byte", and "%.2x" is the actual format string, like in printf. In this case: 2-character hexadecimal number, leading zeros if shorter.

Modicum

You need a -v or it will drop repeated bytes and replace them with an asterisk. – Kreindler

I wonder if hexdump itself is able to append the newline (only) to the end of output.. (obvious appendage of ; echo makes it impossible to use as bash alias) – Exuberant

My alias: alias to_hex="hexdump -ve '1/1 \"%.2x\"' && echo" – Earle

The iteration count and byte count default to one, so 1/1 may be omitted, leaving the hexdump -ve '"%.2x"' – Assorted

@mykhal, it's possible, if you know the number of bytes in output. Say if you use hexdump to output only 13 first bytes: hexdump -n 13 -e '13/1 "%.2x" "\n"' – Assorted

You can reverse this output back into a binary with xdd -p -r < dump > file – Kilbride

Answer 3

It seems to depend on the details of the version of od. On OSX, use this:

od -t x1 -An file |tr -d '\n '

(That's print as type hex bytes, with no address. And whitespace deleted afterwards, of course.)

Pursy

I would also add -v in this context, otherwise it skips repeats with *. – Gd

Answer 4

Perl one-liner:

perl -e 'local $/; print unpack "H*", <>' file

Demp

Verified. Matches "xxd -p file | tr -d '\n'". – Oppen

fyi To reverse the process: perl -e 'local $/; print pack "H*", <>' <hexascii.txt >file – Oppen

The "local $/" is unnecessary. – Oppen

Update to last comment: The "local $/" is unnecessary for "pack". For unpack, you need it but can alternatively put "undef $/". $/ is the line separator (default NL). undefined puts it into slurp-mode. So <> referenced in a string context pulls the whole binary file without parsing it into lines. – Oppen

Alternate form: perl -e 'print unpack "H*", join("", <>)' <file – Oppen

Alternate form 1 for reversal (hexascii to binary): perl -pe '$_=pack "H*", $_' <hexascii.txt >file – Oppen

Alternate form 2 for reversal (hexascii to binary), removes any newlines (xxd -ps -r adds them): perl -pe 's/\n//g; $_=pack "H*", $_' <hexascii.txt >file – Oppen

Answer 5

You can use Python for this purpose:

python -c "print(open('file.bin','rb').read().hex())"

...where file.bin is your filename.

Explaination:

1. Open file.bin in rb (read binary) mode.
2. Read contents (returned as bytes object).
3. Use bytes method .hex(), which returns hex dump without spaces or new lines.
4. Print output.

Murrey

Answer 6

The other answers are preferable, but for a pure Bash solution, I've modified the script in my answer here to be able to output a continuous stream of hex characters representing the contents of a file. (Its normal mode is to emulate hexdump -C.)

Kreindler

Answer 7

This code produces a "pure" hex dump string and it runs faster than the all the other examples given. It has been tested on 1GB files filled with binary zeros, and all linefeeds. It is not data content dependent and reads 1MB records instead of lines.

perl -pe 'BEGIN{$/=\1e6} $_=unpack "H*"'

Dozens of timing tests show that for 1GB files, these other methods below are slower. All tests were run writing output to a file which was then verified by checksum. Three 1GB input files were tested: all bytes, all binary zeros, and all LFs.

hexdump -ve '1/1 "%.2x"'                    #  ~10x  slower
od -v -t x1 -An | tr -d "\n "               #  ~15x  slower
xxd -p | tr -d \\n                          #   ~3x  slower
perl -e 'local \$/; print unpack "H*", <>'  # ~1.5x  slower
- this also slurps the whole file into memory

To reverse the process:

perl -pe 'BEGIN{$/=\1e6} $_=pack "H*",$_'

Morganite

Answer 8

tldr;

$ od -t x1 -A n -v <empty.zip | tr -dc '[:xdigit:]' && echo 
504b0506000000000000000000000000000000000000
$

Explanation:

Use the od tool to print single hexadecimal bytes (-t x1) --- without address offsets (-A n) and without eliding repeated "groups" (-v) --- from empty.zip, which has been redirected to standard input. Pipe that to tr which deletes (-d) the complement (-c) of the hexadecimal character set ('[:xdigit:]'). You can optionally print a trailing newline (echo) as I've done here to separate the output from the next shell prompt.

References:

• https://pubs.opengroup.org/onlinepubs/9699919799/utilities/od.html
• https://pubs.opengroup.org/onlinepubs/9699919799/utilities/tr.html

Sc

Answer 9

I think this is the most widely supported version (requiring only POSIX defined tr and od behavior):

cat "$file" | od -v -t x1 -A n | tr -d ' \n'

This uses od to print each byte as hex without address without skipping repeated bytes and tr to delete all spaces and linefeeds in the output. Note that not even the trailing linefeed is emitted here. (The cat is intentional to allow multicore processing where cat can wait for filesystem while od is still processing previously read part. Single core users may want replace that with < "$file" od ... to save starting one additional process.)

Confederation

No need to use cat here, just pass the filename to od. – Tenderhearted

I agree that historically connecting stdin is considered better and it's still the correct solution for single core CPUs. However, with modern systems with many idle cores in a single socket CPU, I think it's better to allow cat to read pipe buffer worth of input in parallel with the od at all times. This reduces the possibility of od stalling for reading the file. – Confederation

I did some microbenchmarks and it seems that using the cat at the start does allow using multiple CPU cores for the task in parallel but in practice, the intercommunication between the cores causes so much extra work that avoiding the use of cat at the start of pipeline is still faster – at least for locally accessible files. In case the file was on remote network drive with slow connection, using the cat here could improve performance. That said, the od can only handle about 4 MB/s so it will be the bottleneck in most cases. – Confederation