Convert binary data to hexadecimal in a shell script

Question

I want to convert binary data to hexadecimal, just that, no fancy formatting and all. hexdump seems too clever, and it "overformats" for me. I want to take x bytes from the /dev/random and pass them on as hexadecimal.

Preferably I'd like to use only standard Linux tools, so that I don't need to install it on every machine (there are many).

Addition

I had to use /dev/urandom, /dev/random just froze – Donn

@AquariusPower random blocks when it does not have enough random data, while urandom does not (IIRC loops over what it has) – Addition

Answer 1

Perhaps use xxd:

% xxd -l 16 -p /dev/random
193f6c54814f0576bc27d51ab39081dc

Canakin

xxd is part of vim, so it might not always be installed. – Fennel

Note that you can use -c to change the number of bytes per line. Unfortunately you can only set it to 256 after which you need to have some newlines. – Apatetic

I just wanted to say thank you and that this is the fastest ive ever seen for generating data without doing overthe top work. i literally did a 2gb file in a few seconds I needed 32 byte strings in hex and this did the trick time xxd -c 32 -l 1024000000 -ps /dev/urandom 32bytehexnewtest1.txt ; real 0m17.484s – Chaunce

Answer 2

Watch out!

hexdump and xxd give the results in a different endianness!

$ echo -n $'\x12\x34' | xxd -p
1234
$ echo -n $'\x12\x34' | hexdump -e '"%x"'
3412

Simply explained. Big-endian vs. little-endian :D

Iatrics

Use echo -n $'\x12\x34' | hexdump -e '/1 "%x"' to get the same endianness. – Fabron

Also watch out! hexdump will strip leading zeros. – Cooperative

Answer 3

With od (GNU systems):

$ echo abc | od -A n -v -t x1 | tr -d ' \n'
6162630a

With hexdump (BSD systems):

$ echo abc | hexdump -ve '/1 "%02x"'
6162630a

From Hex dump, od and hexdump:

"Depending on your system type, either or both of these two utilities will be available--BSD systems deprecate od for hexdump, GNU systems the reverse."

Duggan

After 4 years (not much, I know), most times I read a thread about *nix, I learn about something really old and still completely new for me. +1 for the od, never heard about it, very useful and present even on Cygwin. ;-) – Mauromaurois

Note the -v option and the leading zero %0 in the format are crucial for hexdump to work properly. Many other answers on this page are missing those. – Laddie

If you have bash, then you can save the tr command and use a temporary variable: "r=$(echo whatever | od -A n -v -t x1); echo ${r// }" – Rn

Answer 4

Perhaps you could write your own small tool in C, and compile it on-the-fly:

int main (void) {
  unsigned char data[1024];
  size_t numread, i;

  while ((numread = read(0, data, 1024)) > 0) {
    for (i = 0; i < numread; i++) {
      printf("%02x ", data[i]);
    }
  }

  return 0;
}

And then feed it from the standard input:

cat /bin/ls | ./a.out

You can even embed this small C program in a shell script using the heredoc syntax.

Leghorn

C, for this? That's an overkill. – Telesthesia

Well, but you have full control over the formatting and the behaviour :-) – Leghorn

that's always an option, but I was quite sure it's been solved before :) – Addition

@Telesthesia - This seems to be the simplest solution for a C programmer. The best I can tell, all the other solutions do not hex encode a binary file. I am befuddled at how difficult this task has become. Is it really that f**k'ing hard to hex encode a binary file? – Helbonnah

Incredible, but this may well be the best answer to this question. – Sclerous

Answer 5

All the solutions seem to be hard to remember or too complex. I find using printf the shortest one:

$ printf '%x\n' 256
100

But as noted in comments, this is not what author wants, so to be fair, below is the full answer.

... to use above to output actual binary data stream:

printf '%x\n' $(cat /dev/urandom | head -c 5 | od -An -vtu1)

What it does:

• printf '%x\n' .... - prints a sequence of integers , i.e. printf '%x,' 1 2 3, will print 1,2,3,
• $(...) - this is a way to get output of some shell command and process it
• cat /dev/urandom - it outputs random binary data
• head -c 5 - limits binary data to 5 bytes
• od -An -vtu1 - octal dump command, converts binary to decimal

As a testcase ('a' is 61 hex, 'p' is 70 hex, ...):

$ printf '%x\n' $(echo "apple" | head -c 5 | od -An -vtu1)
61
70
70
6c
65

Or to test individual binary bytes, on input let’s give 61 decimal ('=' char) to produce binary data ('\\x%x' format does it). The above command will correctly output 3d (decimal 61):

$printf '%x\n' $(echo -ne "$(printf '\\x%x' 61)" | head -c 5 | od -An -vtu1)
3d

Doff

@МалъСкрылевъ I have updated my answer, you are right, I missed the main point of question - I was probably looking for simplest way of converting decimal to hex in shell, and this answer shows up in google as first. – Doff

hey, I know this is old but for the second code box couldnt you just skip the cat and just do the head directly? printf '%x\n' $(head -c 5 /dev/urandom | od -An -vtu1) or is there a specific reason for the cat command? Also, if you wanted to have the hex only separated by a new line at the end I came up with this echo $(printf "%x" $(head -c 5 /dev/urandom | od -An -vtu1)) although I dont know the performance costs for these and mainly why im asking about the cat command being cut out. I'm pretty new to command line but this information all helps. – Chaunce

@cigolon head is better than cat. I often use time command to measure performance of commands. – Doff

thats the reason why I was asking this. printf '%x\n' $(cat /dev/urandom | head -c 5 | od -An -vtu1) VS printf '%x\n' $(head -c 5 /dev/urandom | od -An -vtu1) the second command without the cat pipe is a little bit faster overall. – Chaunce

Answer 6

If you need a large stream (no newlines) you can use tr and xxd (part of Vim) for byte-by-byte conversion.

head -c1024 /dev/urandom | xxd -p | tr -d $'\n'

Or you can use hexdump (POSIX) for word-by-word conversion.

head -c1024 /dev/urandom | hexdump '-e"%x"'

Note that the difference is endianness.

Apatetic

Use echo abc | hexdump -e '/1 "%02x"' to get network order and a 0 for 0x0a. – Fabron

Answer 7

dd + hexdump will also work:

dd bs=1 count=1 if=/dev/urandom 2>/dev/null  | hexdump -e '"%x"'

Telesthesia

thanks, I started this way but couldn't make hexdump do what I want. I was quite sure that it has the option I need, but couldn't find this in the man page – Addition

With this solution (hexdump -e '"%x"'): '\n' -> 'a' (missing leading '0'), 'abcde' -> '6463626165' (incorrect byte order). This could be very bad in non-random-data applications! – Fennel

Answer 8

Sometimes perl5 works better for portability if you target more than one platform. It comes with every Linux distribution and Unix OS. You can often find it in container images where other tools like xxd or hexdump are not available. Here's how to do the same thing in Perl:

$ head -c8 /dev/urandom | perl -0777 -ne 'print unpack "H*"'
5c9ed169dabf33ab

$ echo -n $'\x01\x23\xff' | perl -0777 -ne 'print unpack "H*"'
0123ff

$ echo abc | perl -0777 -ne 'print unpack "H*"'
6162630a

Note that this uses slurp more, which causes Perl to read the entire input into memory, which may be suboptimal when the input is large.

Gadget

Answer 9

These three commands will print the same (0102030405060708090a0b0c):

n=12
echo "$a" | xxd -l "$n" -p
echo "$a" | od  -N "$n" -An -tx1 | tr -d " \n" ; echo
echo "$a" | hexdump -n "$n" -e '/1 "%02x"'; echo

Given that n=12 and $a is the byte values from 1 to 26:

a="$(printf '%b' "$(printf '\\0%o' {1..26})")"

That could be used to get $n random byte values in each program:

xxd -l "$n" -p                   /dev/urandom
od  -vN "$n" -An -tx1            /dev/urandom | tr -d " \n" ; echo
hexdump -vn "$n" -e '/1 "%02x"'  /dev/urandom ; echo

Fabron