skip the first 32k of stdin with dd?

Question

If I have a file on a file system I can do something like this with dd:

dd  if=/my/filewithaheader.bin bs=32k skip=1 | gunzip | tar tvf

however if I try something like this:

./commandthatputsstuffonstdout |  dd  bs=32k skip=1 | gunzip | tar tvf

I get the error: dd: 'standard input': cannot skip to specified offset.

How can I fix this, can it be done with dd, or is there another unix command I can use

Panpipe

Answer 1

You could use tail. Say:

./commandthatputsstuffonstdout | tail -c +1025 ...

to skip the first 1024 bytes of output produced by your command.

From man tail:

   -c, --bytes=K
          output the last K bytes; alternatively,  use  -c  +K  to  output
          bytes starting with the Kth of each file

Hatchett

Oh what an ugly semantics. For skipping 1024 bytes, giving +1024 would be more logical. But no, they chose otherwise :-( – Argos

Answer 2

I've just run into this too, and using the fullblock iflag prevents the short read and subsequent abort.

Example:

gzip -d < ./disk_image.dd.gz | \
    dd bs=4M skip=32768 iflag=fullblock,skip_bytes of=./partial_image.dd

Pentobarbital

skip_bytes is only necessary if you want to use bytes instead of bs for skip, hence no relation to question afaik. – Ceolaceorl

Answer 3

Bit late answer but this dd example worked for me.

Create example source file:

$ dd if=/tmp/somefile of=/tmp/test skip=50 bs=100 count=1

Skip 50 bytes and copy 10 bytes after it to test_skip file:

$ dd if=/tmp/test of=/tmp/test_skip skip=50 bs=1 count=10
10+0 records in
10+0 records out
10 bytes (10 B) copied, 8.2091e-05 s, 122 kB/s

Or data from stdin:

cat /tmp/test| dd of=/tmp/stdin bs=1 skip=50 count=1

Verify output:

$ hexdump /tmp/test
0000000 ebf3 e8fd df1b 0aa1 faa3 1fba 1817 1267
0000010 1402 f539 fb69 f263 f319 084b 0b26 1150
0000020 182a f98d 030c e0b0 e47c f13d ef3b 1146
0000030 0b7e 0f72 0e58 f2bd f403 ee95 e529 0567
0000040 f88e 1994 0e83 12e5 11e7 fd4b 032f f4f0
0000050 fc9d 010a 0ab6 06b6 1224 f5cb 01e4 e67a
0000060 ebe0 f1a0                              
0000064

$ hexdump /tmp/test_skip
0000000 0f72 0e58 f2bd f403 ee95

Source file offset 50 is byte: 0x0f

Nelda

Please note that the question refers to skipping data coming from a standard input (stdin) not a file; therefore this does not answer the question. – Panpipe

Very true, dd can be used with stdin too. $ cat /tmp/test| dd of=/tmp/stdin bs=1 skip=50 count=10 – Nelda

Yes - but have you tried it with skip? Look at the question. When I tried it like that I got an error dd: 'standard input': cannot skip to specified offset; which is why I asked the question here. To be honest the answer already received here before in 2013 solved my problem, and I accepted the answer. I'm just dropping you a couple of hints to help your interaction on SE in the future, or you might get downvoted answers – Panpipe

Thanks for your tips. I tried and repeated your attempt like this: dd if=/dev/zero of=/tmp/zeros bs=1 count=32k echo "teststrings" >> /tmp/zeros cat /tmp/zeros| dd of=/tmp/test_3 bs=1 count=11 skip=32k cat /tmp/test_3 teststrings[henpel@~]$ – Nelda

dd if=/dev/zero of=/tmp/zeros bs=1 count=32k; echo "teststrings" >> /tmp/zeros; cat /tmp/zeros| dd of=/tmp/test_3 bs=1 count=11 skip=32k; cat /tmp/test_3; Semicolons added for previous comment. – Nelda

Your approach likely works because you have bs=1, and likely won't work with any other block size (e.g: bs=4M)... bs=1 will be horribly inefficient, and will make processing large datasets very slow. See my answer for a more suitable approach. – Pentobarbital