Binary "tail" a file

Question

I would guess most people on this site are familiar with tail, if not - it provides a "follow" mode that as text is appended to the file tail will dump those characters out to the terminal.

What I am looking for (and possibly to write myself if necessary) is a version of tail that works on binary files. Basically I have a wireless link that I would like to trickle a file across as it comes down from another network link. Looking over the tail source code it wouldn't be too hard to rewrite, but I would rather not reinvent the wheel! This wouldn't strictly be "tail" as I would like the entire file to be copied, but it would watch as new bytes were added and stream those.

Ideas?

Indicia

Answer 1

There is also the bintail application which appears to be more robust than the aforementioned script.

The bintail package contains a single application, bintail. The program reads a normal file from disk, and pipes the output to stdout, byte-by-byte, with no translation, similar to what tail(1) does to text files. This is useful for "tailing" binary files, such as WAV files, while they are being written in realtime. This app is a work in progress, but it already does what it was designed to do for me.

Urbanna

Thank you, it is exactly what I needed, to redirect output from "tcpflow" to a nodejs stream :) It didn't work with "tail -f". – Melodymeloid

In linux binutils tail -c +1 -f somefile works just as well. – Swifter

Answer 2

Pipe it to hexdump:

tail -f somefile | hexdump -C

Quartas

I wasn't 100% sure myself but I tried it and it works just fine. – Quartas

Wouldn't tail -f only output new data when it saw a newline in the binary file? I doubt it unbuffers its stdout. – Timothy

Good point Chris, I didn't think of that. So I just tested it now on Debian and yes, it still works if there are no newlines in the stream although that behaviour might be different on different platforms. – Quartas

The use of hexdump is a red herring, isn't it? Or perhaps just an illustration of somewhere to send the binary data. I don't see anything in the question asking for a hexdump, that's all... – Venom

Note hexdump only logs to the screen in increments of 16 bytes. – Shontashoo

Answer 3

Linux coreutils tail(1) works just fine on binary files. For most applications, you just need to avoid its line-orientation, so that the output doesn't begin in some random spot in the middle of a data structure. You can do that by simply starting at the beginning of file, which is also exactly what you asked for:

tail -c +1 -f somefile

works just fine.

Swifter

Answer 4

This hastily coded Python script for Windows may be of assistance:

# bintail.py -- reads a binary file, writes initial contents to stdout,
# and writes new data to stdout as it is appended to the file.

import time
import sys
import os
import msvcrt
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)

# Time to sleep between file polling (seconds)
sleep_int = 1

def main():
    # File is the first argument given to the script (bintail.py file)
    binfile = sys.argv[1]

    # Get the initial size of file
    fsize = os.stat(binfile).st_size

    # Read entire binary file
    h_file = open(binfile, 'rb')
    h_bytes = h_file.read(128)
    while h_bytes:
        sys.stdout.write(h_bytes)
        h_bytes = h_file.read(128)
    h_file.close()


    # Loop forever, checking for new content and writing new content to stdout
    while 1:
        current_fsize = os.stat(binfile).st_size
        if current_fsize > fsize:
            h_file = open(binfile, 'rb')
            h_file.seek(fsize)
            h_bytes = h_file.read(128)
            while h_bytes:
                sys.stdout.write(h_bytes)
                h_bytes = h_file.read(128)
            h_file.close()
            fsize = current_fsize
        time.sleep(sleep_int)

if __name__ == '__main__':
    if len(sys.argv) == 2:
        main()
    else:
        sys.stdout.write("No file specified.")

Urbanna

Answer 5

less somefile

Then press shift F

Anaesthesiology

I don't quit see how I could use less to redirect to a file output and press Shift+F... – Indicia

Answer 6

Strictly speaking, you need to write a program to do this, as tail is not specified to work on binary files. There are also buffering issues you probably want to avoid if you want to receive the new "trickled" data as soon as possible.

Quirinus

Well, looking again I saw that you tagged your question gnu-coreutils. So if you know you'll be using the gnu implementation of tail, it's probably binary safe and probably does not have problematic buffering (check and see). – Quirinus

Answer 7

This isn't tail -- this is progressively copying a file. Look at rsync.

Pediatrics

I wonder that this answer is the accepted one where there are two answers which match much more the question: https://mcmap.net/q/655370/-binary-quot-tail-quot-a-file https://mcmap.net/q/655370/-binary-quot-tail-quot-a-file. rsync is not helpful here because it cannot stream data. it is limited to (relatively static) files on the disk – Heraldic

@Daniel Alder. rsync can be rerun, sending only the new data. – Pediatrics

streaming means you are inside a cgi script run or piping to netcat etc. But that was more a question at @Indicia – Heraldic

Answer 8

I use this as it works on live streams too:

cat ./some_file_or_dev | hexdump -C

sample dumping my key presses (and releases):

[user@localhost input]$ sudo cat /dev/input/event2 | hexdump -C
00000000  81 32 b1 5a 00 00 00 00  e2 13 02 00 00 00 00 00  |.2.Z............|
00000010  04 00 04 00 36 00 00 00  81 32 b1 5a 00 00 00 00  |....6....2.Z....|
00000020  e2 13 02 00 00 00 00 00  01 00 36 00 01 00 00 00  |..........6.....|
00000030  81 32 b1 5a 00 00 00 00  e2 13 02 00 00 00 00 00  |.2.Z............|
00000040  00 00 00 00 00 00 00 00  81 32 b1 5a 00 00 00 00  |.........2.Z....|
00000050  a3 af 02 00 00 00 00 00  04 00 04 00 36 00 00 00  |............6...|
00000060  81 32 b1 5a 00 00 00 00  a3 af 02 00 00 00 00 00  |.2.Z............|
^C

• http://man7.org/linux/man-pages/man1/cat.1.html

• http://man7.org/linux/man-pages/man1/hexdump.1.html

Ameliorate

The problem with hexdump is that it waits for multiples of 16 bytes to print each line, but as long as you're aware of that issue, it's ok. – Gaiter

That's probably because of -C switch (ascii column) – Ameliorate

Answer 9

I use this command (1 means the number of bytes to interpret) : tail -f | od -x1

Messroom