Bash read/write file descriptors -- seek to start of file

Question

I tried to use the read/write file descriptor in bash so that I could delete the file that the file descriptor referred to afterward, as such:

F=$(mktemp)
exec 3<> "$F"
rm -f "$F"

echo "Hello world" >&3
cat <&3

but the cat command gives no output. I can achieve what I want if I use separate file descriptors for reading and writing:

F=$(mktemp)
exec 3> "$F"
exec 4< "$F"
rm -f "$F"

echo "Hello world" >&3
cat <&4

which prints Hello world.

I suspected that bash doesn't automatically seek to the start of the file descriptor when you switch from writing to reading it, and the following combination of bash and python code confirms this:

fdrw.sh

exec 3<> tmp
rm tmp

echo "Hello world" >&3
exec python fdrw.py

fdrw.py

import os  

f = os.fdopen(3)
print f.tell()
print f.read()

which gives:

$ bash fdrw.sh
12

$ # This is the prompt reappearing

Is there a way to achieve what I want just using bash?

Verrazano

why would you want to delete the file before reading/writing? – Whinstone

In Unix, when you remove a file, the file isn't actually deleted until all open file descriptors to it are closed. Thus, deleting a temporary file right after opening is common practice, since it guarantees that no other process can maliciously alter the file and that the file is closed after your process closes the file or exits. – Verrazano

Why don't you like your own method of having separate read and write descriptors? That seems like the simplest way. – Glogau

Answer 1

No. bash does not have any concept of "seeking" with its redirection. It reads/writes (mostly) from beginning to end in one long stream.

Radford

Basically, then, the only reason for read/write descriptors in bash is to pass them to an exec'ed process? – Verrazano

In order to provide more channels than just stdin, stdout, and sterr, yes. – Radford

It is also commonly used with network connections... exec {w}<>/dev/tcp/www.google.com/80 And you can use it to write to the MIDDLE of a file. You open it read/write, read N chars, then write to it. Search for "Read and Write" in antofthy.gitlab.io/info/shell/file_handles.txt for an example. – Lazes

Answer 2

I found a way to do it in bash, but it's relying on an obscure feature of exec < /dev/stdin which actually can rewind the file descriptor of stdin according to http://linux-ip.net/misc/madlug/shell-tips/tip-1.txt:

F=$(mktemp)
exec 3<> "$F"
rm -f "$F"

echo "Hello world" >&3
{ exec < /dev/stdin; cat; } <&3

The write descriptor isn't affected by that so you can still append output to descriptor 3 before the cat.

Sadly I only got this working under Linux not under MacOS (BSD), even with the newest bash version. So it doesn't seem very portable.

Screening

See my expanded testing below. – Lazes

Answer 3

If you ever do happen to want to seek on bash file descriptors, you can use a subprocess, since it inherits the file descriptors of the parent process. Here is an example C program to do this.

seekfd.c

#define _FILE_OFFSET_BITS 64
#include <string.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char* argv[])
{
    /* Arguments: fd [offset [whence]]
     * where
     * fd: file descriptor to seek
     * offset: number of bytes from position specified in whence
     * whence: one of
     *  SEEK_SET (==0): from start of file
     *  SEEK_CUR (==1): from current position
     *  SEEK_END (==2): from end of file
     */
    int fd;
    long long scan_offset = 0;
    off_t offset = 0;
    int whence = SEEK_SET;
    int errsv; int rv;
    if (argc == 1) {
        fprintf(stderr, "usage: seekfd fd [offset [whence]]\n");
        exit(1);
    }
    if (argc >= 2) {
        if (sscanf(argv[1], "%d", &fd) == EOF) {
            errsv = errno;
            fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
            exit(1);
        }
    }
    if (argc >= 3) {
        rv = sscanf(argv[2], "%lld", &scan_offset);
        if (rv == EOF) {
            errsv = errno;
            fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
            exit(1);
        }
        offset = (off_t) scan_offset;
    }
    if (argc >= 4) {
        if (sscanf(argv[3], "%d", &whence) == EOF) {
            errsv = errno;
            fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
            exit(1);
        }
    }

    if (lseek(fd, offset, whence) == (off_t) -1) {
        errsv = errno;
        fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
        exit(2);
    }

    return 0;
}

Verrazano

I'm certian there would be aperl oneliner that will do the same thing and be more portible – Lazes

@Lazes I'm not sure about the "more portable". It's a lot easier to cross-compile a C program written to the standard without extensions than to cross-compile the Perl interpreter (which, back in the day, was once inside my job description -- at an embedded systems shop where we targeted hardware that wasn't large enough to compile perl natively; porting the Perl interpreter was not a fun one). – Chervil

I suppose it depends on your point of view. Portible to me is not needing to re-compile a program for multiple systems, just using it directly, multiple Linuxes, Macs, Solaris, etc etc etc. But I can see you point of view too.. A link to a perl version is given in my answer (really simple). this version also lets you use 'tell', or 'truncate' and can use names rather than numbers. – Lazes

Answer 4

Try changing the sequence of commands:

F=$(mktemp tmp.XXXXXX)
exec 3<> "$F"
echo "Hello world" > "$F"
rm -f "$F"

#echo "Hello world" >&3
cat <&3

Diapason

@Dennis this solution actually works. The cat isn't reading from the apparently deleted file. It's reading from the descriptor that's still open. You can still access a file's contents with that descriptor, even though the last (hard) link to it has been removed. – Glogau

Problem with this solution is that the echo can take a long time, which means the temporary file stays on the filesystem for a long time. If this is acceptable then you can simply use filename, not fd 3, for redirection. – Freeloader

Answer 5

When you open a file descriptor in bash like that, it becomes accessible as a file in /dev/fd/. On that you can do cat and it'll read from the start, or append (echo "something" >> /dev/fd/3), and it'll add it to the end. At least on my system it behaves this way. (On the other hand, I can't seem to be able to get "cat <&3" to work, even if I don't do any writing to the descriptor).

Bawdry

Answer 6

#!/bin/bash
F=$(mktemp tmp.XXXXXX)
exec 3<> $F
rm $F

echo "Hello world" >&3
cat /dev/fd/3

As suggested in other answer, cat will rewind the file descriptor for you before reading from it since it thinks it's just a regular file.

Piranesi

Answer 7

To 'rewind' the file descriptor, you can simply use /proc/self/fd/3

Test script :

#!/bin/bash

# Fill data
FILE=test
date +%FT%T >$FILE

# Open the file descriptor and delete the file
exec 5<>$FILE
rm -rf $FILE

# Check state of the file
# should return an error as the file has been deleted
file $FILE

# Check that you still can do multiple reads or additions
for i in {0..5}; do
    echo ----- $i -----

    echo . >>/proc/self/fd/5
    cat /proc/self/fd/5

    echo
    sleep 1
done

Try to kill -9 the script while it is running, you will see that contrary to what happens with the trap method, the file is actually deleted.

Grenier

Answer 8

Expansion on the answer by @sanmai...

And confirmation of what is going on...

#/bin/bash
F=$(mktemp tmp.XXXXXX)
exec 3<>$F     # open the temporary file for read and write
rm $F          # delete file, though it remains on file system

echo "Hello world!" >&3    # Add a line to a file
cat /dev/fd/3              # Read the whole file
echo "Bye" >>/dev/fd/3     # Append another line
cat /dev/fd/3              # Read the whole file
echo "Goodbye" >&3         # Overwrite second line
cat /dev/fd/3              # Read the whole file

cat <&3                    # Try to Rewind (no output)
echo "Cruel World!" >&3    # Still adds a line on end
cat /dev/fd/3              # Read the whole file

shell_seek 3 6 0           # seek fd 3 to position 6
echo -n "Earth" >&3        # Overwrite 'World'
shell_seek 3               # rewind fd 3
cat <&3                    # Read the whole file put 3 at end

Note that the echo Goodbye overwrites the second lineas the file descriptor &3 had not changed by the cat!

So I tried using cat <&3 which did not output anything, probably as the file descriptor was at the end of the file. To see it if it rewinds the descriptor it was given. It does not.

The last part is to use the 'C' program that was provided, compiled and named shell_seek and yes it seems it works as the first 'World' was replaced by 'Earth', the rewind (seek to start) worked allowing the last cat to again read the whole file. It would put the fd at the end of the file again!

Doing it using perl instead of C was not that hard either. For example perl -e 'open(FD,">&3"); seek(FD,0,0);' will rewind file descriptor 3 back to the start of the file.

I have now made a perl version of shell_seek so I don't have to re-compile it all the time for different systems. Not only that but the script can also 'tell' you the current file descriptor offset, and also 'truncate' the file that file descriptor points too. Both operations are commonly used when using seek, so it seemed a good idea to include those functions. You can download the script from... https://antofthy.gitlab.io/software/#shell_seek

Lazes

cat /dev/fd/3 doesn't rewind the descriptor in macOS (BSD?), so it's also not portable (I even tested with GNU cat and it still doesn't work). Curiously, it works with tail -r even for <&3, so one could do something like tail -r <&3 | tail -r but it looks ugly and there is no -r option for GNU tail. Alternatively, one could do tail -100000 <&3 but choosing an arbitrary number here also looks ugly (or one has to know the max number of expected lines). tail +0 <&3 doesn't work for some reason. Also, with GNU tail under macOS it doesn't work, so be sure to use /usr/bin/tail. – Screening

Thanks for the update about MacOS. Better to use the shell_seek C program, or expanded perl script provided to more precisely control the file descriptor. – Lazes

Answer 9

You can use a named pipe:

mkfifo pipeA 
exec 3<>pipeA 
rm pipeA

echo "Hello world" >&3
read x <&3
echo $x

echo "Hello world" >&3
cat <&3

Demandant