git stderr output can't pipe
Asked Answered
P

3

13

I'm writing a graphical URI handler for git:// links with bash and zenity, and I'm using a zenity 'text-info' dialog to show git's clone output while it's running, using FIFO piping. The script is about 90 lines long, so I won't bother posting it here, but here's the most important lines:

git clone "$1" "$target" 2>&1 | cat >> /tmp/githandler-fifo &
cat /tmp/githandler-fifo | zenity --text-info --text='Cloning git repository' &

I'm using FIFO instead of a direct pipe to allow them to run asynchronously and allow for killing git if the zenity window is closed.

Problem is, the only line that appears from git's output is the first:

Initialized empty Git repository in /home/delan/a/.git/

The other lines with counting objects, etc. don't show or are shown on the terminal.

Current reason

The current consensus as to why this isn't working seems to be that cat is non-blocking and quits after the first line, only passing that to zenity and not the rest. My aim is to force blocking for the read, and have zenity's text info dialog show all output progressively.

git outputs progress messages (anything other than the "Initialized" message) on stderr, but the moment I try to pipe stderr to a file or to merge with stdout, the messages disappear.

Fix attempt 1

I've tried to write two blocking versions of cat's functions in C, bread and bwrite, like this:

#include <stdio.h>
main(int argc, char **argv) {
    int c;
    for (;;) {
        freopen(argv[1], "r", stdin);
        while ((c = getchar()) != EOF)
            putchar(c);
    }
}

#include <stdio.h>
main(int argc, char **argv) {
    int c;
    for (;;) {
        freopen(argv[1], "w", stdout);
        while ((c = getchar()) != EOF)
            putchar(c), fputs("writing", stderr);
    }
}

They work nicely because they block and don't quit on EOF, but it hasn't quite solved the problem yet. At the moment, using one, the other, or both, works in theory, but in practice, zenity shows nothing at all now.

Fix attempt 2

@mvds suggested that using a regular file, in combination with tail -f rather than cat, may do this. Surprised at such a simple solution (thanks!) I tried it but unfortunately, only the first line showed up in zenity and nothing else.

Fix attempt 3

After doing some strace'ing and inspecting git's source code, I realise that git outputs all its progress information (anything past the "Initialized" message) on stderr, and the fact that this is the first line and my assumption that it's because of cat quitting early on EOF was a coincidence/misguided assumption (git doesn't EOF until the program ends).

The situation seemed to become a lot simpler, as I shouldn't have to change anything from the original code (at the start of the question) and it should work. Mysteriously, however, the stderr output 'vanishes' when redirected - and this is only something that happens in git.

Test case? Try this, and see if you see anything in the file (you won't):

git clone git://anongit.freedesktop.org/xorg/proto/dri2proto 2> hurr

This goes against everything I know about stderr and redirection; I've even written a little C program that outputs on stderr and stdout to prove to myself that redirection just doesn't work for git.

Fix attempt 4

In line with Jakub Narębski's answer, as well as replies to emails I sent to the git mailing list, --progress is the option I need. Note that this option only works after the command, and not before clone.

Success!

Thank you very much for all your help. This is the fixed line:

git clone "$1" "$target" --progress > /tmp/githandler-fifo 2>&1 &

Prat answered 31/10, 2010 at 11:8 Comment(2)
gosh.. you've saved my life.... happend to have the same problemCardwell
Success! Thanks for that. --progress was the missing piece of the puzzle for me.Greenbrier
P
19

I think that at least some of progress reports gets silenced when output is not a terminal (tty). I'm not sure if it applies to your case, but try to pass --progress option to 'git clone' (i.e. use git clone --progress <repository>).

Though I don't know if it is what you wanted to have.

Pacifica answered 31/10, 2010 at 15:13 Comment(6)
I suspect this could be it. It's easy to test, too - just redirect to a normal file instead of a FIFO and fiddle until you know you have the output you want.Deroo
Unfortunately, git outputs "Unknown option: --progress", then a line break, then the syntax/usage summary. Interestingly, this time, it's all output on zenity, and the FIFO works. Also, git outputs fully and just fine to a file; it's probably zenity's non-blocking reads that's the culprit.Prat
@Delan: Ah, so not this. (As for the --progress option, presumably that means you're using too old a version of git - that feature was added in v1.7.1.)Deroo
@Delan: Actually, I'm a bit confused, looking over all your comments. You say here that it works fine going to a file, but elsewhere you say that if you try to redirect stderr it disappears. Which is it?Deroo
Oddly, I'm using 1:1.7.1-1.1 (which shouldn't be too old) but it still won't take --progress.Prat
Actually, it's because I put --progress before clone, which doesn't work. It works with --progress after the command, thanks!Prat
C
6

For one thing, the output redirection is parsed right-to-left, so

git clone "$1" "$target" 2>&1 > /tmp/githandler-fifo &

is not equal to

git clone "$1" "$target" > /tmp/githandler-fifo 2>&1 &

The latter will redirect stderr to stdout, and then stdout (including stderr) to the file. The former will redirect stdout to the file, and then show stderr on stdout.

As for piping to zenity (which I don't know), I think you may be making things overly complicated with the named pipe. Using strace may shed some light on the inner workings of the processes you're firing up. For the inexperienced, named pipes make things worse compared to normal pipes.

Chippewa answered 31/10, 2010 at 12:10 Comment(9)
I'm using a named pipe for the mere purpose of being able to background both git and zenity, which then allows me to check if zenity has quit before git, and kill git if that happens. Thanks for letting me know about the right-to-left redirection parsing. I've tried that, but it doesn't quite fix the problem here. Now, the first line appears in zenity, and the rest don't show anywhere (not even in the terminal).Prat
as an alternative with less complications, you may want to redirect to a file and then pipe tail -f. This avoids typical named pipe issues.Chippewa
Regarding right-to-left redirection execution -- this is just wrong. Redirections are executed in natural order. Redirections 2>&1 >file are handled as follows: after the first redirection, fd 2 points to the same file as fd 1 (i.e., what was stdout, e.g. the terminal). After the second redirection, fd 1 points to the file, and fd 2 still points to the terminal, because the redirection was made before fd 1 changed.Wampum
@mvds, I've tried that (redirecting git to a regular file and piping tail -f to zenity) and it works for the first line, but the further output shows on the terminal. Maybe git is doing some weird terminal-only magic?Prat
@Delan, if that's even the case with the correct order (thanks for clarifying @Roman) you should strace git to find out what's going on.Chippewa
Using a combination of strace and inspecting the source of git, all that's happening is that the progress messages are printed on stderr. The only thing I don't understand is, if I redirect stderr to stdout, so that I can pipe to cat or a file (using 2>&1), the progress messages vanish.Prat
@Delan: Vanishing output is exactly what Jakub was mentioning.Deroo
@Delan: exactly as others have pointed out, git may call isatty() or equivalent to find out if the stdout/stderr is a typewriter, and if not, it will not give progress messages. Such a distinction is very common actually: compare how even a simple ls will give different output to either stdout or a file (columns vs list and colors vs b/w)Chippewa
as for a fix, 3 suggestions: use appropriate options to git to force tty-like output, hack out isatty() from source, or preload a library which makes isatty() always return 1.Chippewa
B
1

Given the experiment with the FIFO called 'a', I think the problem lies in the way zenity processes its input. What happens if you type into zenity from the keyboard? (Suspicion: it behaves as you'd want, reading to EOF.) However, it could be that zenity handles terminal input (tty input) using regular blocking I/O but uses non-blocking I/O for all other device types. Non-blocking I/O is fine for input from files; it is less desirable for input from pipes or FIFOs, etc. If it did use non-blocking I/O, zenity would get the first line of output, and then exit the loop thinking it was done because its second read attempt would indicate that there was nothing else immediately available.

Demonstrating that this is what is happening (or not) will be tricky. I would be looking to 'truss' or 'strace' or other system call monitor to track what zenity is doing.

As to workarounds...if the hypothesis is correct, then you'll need to persuade zenity that it is reading from a terminal and not a FIFO, so you'll probably need to rig up a pseudo-tty (or pty); the first process would write to the master end of the pty and you'd arrange for zenity to read from the slave end of the pty. You might still use the FIFO too - though it makes a long chain of command.

Boxthorn answered 31/10, 2010 at 18:12 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.