How can I spy on communication between a process and a terminal?

Question

I have a Linux process developed by a third-party that communicates with a terminal. For debugging I want to see the communication going back in forth.

One might think cat would do the trick (to see one direction):

    ./third-party-app &
    cat /dev/tty

...but it does not. Rather, cat will steal half of the data intended for the application, which is pretty much worthless.

third-party-app is hard-coded to assume /dev/tty.

One way I found to spy on the communication is to rename the /dev/tty device to, say, /dev/real_tty and create a named pipe called /dev/tty in its place. Then running:

    cat /dev/real_tty | tee /dev/tty &

...will at least let me see the output of /dev/real_tty, by copying the data from /dev/real_tty to the named pipe /dev/tty and stdout.

This sort of works but it feels really dodgy, and relies on the trickery of replacing the device. It also doesn't work in both directions, because named pipes only carry data in one direction.

What's the right way to do this?

If anyone's wondering, the TTY device is a RS-232 link to a microcontroller. The information is not sensitive or secured. All processes (application and spies) can run as root.

Audition

Answer 1

You could take a look at slsnif. It does exactly what you want, or if you're interested in writing one yourself the source is available to see how it works.

Gertrude

Great suggestion. I'll take a look and see how it goes. – Audition

It does at least half the job without "stealing" data from the stream, although it doesn't show data being sent to the serieal port. I may have not found the right options yet. – Audition

Annoyingly I've had to do this at least once before (it was to log a process going over ssh) but I can't find any notes on how I did it or remember the details. The slightly more convoluted way is to use psuedo ttys. Vague memory says that one pair is opened as /dev/tty (probably using symlinks as they have strange names /dev/pttym and /dev/pttyc) and the other end connects to the real tty device. It's more complex as you'll need to write a C wrapper that selects on the FDs and passes blocks back and forth while saving a copy to disk. – Gertrude

Answer 2

Have you considered using strace/ltrace? You can see the system calls it is making, in particular you can see the write/ioctl etc calls being made.

Inhere

Answer 3

RS-232? Just tap the RxD/TxD/GND lines with clips. It's been a forever since I've seen any device even care about DCD, DTR, etc.

Specialistic

That's cheating! :). Absolutely, that would work, but getting the RS-232 output interleaved with the third party app's stdout debug messages be very helpful for solving timing problems. – Audition

Answer 4

There are some alternatives:

Do It Youself with GDB: Redirecting Output from a Running Process

CryoPID allows you to capture the state of a running process in Linux and save it to a file. This file can then be used to resume the process later on, either after a reboot or even on another machine.

Distributed MultiThreaded CheckPointing is a tool to transparently checkpointing the state of an arbitrary group of programs spread across many machines and connected by sockets.

Shopwindow

The question is not about std output. It is about the app writing to a specific device, in particular a link to a microcontroller. – Inhere

Close, but I don't think that will do it. I am interested in both the input and output. The application might use a single file descriptor for both reading and writing the TTY, and I am then left with the same problem I had before with the named pipes. The TTY is not stdin/stdout. I was also hoping for something a little less invasive than breaking in with gdb.... – Audition

@Moron so what? You don't have to pick 1 as the number of the fd you replace :) – Westland

@hobbs: If it is a device on which the app does some ioctl, then redirecting to a pipe might not work! I hope you get my point now. – Inhere

@Moron: good point -- the app does use ioctl to configure the RS-232 baud rate – Audition

Answer 5

The script program exists to do this using psudo-terminal. The device /dev/tty is usually special and refers to the current process's controlling terminal, so you may not have had to resort to renaming things.

script opens a psudo-terminal and then runs another instance of your shell with that new shell as its controlling terminal (so /dev/tty refers to this psudo-terminal for this shell and its child processes). The -c option lets you run a particular command rather than your shell.

The main problem with script is that it is impossible to tell which way the data captured in the output file (./typescript by default) was going -- data flowing both ways is dumped to the same file and looks similar to what appears on the screen when using an interactive terminal (except for including escapes, carriage returns, and goofy stuff like that as well as the normally displayed characters).

Anyway, I know that this question has long since been answered, but I thought that if anyone were to search for a similar solution and were not using a real serial port this may help them.

Transubstantiate

Answer 6

Not simple (not for me at least), but a mechanism that should work for the tty serial drivers is a line discipline.

Berghoff

Answer 7

People here have already made good suggestions, but here's another:

You can also write a shared library with your own write() that does some work before calling the write() from libc.so. Then you can use the LD_PRELOAD environment variable to load your library when the process starts.

Vedis

s/LDD_PRELOAD/LD_PRELOAD/ Unfortunately, SO won't allow making a one char edit even when it is a syntax error fix so I have to use a comment, sorry. – Serrato

@Serrato - You're right. Edited. Though looking back at this 5 years later I don't think it's a great answer. – Vedis

:) I think it could have been a great answer if it was more complete. Include an actually working code snippet (e.g. how can write avoid calling itself when it calls the libc entry-point with the same name) and compilation line to create the preloaded module. This is very valuable because it is a general solution to many similar problems: "how to slightly modify something in libc that already exists." – Serrato

Answer 8

Use socat:

# socat -v -v /dev/real_tty exec:"./third-party-app",pty,ctty,setsid,echo=0

The -v option should show communication between the TTYs.

Here's a nice example reference:

• SOCAT Examples

Finisterre