How to intercept transparently stdin/out/err

Question

I would like to catch all the inputs and output of a commandline program (namely, GDB, but currently changed to ls or cat for simplicity) and redirect it into a file, for later analysis.

I couldn't get anything close to working, but I can't understand what's wrong. Here is my last attempt:

#!/usr/bin/env python2

import subprocess
import sys
import select
import os

def get_pipe():
  fd_r, fd_w = os.pipe()
  return os.fdopen(fd_r, "r"), os.fdopen(fd_w, "w")

out_r, out_w = get_pipe()
err_r, err_w = get_pipe()
in_r, in_w = get_pipe()

proc = subprocess.Popen(["ls"] + sys.argv[1:], stdin=in_r, stdout=out_w, stderr=err_w)

out_w.close()
err_w.close()
in_r.close()

running = True
while running:
  input_lst, output_lst, x_lst = select.select([sys.stdin],[out_r, err_r],  [], 0.5)

  if out_r in output_lst+input_lst:
    data = out_r.readlines()
    print "*",data,"*"
  if err_r in output_lst+input_lst:
    data = err_r.readlines()
    print "+",data,"+"
  if sys.stdin in input_lst:
    data = sys.stdin.readline()
    print "-", data, "-"
    in_w.write(data)

  # don't try to quit if we received some data
  if len(input_lst+output_lst+x_lst) != 0:
    continue
  retcode = proc.poll()
  if retcode is not None:
    running = False

out_r.close()
err_r.close()
in_w.close
exit(retcode)

I tried several other options, like - writing a file wrapper, which was supposed to write to an external file everything written into stdin / read from stdout-err - named pipes - ...

but the best I obtained was the very first lines of "ls".

Moreover, GDB relies on readline for CLI edition, and I feel like it won't be that easy to catch transparently!

Futrell

Answer 1

So after quite a while of research, I found a solution to this problem:

with non blocking reads and writes, we just have to wait for the input file to run out of data (and thrown an exception), and then operate on the same for the output (stdout and stderr):

#!/usr/bin/python2

import sys, os
import subprocess
import fcntl

dump = open("/tmp/dump", "w")
dump.write("### starting %s ###" % " ".join(sys.argv))

proc = subprocess.Popen(["<real app>"] + sys.argv[1:], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

def nonblocking(fd):
  fl = fcntl.fcntl(fd, fcntl.F_GETFL)
  fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

nonblocking(proc.stdin)
nonblocking(proc.stdout)
nonblocking(proc.stderr)

nonblocking(sys.__stdin__)
nonblocking(sys.__stdout__)
nonblocking(sys.__stderr__)

def me_to_proc():
  x_to_y(sys.__stdin__, proc.stdin, "~in> ")

def proc_to_me():
  x_to_y(proc.stdout, sys.__stdout__, "<out~ ")

def proc_to_me_err():
  x_to_y(proc.stderr, sys.__stderr__, "<err~ ")

def x_to_y(x, y, prefix=""):
  try:
    while True:
       line = x.readline()
       to_dump = "%s%s" % (prefix, line)
       print >> dump, to_dump
       print to_dump
       y.write(line)
       y.flush()
       dump.flush()
  except:
    pass

recode = None
while recode is None:
  proc_to_me()
  #proc_to_me_err()
  me_to_proc()

  retcode = proc.poll()

exit(retcode)

just replace your original binary with this script, and change <real app> to create the actual process. In- and out-put information will be written on screen and dumpted into /tmp/dump.

(I'm not sure however about the termination criteria, I didn't check that in details)

Futrell

any similar solution for c++ (but without forking)? #26480832 – Constrict

10 years later but still: Many thanks, you saved my day! – Marimaria