How do I start and stop a Linux program using the subprocess module in Python?

Question

I’m writing a web app that uses Selenium to screen-scrape another website. This screen-scraping only happens once a day, so I’d rather not leave Selenium and Xvfb running all the time.

I’m trying to figure out how to start Xvfb and Selenium from Python, and then stop them once the screen-scraping’s done.

If I was doing it manually, I’d start them at the command line, and hit CTRL C to stop them. I’m trying to do the same thing from Python.

I seem to be able to successfully start Xvfb like this:

xvfb = Popen('Xvfb :99 -nolisten tcp', shell=True)

But when I’ve tried to terminate it:

xvfb.terminate()

and then tried to start it again (by repeating my initial command), it tells me it’s already running.

Torrefy

Try xvfb.wait() after xvfb.terminate(). Failing that, try xvfb.kill(). – Quota

Ctrl-C sends SIGTERM to a Unix process. Your user python process cannot send SIGTERM to your Xvfb running as root. Get your child's pid, execute "sudo kill <pid>", then "xvfb.wait()". – Vaccaro

@Spike: ah, okay, that makes sense. Maybe I should at least have Xvfb running all the time, as it seems to need a sudoer to start it, and that, I imagine, is going to be a bit awkward from Python? (So far I’ve just been typing my password in at the Python shell, which obviously won’t work when this is normally running.) – Torrefy

Answer 1

I don't know why you want to run Xvfb as root. Your usual X server only needs to run as root (on many but not all unices) only so that it can access the video hardware; that's not an issue for Xvfb by definition.

tempdir = tempfile.mkdtemp()
xvfb = subprocess.Popen(['Xvfb', ':99', '-nolisten', 'tcp', '-fbdir', tempdir])

When you terminate the X server, you may see a zombie process. This is in fact not a process (it's dead), just an entry in the process table that goes away when the parent process either reads the child's exit status or itself dies. Zombies are mostly harmless, but it's cleaner to call wait to read the exit status.

xvfb.terminate()
# At this point, `ps -C Xvfb` may still show a running process
# (because signal delivery is asynchronous) or a zombie.
xvfb.wait()
# Now the child is dead and reaped (assuming it didn't catch SIGTERM).

Fulviah

@Gilles: ah, gotcha — I’d copied and pasted the command I use to start Xvfb from a web page somewhere. I didn’t realise it was the fbdir option causing it to not work if I didn’t run it as root. I’ve removed that now and can run it as my usual user, but .terminate() and .kill() still don’t seem to stop Xvfb. – Torrefy

@Paul: I made a couple of typos in the Popen call but I think you got it right. Are you sure .terminate() and .kill() aren't terminating the process? A zombie may remain (it would disappear when your Python script terminates), see my edit. – Habile

@Gilles: Yeah, it looks like the process is hanging around. After .terminate(), .kill(), and after I exit the Python shell, ps -C Xvfb still reports that Xvfb is running. When I attempt to start Xvfb again, it also says the server is already active. (I think that’s just because the file /tmp/.X99-lock still exists, but ps -C Xvfb confirms that the process is still running.) – Torrefy

@Gilles: AHA — but when I use Popen the way you suggested (i.e. with arguments, rather than a string and shell=True), then everything works just as you said. Phew! Many, many thanks for looking into this, I’m missing a fair bit of basic Unix/Linux knowledge, and so tend to get a bit stumped when trying to debug problems. May a thousand upvotes be granted to your answers. – Torrefy

@Paul: Ah, so you were killing the shell but not Xvfb. If you needed shell expansion before calling Xvfb but not after, you could run Popen("exec Xvfb …", shell=True) so that the Xvfb process would replace the shell instead of running as a child process. If you needed the shell afterwards, you'd need to communicate the process ID of Xvfb to the python code somehow. But here, it's easiest all around to not use a shell. – Habile

@Gilles: right — that makes sense, I see where it was going wrong now. Definitely no need for a shell here. – Torrefy

Answer 2

I assume you can parametrize your system to allow any user to launch Xvfb as explained here solving all your problems

EDIT the correct command line is

sudo chmod u+s `which Xvfb` 

Hebraic

Could be — which part of that allows any user to launch Xvfb though? Is it sudo mod u+s which Xvfb``? I’m not familiar with the mod command, and it’s somewhat immune to Googling — could you expand on it? – Torrefy

in fact it should be chmod u+s as explained here en.wikipedia.org/wiki/Chmod – Hebraic

I doubt that the mod command exist to have help on a command just write man command – Hebraic

@Xavier: ah! Indeed, that works perfectly for making Xvfb runnable by users who aren’t root. Unfortunately, .terminate() and .kill() still don’t seem to stop the process. – Torrefy

@Paul you are right I was missing that in fact it makes a root program launch and identify as root. But your program still runs as root – Hebraic

@Paul you can make your own program run as root even if it's quite dangerous – Hebraic