Python script using subprocess and xclip hangs if piped in bash

Question

I have a python script that needs to output some values to stdin and copies another string to clipboard. I'm using the module subprocess to execute the xclip utility through Popen something like this:

# clip.py
import subprocess
from subprocess import Popen

print('PRINT_ME')
p1 = Popen(['xclip', '-selection', 'clipboard'], stdin=subprocess.PIPE)
p1.communicate(input=('PASTE_ME'.encode()))

The script works as intended: PRINT_ME is echoed in bash and PASTE_ME is available to paste, and returns imediatly.

A problem appears when the script output is piped to another command. Suppose one wants to redirect to a file and stdin using tee:

$ python clip.py|tee file.txt

The program works as supposed but not returns, i.e. shell don't give control back.

How can this could be fixed?

Some important info: the xclip utility forks itself (to cope with implementation of clipboards on X) maintaining the string avaliable to copy, when used in shell it returns immediately (it forks to background). It appears that the shell is attaching both clip.py and xclip stdin/stdout to tee.
If xclip -selection clipboard is found using ps u and killed the command returns.

I'm using Python 3.4.
Thanks

Gwin

Answer 1

It's not due to forking of xclip, it's due to how Python handles Popen.wait() (which is invoked with your call to communicate()) - from Python perspective xclip (silent by default) didn't close its stream so it waits... That's why everything works except for Python moving past your p1.communicate() line when piping its streams to something else (tee in this case) - it waits for all of its file handles to close...

You can either manually open and close your streams, or just configure xclip to filter STDIN to STDOUT and keep Python happy:

import subprocess

p1 = subprocess.Popen(['xclip', '-selection', 'clipboard', '-f'], stdin=subprocess.PIPE)
p1.communicate(input=('PASTE_ME'.encode()))
# etc.

Didn't test it, but it should work. If you don't want for xclip to print out to your current STDOUT just redirect it when instantiating subprocess.Popen() to None (or subprocess.DEVNULL for Python 3.x) or to any other stream handle you want.

Braunstein

Thanks for your answer, both ways worked fine! I've tested various modifications and found interesting that any code after p1.communicate (in the original defective code) are executed, the program halts to exit. And from the two solutions proposed which is more 'pythonic' and side-effects free? – Gwin

You're right, I've expressed myself badly - Popen opens a FH for xclip's STDOUT (and ties it to the current STDOUT) and since xclip never closes the stream it keeps waiting in the end so it can cleanly exit the subprocess. As for Pythonic - it is an illusive term that means different things to different people, but most people tend to agree that KISS approach is the epitome of being Pythonic - thus, unless you need to do something fancy with STDIN/STDOUT, just use Popen.communicate() and save yourself some trouble when xclip already allows you to set it up to play nicely with Python. – Braunstein