How do I get a thread safe print in Python 2.6?

Question

print in Python is not thread safe according to these articles.

A Python 3 work-around is offered in the latter article.

How do I get a thread safe print in Python 2.6?

Rayon

Those articles are about Python 3. Where are the articles stating that print in Python 2.x is not thread-safe? – Daughtry

Ignacio: I've seen it myself :-) Try spinning up a few threads all printing to stdout. The lines will be all messed up. – Rayon

The second article bramcohen.livejournal.com/70686.html seems to just use a Lock, there isn't anything Python 3 specific about it I think. – Brocklin

Answer 1

Interesting problem -- considering all the things that happen within a print statement, including the setting and checking of the softspace attribute, making it "threadsafe" (meaning, actually: a thread that's printing only yields "control of standard output" to another thread when it's printing a newline, so that each entire line that's output is guaranteed to come from a single thread) was a bit of a challenge (the usual easy approach to actual thread safety -- delegating a separate thread to exclusively "own" and handle sys.stdout, communicate to it via Queue.Queue -- isn't all that useful, since the problem is not thread safety [[even with a plain print there is no risk of crashing and the characters that end up on standard output are exactly those which get printed]] but the need for mutual exclusion among threads for an extended range of operations).

So, I think I made it...:

import random
import sys
import thread
import threading
import time

def wait():
  time.sleep(random.random())
  return 'W'

def targ():
  for n in range(8):
    wait()
    print 'Thr', wait(), thread.get_ident(), wait(), 'at', wait(), n

tls = threading.local()

class ThreadSafeFile(object):
  def __init__(self, f):
    self.f = f
    self.lock = threading.RLock()
    self.nesting = 0

  def _getlock(self):
    self.lock.acquire()
    self.nesting += 1

  def _droplock(self):
    nesting = self.nesting
    self.nesting = 0
    for i in range(nesting):
      self.lock.release()

  def __getattr__(self, name):
    if name == 'softspace':
      return tls.softspace
    else:
      raise AttributeError(name)

  def __setattr__(self, name, value):
    if name == 'softspace':
      tls.softspace = value
    else:
      return object.__setattr__(self, name, value)

  def write(self, data):
    self._getlock()
    self.f.write(data)
    if data == '\n':
      self._droplock()

# comment the following statement out to get guaranteed chaos;-)
sys.stdout = ThreadSafeFile(sys.stdout)

thrs = []
for i in range(8):
  thrs.append(threading.Thread(target=targ))
print 'Starting'
for t in thrs:
  t.start()
for t in thrs:
  t.join()
print 'Done'

The calls to wait are intended to guarantee chaotically mixed output in the absence of this mutual exclusion guarantee (whence the comment). With the wrapping, i.e., the above code exactly as it looks there, and (at least) Python 2.5 and up (I believe this may run in earlier versions, too, but I don't have any easily at hand to check) the output is:

Thr W -1340583936 W at W 0
Thr W -1340051456 W at W 0
Thr W -1338986496 W at W 0
Thr W -1341116416 W at W 0
Thr W -1337921536 W at W 0
Thr W -1341648896 W at W 0
Thr W -1338454016 W at W 0
Thr W -1339518976 W at W 0
Thr W -1340583936 W at W 1
Thr W -1340051456 W at W 1
Thr W -1338986496 W at W 1
  ...more of the same...

The "serialization" efect (whereby the threads appear to "nicely round-robin" as above) is a side effect of the fact that the thread that gets to be the currently-printing one is seriously slower than the others (all those waits!-). Commenting out the time.sleep in wait, the output is instead

Thr W -1341648896 W at W 0
Thr W -1341116416 W at W 0
Thr W -1341648896 W at W 1
Thr W -1340583936 W at W 0
Thr W -1340051456 W at W 0
Thr W -1341116416 W at W 1
Thr W -1341116416 W at W 2
Thr W -1338986496 W at W 0
  ...more of the same...

i.e. a more typical "multithreaded output"... except for the guarantee that each line in the output comes entirely from one single thread.

Of course, a thread that does, e.g., print 'ciao', will keep "ownership" of standard output until it finally does perform a print without a trailing comma, and other threads wanting to print may sleep for quite a while (how else can one guarantee that each line in the output comes from a single thread? well, one architecture would be to accumulate partial lines to thread local storage instead of actually writing them to standard output, and only do the writing on receipt of the \n... delicate to interleave properly with softspace settings, I fear, but probably feasible).

Vitovitoria

Answer 2

Through experimentation, I found that the following works, is simple, and suits my needs:

print "your string here\n",

Or, wrapped in a function,

def safe_print(content):
    print "{0}\n".format(content),

My understanding is that the implicit newline of a normal print is actually output to stdout in a separate operation, causing the race condition with other print operations. By removing this implicit newline with the added ,, and instead including the newline in the string, we can avoid this problem.

---

2020 Edit: Here's the Python 3 version of this (thanks to Bob Stein in the comments for the inspiration):

def safe_print(*args, sep=" ", end="", **kwargs):
    joined_string = sep.join([ str(arg) for arg in args ])
    print(joined_string  + "\n", sep=sep, end=end, **kwargs)

As pointed out by Bob Stein, relying on print to join multiple passed arguments results in garbled output, so we have to do it ourselves.

---

2017 Edit: this answer is starting to pick up some steam, so I just wanted to make a clarification. This doesn't actually make print "thread safe" exactly. The output may be in the wrong order if the prints happen microseconds apart from each other. What this does do, however, is avoid garbled output coming from print statements executed from concurrent threads, which is what most people really want when asking this question.

Here is a test to show what I mean:

from concurrent.futures import ThreadPoolExecutor


def normal_print(content):
    print content

def safe_print(content):
    print "{0}\n".format(content),


with ThreadPoolExecutor(max_workers=10) as executor:
    print "Normal Print:"
    for i in range(10):
        executor.submit(normal_print, i)

print "---"

with ThreadPoolExecutor(max_workers=10) as executor:
    print "Safe Print:"
    for i in range(10):
        executor.submit(safe_print, i)

Output:

Normal Print:
0
1
23

4
65

7
 9
8
----
Safe Print:
1
0
3
2
4
5
6
7
8
9

Islander

If you send your python script to the background by putting an & at the end of the command line in bash, and then exit your bash shell, the thread will crash when it hits this specially crafted print statement with "IOError: [Errno 5] Input/output error". You must redirect standard output somewhere before sending to a background job and then logging off. I'm using Python 2.7.12, by the way. – Polonium

By the way, if I do a normal print and send the program to the background and logoff, I get the problem this guy is describing when the python script completes and the interpreter is shutting down (not when the print statement occurs). #12790828 So, there is something different in how the standard output is being processed with the implicit and explicit newline. – Polonium

print("One string with no commas and its own LF\n", end="") is the Python 3 version. For me it seemed those two criteria made print() atomic. But print("Separate", "strings", "joined by commas\n", end="") was vulnerable to being hacked apart. – Pfaff

@BobStein Updated with Python 3 version – Islander

Answer 3

The issue is that python uses seperate opcodes for the NEWLINE printing and the printing of the object itself. The easiest solution is probably to just use an explicit sys.stdout.write with an explicit newline.

Hangchow

From my recent experience, this is absolutely true. I'm not exactly sure why it happens, but the print statement (even when STDOUT is properly serialized and flushed) will output erratic newlines. You must use sys.stdout.write(s + '\n') to avoid this. – Shingly

Just using sys.stdout.write is no guarantee of serialized output in a multithreaded environment. You also need a Lock. – Harville

Answer 4

I dont know if there is any better way instead this locking mechanism, but atleast it looks easy. I am also not sure if printing really isnt thread safe.

Edit: Okay tested it my self now, you are right, you can get really wierd looking output. And you dont need the future import, its just there, because i use Python 2.7.

from __future__ import print_function
from threading import Lock

print_lock = Lock()
def save_print(*args, **kwargs):
  with print_lock:
    print (*args, **kwargs)

save_print("test", "omg", sep='lol')

Irksome

@evilpie: I reckon you are using python3 after all. Eg. in python 2.7 - print ("test1", "test2") is actually executed as print tuple("test1", "test2"). Therefore the *args, **kwargs parameters are not correct and are not the same as print "test1", "test2". – Pastoralist