Thread._wait_for_tstate_lock() never returns

Question

My program appears to run in a deadlock sometimes when I hit Ctrl+C. I'm trying to catch the keyboard interrupt and gracefully stop all running threads, but I'm not quite there yet.

I'm using a concurrent.futures.ThreadPoolExecutor. To find the location of the deadlock, I'm using the this receipe from ActiveState.

Now, here's the full stacktrace:

# ThreadID: 4856
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\threading.py", line 884, in _bootstrap
  self._bootstrap_inner()
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\threading.py", line 916, in _bootstrap_inner
  self.run()
File: "C:\Users\niklas\repos\nodepy\craftr\lib\utils\tracer.py", line 66, in run
  self.stacktraces()
File: "C:\Users\niklas\repos\nodepy\craftr\lib\utils\tracer.py", line 80, in stacktraces
  fout.write(stacktraces())
File: "C:\Users\niklas\repos\nodepy\craftr\lib\utils\tracer.py", line 28, in stacktraces
  for filename, lineno, name, line in traceback.extract_stack(stack):

# ThreadID: 6068
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\runpy.py", line 193, in _run_module_as_main
  "__main__", mod_spec)
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code
  exec(code, run_globals)
File: "C:\Users\niklas\repos\nodepy\craftr\.nodepy_modules\.bin\craftr.exe\__main__.py", line 9, in <module>
  sys.exit(nodepy.main.main())
File: "c:\users\niklas\repos\nodepy\nodepy\nodepy\main.py", line 103, in main
  ctx.load_module(ctx.main_module, do_init=False)
File: "c:\users\niklas\repos\nodepy\nodepy\nodepy\context.py", line 253, in load_module
  module.load()
File: "c:\users\niklas\repos\nodepy\nodepy\nodepy\loader.py", line 43, in load
  exec(code, vars(self.namespace))
File: "C:\Users\niklas\repos\nodepy\craftr\lib\main.py", line 110, in <module>
  sys.exit(main())
File: "C:\Users\niklas\repos\nodepy\craftr\lib\main.py", line 106, in main
  return backend.build_main(backend_args, session, module)
File: "C:\Users\niklas\repos\nodepy\craftr\lib\build_backends\default.py", line 194, in build_main
  executor.run(actions)
File: "C:\Users\niklas\repos\nodepy\craftr\lib\build_backends\default.py", line 171, in run
  self.wait()
File: "C:\Users\niklas\repos\nodepy\craftr\lib\build_backends\default.py", line 137, in wait
  self.pool.shutdown(wait=True)
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\concurrent\futures\thread.py", line 144, in shutdown
  t.join()
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\threading.py", line 1056, in join
  self._wait_for_tstate_lock()
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\threading.py", line 1072, in _wait_for_tstate_lock
  elif lock.acquire(block, timeout):

I can't make sense from this traceback. It appears that Thread._wait_for_tstate_lock() never returns (I checked multiple times, it always hangs at that line). There is no thread running other than the main thread (6068) and the tracer thread (4856).

I don't quite understand the implementation details of threading.Thread. What could cause Thread._tstate_lock.acquire() to block indefinitely?

---

Update 2017/11/07 -- 01:45am CEWT

This seems to happen when pool.shutdown() is called multiple times...

Hammack

Answer 1

I'm not 100% certain it's the reason you're seeing this as you are using Windows, but I encountered this on Linux with Python 3.6 in a similar scenario.

I was using .shutdown() on a concurrent.futures.ThreadPoolExecutor and the program would seem to hang up. Sometimes it would finally exit after 30-60 seconds.

Ctrl-C always resulted in a traceback showing it was sitting in _wait_for_tstate_lock()

Note: in Python 3 a second Ctrl-C actually exits

My problem occurred when the function submitted was using time.sleep() in a loop.

Looking at the HtmlFileTracer implementation in current nodepy code on github I see a similar scenario to what I was doing, (continuously loop and sleep for an interval unless some sort of flag was set)

Magisterial

Answer 2

If you can (i.e. if your logic allows), set daemon=True:

• Subclass ThreadPoolExecutor;
• Override _adjust_thread_count method, setting daemon=True in the Thread creation.

This will ensure the program finishes even if some threads hang.

Crum