Multi-thread support for python asyncio gRPC clients

Question

I have an asyncio gRPC client that is used in a multithreaded environment. When multiple threads connect to the service via the client simultaneously, I see a stream of the following errors:

2021-01-27 09:33:56,937 ERROR [asyncio] [thread_0] Exception in callback PollerCompletionQueue._handle_events()()
handle: )()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi", line 147, in grpc._cython.cygrpc.PollerCompletionQueue._handle_events
BlockingIOError: [Errno 11] Resource temporarily unavailable
2021-01-27 09:33:56,937 ERROR [asyncio] [thread_1] Exception in callback PollerCompletionQueue._handle_events()()
handle: )()>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi", line 147, in grpc._cython.cygrpc.PollerCompletionQueue._handle_events
BlockingIOError: [Errno 11] Resource temporarily unavailable

The requests appear to be completing successfully, however, the messages are flooding my logs and making me nervous!

In my tests, each thread creates its own channel and submits its own async requests. The errors occur regardless of the load on the service. The errors do not occur if the clients are running in different processes.

My setup:

• Python version: 3.8.6
• grpcio version: 1.35.0

Any insight is appreciated!

Foursquare

Answer 1

The gRPC AsyncIO uses a UDS to communicate between C extension and Python. From your log, there is a race condition for the fd access. The AsyncIO API supports multi-threading, but this looks like a new issue (it helps to create an issue on https://github.com/grpc/grpc/issues).

The fix of the race condition could be tricky, since AsyncIO uses AsyncIO locks which are not thread-safe. If we protect the fd with thread-safe locks, it might block the AsyncIO loop. Feel free to propose or contribute solutions.

AsyncIO performs best if you let all clients run on one single thread. The event loop will handle the execution of coroutines well without thread hops. If the goal is to saturate all computational power on the machine, as you mentioned, it's better to use multi-processing.

Link to basic gRPC AsyncIO example: https://github.com/grpc/grpc/blob/master/examples/python/helloworld/async_greeter_client.py

Yuji

thanks very much for the prompt reply. I have managed to fix the issue on my end by switching to a multi-processing model. Is it your preference that I create an issue on GitHub? Or should we leave it as a limitation of asyncio and UDS? – Foursquare

Hi, we do have a test case for many loop across threads. This is a feature we want to support. It really helps the library to improve, if you can create an issue with reproduction code. If the reproduction is too hard, just describe the problem as this SO post is fine. github.com/grpc/grpc/blob/master/src/python/grpcio_tests/… – Yuji

hey @Foursquare could you share your code that switches to a multi-processing model? I have the exact same issue, I see a lot of those logs coming, but everything seems to be handled fine in the end. I'm using the "grpc.server" function to initiate a server, calling .start() on that object starts it. I can share that code if useful. – Prober

Hi @jlos, apologies no code handy. I think we merely switched our client to using processpool from threadpool. No changes were made on the serving side. – Foursquare