Using multiple cores with Python and Eventlet

Question

I have a Python web application in which the client (Ember.js) communicates with the server via WebSocket (I am using Flask-SocketIO). Apart from the WebSocket server the backend does two more things that are worth to be mentioned:

• Doing some image conversion (using graphicsmagick)
• OCR incoming images from the client (using tesseract)

When the client submits an image its entity is created in the database and the id is put in an image conversion queue. The worker grabs it and does image conversion. After that the worker puts it in the OCR queue where it will be handled by the OCR queue worker.

So far so good. The WS requests are handled synchronously in separate threads (Flask-SocketIO uses Eventlet for that) and the heavy computational action happens asynchronously (in separate threads as well).

Now the problem: the whole application runs on a Raspberry Pi 3. If I do not make use of the 4 cores it has I only have one ARMv8 core clocked at 1.2 GHz. This is very little power for OCR. So I decided to find out how to use multiple cores with Python. Although I read about the problems with the GIL) I found out about multiprocessing where it says The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads.. Exactly what I wanted. So I instantly replaced the

from threading import Thread
thread = Thread(target=heavy_computational_worker_thread)
thread.start()

by

from multiprocessing import Process
process = Process(target=heavy_computational_worker_thread)
process.start()

The queue needed to be handled by the multiple cores as well So i had to change

from queue import Queue
queue = multiprocessing.Queue()

to

import multiprocessing
queue = multiprocessing.Queue()

as well. Problematic: the queue and the Thread libraries are monkey patched by Eventlet. If I stop using the monkey patched version of Thread and Queue and use the one from multiprocsssing instead then the request thread started by Eventlet blocks forever when accessing the queue.

Now my question:

Is there any way I can make this application do the OCR and image conversion on a separate core?

I would like to keep using WebSocket and Eventlet if that's possible. The advantage I have is that the only communication interface between the processes would be the queue.

Ideas that I already had: - Not using a Python implementation of a queue but rather using I/O. For example a dedicated Redis which the different subprocesses would access - Going a step further: starting every queue worker as a separate Python process (e.g. python3 wsserver | python3 ocrqueue | python3 imgconvqueue). Then I would have to make sure myself that the access on the queue and on the database would be non-blocking

The best thing would be to keep the single process and make it work with multiprocessing, though.

Thank you very much in advance

Symbol

I do not know the Evenlet library, so maybe my answer will not be applicable. Use multithreading in a "main" program. Within each "Process" in that main application, use "subprocess" to call subprograms (python program calling a python program, even if this seems strange). Use the "Eventlet" library (or any non process-safe library) only in these subprocesses. Do not use them in the main program. You will not be able to use "Queue", but you can pass data through files (example: program A writes an image file then terminates, program B is started and reads this file). – Pavid

Answer 1

Eventlet is currently incompatible with the multiprocessing package. There is an open issue for this work: https://github.com/eventlet/eventlet/issues/210.

The alternative that I think will work well in your case is to use Celery to manage your queue. Celery will start a pool of worker processes that wait for tasks provided by the main process via a message queue (RabbitMQ and Redis are both supported).

The Celery workers do not need to use eventlet, only the main server does, so this frees them to do whatever they need to do without the limitations imposed by eventlet.

If you are interested in exploring this approach, I have a complete example that uses it: https://github.com/miguelgrinberg/flack.

Radon

Thank you, this works like charm. The only problem I am facing now is that I've had the socketio instance in a global variable which of course can not be shared across multiple processes. But this should be solvable by using a dedicated storage to share it or give the variable to the task functions. – Symbol

See the project I referenced in my answer. If you need to emit from the worker processes, then create a socketio instance in each process. The difference is that the socketio instance in your Celery workers will not have a Flask app instance associated with them, so they are emit only. You do not need to share anything between the processes, all communications are done through the message queue. – Radon

I followed the instructions you documented in github.com/miguelgrinberg/Flask-SocketIO/blob/master/docs/… - using the constructor with message_queue='redis://' as a parameter solved the problem :) this way I still only have one instance of socketio allowing me to emit from the worker processes. – Symbol

@MiguelGrinberg This may be a simple question, but it's been bugging me for a while. When we create a Flask SocketIO server instance in another process without passing it the application object, how does this socketio "know" where to emit to? Is this done through the message queue? The socketio that has the application context simply emits to connected clients. The socketio in the external process doesn't have any connected clients (at least that's how it seems to me). – Kariekaril

@Kariekaril yes, the server and the external process coordinate the emit operation by passing messages over the message queue. Even though you are emitting from the external process, the actual emit happens in the server process, as this is the process that owns the socket connections to the clients. – Radon