Getting Keras (with Theano) to work with Celery

Question

I have some keras code which works synchronously to predict a given input, I have even made amendments so it can work with standard multi-threading (using locks in a seperate class from this) however when running via asynchronous celery (even with one worker and one task) I get an error on calling predict on the keras model.

@app.task
def predict_task(param):
  """Run task."""
  json_file = open('keras_model.json', 'r')
  loaded_model_json = json_file.read()
  json_file.close()
  model = model_from_json(loaded_model_json)
  model.load_weights('keras_weights.h5')

  tokenizer_file = open('tokenizer.pickle',
                      'rb')
  tokenizer = pickle.load(tokenizer_file)
  sequences = tokenizer.texts_to_sequences(['example text'])
  one_hot_encoded_text_vector = pad_sequences(sequences, maxlen=140,
                                            padding="post")

  one_hot_encoded_text_vector = np.asarray(one_hot_encoded_text_vector)
  return model.predict(x=one_hot_encoded_text_vector, batch_size=1)

Error:

Current thread 0x00007fff736b2300 (most recent call first):
  File "venv/lib/python3.5/site-packages/theano/compile/function_module.py", line 903 in __call__
  File "venv/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 1227 in __call__
  File "venv/lib/python3.5/site-packages/keras/engine/training.py", line 1301 in _predict_loop
  File "venv/lib/python3.5/site-packages/keras/engine/training.py", line 1800 in predict
  File "venv/lib/python3.5/site-packages/keras/models.py", line 1027 in predict
  File "tasks.py", line 157 in predict_task
  File "venv/lib/python3.5/site-packages/celery/app/trace.py", line 438 in __protected_call__
  File "venv/lib/python3.5/site-packages/celery/app/trace.py", line 240 in trace_task
  File "venv/lib/python3.5/site-packages/celery/app/trace.py", line 349 in _fast_trace_task
  File "venv/lib/python3.5/site-packages/billiard/pool.py", line 367 in workloop
  File "venv/lib/python3.5/site-packages/billiard/pool.py", line 295 in run
  File "venv/lib/python3.5/site-packages/billiard/process.py", line 292 in _bootstrap
  File "venv/lib/python3.5/site-packages/billiard/forking.py", line 105 in __init__
  File "venv/lib/python3.5/site-packages/billiard/process.py", line 137 in start
  File "venv/lib/python3.5/site-packages/billiard/pool.py", line 1068 in _create_worker_process
  File "venv/lib/python3.5/site-packages/celery/concurrency/asynpool.py", line 415 in _create_worker_process
  File "venv/lib/python3.5/site-packages/billiard/pool.py", line 972 in __init__
  File "venv/lib/python3.5/site-packages/celery/concurrency/asynpool.py", line 401 in __init__
  File "venv/lib/python3.5/site-packages/celery/concurrency/prefork.py", line 119 in on_start
  File "venv/lib/python3.5/site-packages/celery/concurrency/base.py", line 131 in start
  File "venv/lib/python3.5/site-packages/celery/bootsteps.py", line 374 in start
  File "venv/lib/python3.5/site-packages/celery/bootsteps.py", line 123 in start
  File "venv/lib/python3.5/site-packages/celery/worker/__init__.py", line 206 in start
  File "venv/lib/python3.5/site-packages/celery/bin/worker.py", line 212 in run
  File "venv/lib/python3.5/site-packages/celery/bin/base.py", line 274 in __call__
  File "venv/lib/python3.5/site-packages/celery/bin/worker.py", line 179 in run_from_argv
  File "venv/lib/python3.5/site-packages/celery/bin/celery.py", line 717 in execute
  File "venv/lib/python3.5/site-packages/celery/bin/celery.py", line 785 in handle_argv
  File "venv/lib/python3.5/site-packages/celery/bin/base.py", line 311 in execute_from_commandline
  File "venv/lib/python3.5/site-packages/celery/bin/celery.py", line 793 in execute_from_commandline
  File "venv/lib/python3.5/site-packages/celery/bin/celery.py", line 81 in main
  File "venv/lib/python3.5/site-packages/celery/__main__.py", line 30 in main
  File "venv/bin/celery", line 11 in <module>
[2018-03-08 02:56:07,060: ERROR/MainProcess] Process 'Worker-1' pid:3459 exited with 'signal 11 (SIGSEGV)'
[2018-03-08 02:56:07,075: ERROR/MainProcess] Task tasks.predict_task[287f85d3-f4dd-4806-ada4-286a4361f805] raised unexpected: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',)
Traceback (most recent call last):
  File "venv/lib/python3.5/site-packages/billiard/pool.py", line 1175, in mark_as_worker_lost
    human_status(exitcode)),
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).

Anele

This won't solve your problem, but it's better to open files in a context manager (as below) because that always closes the file descriptor: with open('tokenizer.pickle', 'rb') as tokenizer_file: tokenizer = pickle.load(tokenizer_file) – Ludlow

Which Theano, Keras and Celery version are you using? python -c "import theano, keras, celery; print(thano.__version__, keras.__version, celery.__version__)" – Ludlow

SIGSEGV means process trying to access an inaccessible memory location. What line exactly causes the issue? Does the model run outside the worker? – Jannjanna