Python Reddis Queue ValueError: Functions from the __main__ module cannot be processed by workers

Question

I'm trying to enqueue a basic job in redis using python-rq, But it throws this error

"ValueError: Functions from the main module cannot be processed by workers"

Here is my program:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

from rq import Connection, Queue
from redis import Redis
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

Decagon

Answer 1

Break the provided code to two files: count_words.py:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

and main.py (where you'll import the required function):

from rq import Connection, Queue
from redis import Redis
from count_words import count_words_at_url # added import!
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

I always separate the tasks from the logic running those tasks to different files. It's just better organization. Also note that you can define a class of tasks and import/schedule tasks from that class instead of the (over-simplified) structure I suggest above. This should get you going.. Also see here to confirm you're not the first to struggle with this example. RQ is great once you get the hang of it.

Sclerotomy

Is there any way around this? I find it makes my project fragmented when, for example, I want one task to create multiple other tasks and submit them. Now my two tasks, though logically related, have to be split into multiple files so that one can import the other. I noticed that the code in rq's Job class checks for instances of string_types, so it's not clear why you can't submit using 'this_filename.some_other_func', but it does't work. – Claptrap

@sheridp, post a new question with details about your problem, will try to help. – Sclerotomy

Thanks @GG_Python, I just created a new question – Claptrap

did that but the output is some # of None – Punish

Answer 2

Currently there is a bug in RQ, which leads to this error. You will not be able to pass functions in enqueue from the same file without explicitly importing it.

Just add from app import count_words_at_url above the enqueue function:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

from rq import Connection, Queue
from redis import Redis
redis_conn = Redis()
q = Queue(connection=redis_conn)

from app import count_words_at_url
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

The other way is to have the functions in a separate file and import them.

Jerroldjerroll