Passing results to depending on job - python rq

Question

How do I pass the result of a job to a job that depends on it?

What I currently do is passing id of the first job to the second,

first = queue.enqueue(firstJob)
second = queue.enqueue(secondJob, first.id, depends_on=first);

And inside secondJob fetching the first job to get the result

first = queue.fetch_job(previous_job_id)
print first.result

Is this the recomended way? Is there any other pattern that I can use to directly pass first job's result to second?

Hanhhank

Answer 1

You can access info about the current job and its dependencies from within the job itself. This negates the need to explicitly pass the id of the first job.

Define your jobs:

from rq import Queue, get_current_job
from redis import StrictRedis

conn = StrictRedis()
q = Queue('high', connection=conn)

def first_job():
    return 'result of the first job'

def second_job():
    current_job = get_current_job(conn)
    first_job_id = current_job.dependencies[0].id
    first_job_result = q.fetch_job(first_job_id).result
    assert first_job_result == 'result of the first job'

Enqueue your jobs:

first = queue.enqueue(first_job)
second = queue.enqueue(second_job, depends_on=first)

Note that the current_job can have multiple dependencies so current_job.dependencies is a list.

Battledore

I don't think a job can have multiple dependencies. That feature never got merged in. – Concavity

It looks like there is a fetch_dependencies method though at least as of v1.10.0: github.com/rq/rq/blob/master/rq/job.py#L476 – Intercom

Answer 2

In my own setup, I use two jobs, the second dependent on the results of the first. They are running on separate queues, and if the first job gets successful results, it places a job in the queue for the second, passing the necessary data when it creates the job. It works fairly well for me.

Hope this helps.

Gawlas

Answer 3

With the rq 0.13.0, we can get parent job by

current_job = get_current_job(redis_conn)
job = current_job.dependency

Sverre