Scrapyd jobid value inside spider

Question

Framework Scrapy - Scrapyd server.

I have some problem with getting jobid value inside the spider.

After post data to http://localhost:6800/schedule.json the response is

status = ok
jobid = bc2096406b3011e1a2d0005056c00008

But I need use this jobid inside the current spider during the process. It can be used for open {jobid}.log file or other dynamic reasons.

class SomeSpider(BaseSpider):
    name = "some"
    start_urls = ["http://www.example.com/"]
    def parse(self, response):
        items = []
        for val in values:
            item = SomeItem()
            item['jobid'] = self.jobid # ???!
            items.append(item)
        return items

But I see this jobid only after the task is finihed :( Thanks!

Basinger

Answer 1

I guess there is an easier way, but you can extract job id from command line args. IIRC, scrapyd launches a spider giving it a jobid in parameters. Just explore sys.args where you need jobid.

Goa

All genius is easy ;) Thanks, mate! Some example: if (len(sys.argv)>2): if ('_job' in sys.argv[3]): self.jobid = sys.argv[3].rsplit('=') – Basinger

@Maxim, glad it worked. Please, don't forget to accept and upvote answers that worked for you. – Goa

It requeres 15 points of reputation. I'll back to this post after some growth ;) Thank you. – Basinger

You can also get it from the SCRAPY_JOB environment variable: os.environ['SCRAPY_JOB'] – Monochromatic

@PabloHoffman what happens if we have multiple schedules running? I get the jobids but I am not sure if they will be correct every time. – Hutcherson

This is now os.environ['SHUB_JOBKEY'] – Bisson

Answer 2

You can get it from the SCRAPY_JOB environment variable:

os.environ['SCRAPY_JOB']

Boutin

This is now os.environ['SHUB_JOBKEY'] – Bisson

Answer 3

In the spider.py -->

class SomeSpider(BaseSpider):
    name = "some"
    start_urls = ["http://www.example.com/"]

    def __init__(self, *args, **kwargs):
        super(SomeSpider, self).__init__(*args, **kwargs)
        self.jobid = kwargs.get('_job')

    def parse(self, response):
        items = []
        for val in values:
           item = SomeItem()
           item['jobid'] = self.jobid # ???!
           items.append(item)
        return items

Daytime