what are the advantages use scrapyd?

Question

The scrapy doc says that:

Scrapy comes with a built-in service, called “Scrapyd”, which allows you to deploy (aka. upload) your projects and control their spiders using a JSON web service.

is there some advantages in comformance use scrapyd?

Natie

Answer 1

Scrapyd allows you to run scrapy on a different machine than the one you are using via a handy web API which means you can just use curl or even a web browser to upload new project versions and run them. Otherwise if you wanted to run Scrapy in the cloud somewhere you would have to scp copy the new spider code and then login with ssh and spawn your scrapy crawl myspider.

Scrapyd will also manage processes for you if you want to run many spiders in parallel; but if you have Scrapy on your local machine and have access to the command-line or a way to run spiders and just want to run one spider at a time, then you're better off running the spider manually.

If you are developing spiders then for sure you don't want to use scrapyd for quick compile/test iterations as it just adds a layer of complexity.

Servitor

a while ago i was actually using celery to control my spiders. What would be the pros and cons of this vs that.. – Twit

How can I secure that web API such that no port scanning people get to know about my Scrapyd deployment and can not use the same API? I know I am late by 3 years, but I failed to find anything regarding my query. – Lael