How do I pass form data with Scrapy from the command line?

Question

How could I pass username and password from the command line? Thanks!

class LoginSpider(Spider):
    name = 'example.com'
    start_urls = ['http://www.example.com/users/login.php']

    def parse(self, response):
        return [FormRequest.from_response(response,
                    formdata={'username': 'john', 'password': 'secret'},
                    callback=self.after_login)]

    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.log("Login failed", level=log.ERROR)
            return

        # continue scraping with authenticated session...

Epiclesis

Answer 1

Open terminal and make sure scrapy installed.

1. scrapy shell

2. from scrapy.http import FormRequest

3. request=FormRequest(url='http://www.example.com/users/login.php',formdata={'username': 'john','password':'secret',})

Information:

• Scrapy 1.0.0

Soleure

Answer 2

you can do

scrapy crawl spidername -a username="john" -a password="secret"

and then

class LoginSpider(Spider):
    name = 'example.com'
    start_urls = ['http://www.example.com/users/login.php']

    def parse(self, response):
        return [FormRequest.from_response(response,
                    formdata={'username': self.username, 'password': self.password},
                    callback=self.after_login)]

    def after_login(self, response):
        # check login succeed before going on
        if "authentication failed" in response.body:
            self.log("Login failed", level=log.ERROR)
            return

        # continue scraping with authenticated session...

Brnaba