Click button on website then scrape web page

Question

I have a website I would like to click a button on then scrape the website using python the html code between the button is:

 <span id="exchange-testing" class="exchange-input nav-link" data track="&amp;lid=testing&amp;lpos=site_settings" data-value="testing">Testing</span>

Is this possible? I am able to scrape all the data I need from the page but I need to click the button first.

Any help would be appreciated

Jaggy

Answer 1

Basically, you have two options:

• high-level approach: automate a real browser using selenium or, in other words, make the browser repeat all the user actions needed to get to the page with the desired data.

• low-level approach: when you click the button, investigate what is happening under the hood - explore the "Network" tab of browser developer tools and see what requests are being made. Then, simulate them in your scraper. Here, you may consider using tools like requests, mechanize for making requests, handling scraping sessions, submitting forms etc and tools like BeautifulSoup, lxml.html for html parsing. Also, Scrapy web-scraping framework is a must see.

Cookstove

Answer 2

If there is a button you want to click then scrape then do the following:

1. Inspect the button element . the link the button will open after clicking it will always be in the html you opened .
2. Scrape this link and do requests.get(link) . this becomes equivalent to clicking button .

but this isn't applicable when the button needs to activate some js function that should've run on clicking the element.

Troglodyte