Clicking link using beautifulsoup in python

Question

In mechanize we click links either by using follow_link or click_link. Is there a similar kind of thing in beautiful soup to click a link on a web page?

Terramycin

Answer 1

BeautifulSoup is an HTML parser.

Further discussion really depends on the concrete situation you are in and the complexity of the particular web page.

If you need to interact with a web-page: submit forms, click buttons, scroll etc - you need to use a tool that utilizes a real browser, like selenium.

In certain situations, for example, if there is no javascript involved in submitting a form, mechanize would also work for you.

And, sometimes you can handle it by simply following the link with urllib2 or requests.

Achieve

Thanks. I was wondering if it is possible to modify the link using python as we do using inspect element in browsers? I tried using mechanize. The link got modified as it appeared after printing it but when clicked it still opened the same page. – Terramycin

@user3286661 ok, could you show the code you are using? What is the purpose of it? Thanks. – Achieve

Oh yes! I have posted a question already. Here it is: #23677942 – Terramycin

@user3286661 gotcha, will take a look at your particular mechanize issue. I also don't think there is smth to add for the current question. – Achieve

@Achieve can you please help out with a problem I have with beautiful soup? – Puerile

Answer 2

print(soup.find('h1',class_='pdp_product_title')) does not give any result <div class="pr2-sm css-1ou6bb2"><h2 class="headline-5-small pb1-sm d-sm-ib css-1ppcdci" data-test="product-sub-title">Women's Shoe</h2><h1 id="pdp_product_title" class="headline-2 css-zis9ta" data-test="product-title">Nike Air Force 1 Shadow</h1></div>

Biannulate