import requests from bs4 import BeautifulSoup from requests_html import HTMLSession url="https://dmarket.com/ingame-items/item-list/csgo-skins?title=recoil%20case" sesion = HTMLSession() ...

I have troubles sorting a wiki table and hope someone who has done it before can give me advice. From the List_of_current_heads_of_state_and_government I need countries (works with the code below) ...

I am working on a Linkedin web scraping project. I am trying to get the list of companies that interest someone (notice I am not using the API). It is a dynamic website, so I would need to scroll d...

I'm trying to use the webdriver and Selenium, it was working fine a couple days ago but I'm currently facing this issue where I receive this error: [Errno 8] Exec format error: '/Users/[USER]/.wdm/...

undetected_chromedriver with webdriver_manager was working well few days ago for scraping websites but out of nowhere it started throwing the error: OSError: [Errno 8] Exec format error: '/Users/p...

google has an API for downloading search suggestions: https://www.google.com/support/enterprise/static/gsa/docs/admin/70/gsa_doc_set/xml_reference/query_suggestion.html unfortunately, as far as i...

I'm creating a web api that scrapes a given url and sends that back. I am using Puppeteer to do this. I asked this question: Puppeteer not behaving like in Developer Console and recieved an answer ...

I'm new to web scraping and want to download all images on a webpage using puppeteer: const puppeteer = require('puppeteer'); let scrape = async () => { // Actual Scraping goes Here... const...

Is there anyone who has tried to extract the individual risk factors from the Risk Factors section i.e. Item 1A from the EDGAR 10-K filings of the company using BeautifulSoup or any other web scrap...

After running the scrapy shell with the defined url, I am getting the attribute error showing the following error: AttributeError: module 'OpenSSL.SSL' has no attribute 'SSLv3_METHOD' scrapy shell ...

I'm wondered if it's possible in google sheet to convert Gregorian calendar to jalali using a function. In fact, I have some date such as : February 20, 2021 4:30 AM I need to display this date in...

As you can see with the sample code below, I'm using Puppeteer with a cluster of workers in Node to run multiple requests of websites screenshots by a given URL: const cluster = require('cluster')...

I'm trying to use phantomJS to capture a screenshot of a URL, however when I call phantomJS (from either the command line or web app) it hangs and seesm to never execute the "exit()" call. I can't ...

I try to scrape this site by Selenium. I want to click in "Next Page" buttom, for this I do: driver.find_element_by_class_name('pagination-r').click() it works for many pages but not for...

What are the steps to upload the crawled data from Scrapy to the Amazon s3 as a csv/jsonl/json file? All i could find from the internet was to upload scraped images to the s3 bucket. I'm currently...

I'm relatively new to using Chrome developer tools/doing XPath searches/this kind of programming in general, so please excuse any incorrect terminology or vague-sounding descriptions. I think the s...

I attempted the following: Utilize the German stock exchange's API (https://api.boerse-frankfurt.de/v1/search/equity_search) to retrieve index values. The API can be accessed externally using para...

I want to log into my amazon account and retrieve purchase history programmatically in java. I did a lot of research and came across screen scraping. Is this the only way or does amazon provide api...

I use this program to get the json data from https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY but since this morning it's not working as it returns <Response [401]>. The link lo...

I am trying to extract the link inside a href but all I am finding it is the text inside the element The website code is the following: <div class="item-info-container "> <a hre...

i have a page with cards i click on the title of each it takes to a page then i click on a link in that page and it takes me another page to extract the data that i want. After that i navigate back...

I have the following code, where I store all src in an array, I would like to store only img with class name xyz const imgs = await page.$$eval('img[src]', imgs => imgs.map(img => img.getAtt...

Chrome 122.0.6261.95 Chrome driver 122.0.6261.94 Python 3.8.3 If I comment out option.add_argument("--headless=new"), it will print(len(elements)) will print 2. Otherwise, can not print a...

I am trying to scrape one of the free proxy listings website but, I just couldn't be able to scrape the proxies. Below is my code: import requests import re url = 'https://free-proxy-list.net/' ...

I am writing an application in node.js that will navigate to a website, click a button on the website, and then extract certain pieces of data from the website. All is going well except for the but...