I want to reverse engineering the contents generated by scrolling down in the webpage. The problem is in the url https://www.crowdfunder.com/user/following_page/80159?user_id=80159&limit=0&...

I am new to all instruments here. My goal is to extract all URLs from a lot of pages which are connected moreless by a "Weiter"/"next" button - that for several URLS. I decided to try that with scr...

I've read through many of the related questions but am still unclear how to do this as there are many software combinations available and many solutions seem outdated. What is the best way to inst...

I have ran across an issue in which my Lua script refuses to execute. The returned response from the ScrapyRequest call seems to be an HTML body, while i'm expecting a document title. I am assuming...

I am trying to scrape a few dynamic websites using Splash for Scrapy in python. However, I see that Splash fails to wait for the complete page to load in certain cases. A brute force way to tackle ...

We've been using scrapy-splash middleware to pass the scraped HTML source through the Splash javascript engine running inside a docker container. If we want to use Splash in the spider, we configu...

My spider.py file is as so: def start_requests(self): for url in self.start_urls: yield scrapy.Request( url, self.parse, headers={'My-Custom-Header':'Custom-Header-Content'}, meta={ 'splash...

I am trying out scrapy with splash to scrape dynamic content off the web, I'm on a windows 10 Home Edition. Is there an way to use Docker tool box instead of docker-desktop so as to work with splas...

I am trying to login to a website using the following code (slightly modified for this post): import scrapy from scrapy_splash import SplashRequest from scrapy.crawler import CrawlerProcess clas...

I installed Splash using this link. Followed all steps to installation, but Splash doesn't work. My settings.py file: BOT_NAME = 'Teste' SPIDER_MODULES = ['Test.spiders'] NEWSPIDER_MODULE = 'Tes...

I have the following code that is partially working, class ThreadSpider(CrawlSpider): name = 'thread' allowed_domains = ['bbs.example.com'] start_urls = ['http://bbs.example.com/diy'] rules ...

What I'm trying to do On avito.ru (Russian real estate site), person's phone is hidden until you click on it. I want to collect the phone using Scrapy+Splash. Example URL: https://www.avito.ru/mo...

I have some issue with Aquarium and splash. They stop working after 30 minutes after the start. A number of pages for loading are 50K-80K. I made cron job for automatically rebooting every 10 minut...

I have a scrapy spider that uses splash which runs on Docker localhost:8050 to render javascript before scraping. I am trying to run this on heroku but have no idea how to configure heroku to start...

I'm using Scrapy to do some crawling with Splash using the Scrapinghub/splash docker container however the container exit after a while by itself with exit code 139, I'm running the scraper on an A...

I use scrapy-splash to crawl web page, and run splash service on docker. commond: docker run -p 8050:8050 scrapinghub/splash --max-timeout 3600 But I got a 504 error. "error": {"info": {"time...

The Scrapy Splash I am using is working just fine on my local machine, but it returns this error when I use it on my Ubuntu server. Why is that? Is it caused by low memory? File "/usr/local/lib64...

I use scrapy-splash to build my spider. Now what I need is to maintain the session, so I use the scrapy.downloadermiddlewares.cookies.CookiesMiddleware and it handles the set-cookie header. I know ...

I'm trying to crawl Google Scholar search results and get all the BiBTeX format of each result matching the search. Right now I have a Scrapy crawler with Splash. I have a lua script which will cli...