Phantom.js / Casper.js with rotating proxy?
Asked Answered
P

1

7

I have a simple goal: load webpages with either phantom.js (out of the box) or casper.js (nice and easier) but using proxy and rotate it from a list if current one is bad (i.e. webpage loads fail or something like that).

I know casper.js has --proxy param but it dictates the user to specify only ONE proxy and use it during runtime.

Question #1 is: how to rotate proxy on the fly programmatically?

I did some research and found this node-requester but it's not integrated with casper.js. I tried to extract out just the proxy feature in the code but didn't really understand how it works in the nutshell (I am not that smart I guess).

So question #2: is there some simple implementation of proxy rotation that works with either phantom.js or casper.js?

I prefer to use the fancy casper.js though but will go down with phantom.js bare as well.

Passional answered 25/5, 2013 at 5:5 Comment(0)
E
6

I had the same issue a while back, I worked with PhantomJS. The solution we ended up with was running PhantomJS as a child process of a larger Java/Scala server which then handled failures and assigned the different proxies when needed (by rerunning with different params in the --proxy arg).

Excitation answered 10/6, 2013 at 15:37 Comment(1)
Its been a couple of years, is there a better solution today?Dali

© 2022 - 2024 — McMap. All rights reserved.