curl through proxy returns no content

Question

I'm working on a PHP script right now which sends requests to our school's servers to get real-time information about class sizes for different courses. The script runs perfectly fine when I don't use a proxy, returning a string full of course numbers and available seats. However, I want to make this a service for the students, and I'm afraid that if I make too many requests my ip will get blocked. So I'm attempting to do this through a proxy, with no success. As soon as I add the CURLOPT_HTTPPROXYTUNNEL and CURLOPT_PROXY fields to my requests nothing gets returned. I'm not even sure how to troubleshoot it at this point since I'm not getting an error message of any kind. Does anyone know what's going on or how to at least troubleshoot it?

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$proxy = explode(':', $proxy);
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'tempcookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'tempcookie.txt');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_REFERER, $ref);
$exec = curl_exec($ch);

echo curl_error($ch);
print_r(curl_getinfo($ch));
echo $exec;

Proxy used for tests: 75.147.173.215:8080

Quadruplex

why not to cache results instead of being fraud? – Hallucinogen

Answer 1

I use the following code if I have to use a proxy with curl:

$proxy = "127.0.0.1:8080"; // or something like that

if($proxy !== null){

    // no need to specify PROXYPORT again
    curl_setopt($ch, CURLOPT_PROXY, $proxy);

    // to make the request go through as though proxy didn't exist
    curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);

}

Psychopath

Answer 2

You can set CURLOPT_STDERR and CURLOPT_VERBOSE curl options to save your errors in a file. Also, you may use curl_error() function. BTW, by default, curl should show all errors in STDERR.

Besides, for general check, you can simply specify selected proxy in you browser configuration properties, try to open specific service in browser and see, whether correct response is returned.

UPDATE:

CURLOPT_HTTPPROXYTUNNEL is used to make curl call CONNECT HTTP method when requesting proxy server (see here for details). I tested code without this option - it worked successfully.

Code I used:

$proxy = "75.147.173.215:8080";
$proxy = explode(':', $proxy);
$url = "http://google.com";

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_HEADER, 1);

$exec = curl_exec($ch);

echo curl_error($ch);
print_r(curl_getinfo($ch));
echo $exec;

Frantz

I have a list of aout 10k of proxies... it's unreal to check it that way – Quadruplex

Just start with one of them, ensure, that it works with browser, then try to make it work from your script. After that try to work with all 10k proxies. – Frantz

Also, some web-sites block access from some public proxies, since they are frequently used for attacks. – Frantz

It's working in browser, ut I'm getting this error: couldn't connect to host when using curl – Quadruplex

What do you specify as parameter of CURLOPT_PROXY option? Generally, it's great to post such code in initial question, since it usually clarifies a lot of questions. – Frantz

One more question - do You know where I can get big list of working proxies? Most of these form my list don't want to cooperate – Quadruplex

Unfortunately, I can't help you with actual list of proxies. I implemented similar functionality ~8 years ago :) – Frantz

Answer 3

Here is a well tested function which i used for my projects with detailed self explanatory comments

---

There are many times when the ports other than 80 are blocked by server firewall so the code appears to be working fine on localhost but not on the server , try using port 80 proxies

function get_page($url){

global $proxy;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
//curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_HEADER, 0); // return headers 0 no 1 yes
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // return page 1:yes
curl_setopt($ch, CURLOPT_TIMEOUT, 200); // http request timeout 20 seconds
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // Follow redirects, need this if the url changes
curl_setopt($ch, CURLOPT_MAXREDIRS, 2); //if http server gives redirection responce
curl_setopt($ch, CURLOPT_USERAGENT,
    "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt"); // cookies storage / here the changes have been made
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // false for https
curl_setopt($ch, CURLOPT_ENCODING, "gzip"); // the page encoding

$data = curl_exec($ch); // execute the http request
curl_close($ch); // close the connection
return $data;
}

Postfix