Warning: file_get_contents: failed to open stream: Redirection limit reached, aborting

Question

I read over 20 related questions on this site, searched in Google but no use. I'm new to PHP and am using PHP Simple HTML DOM Parser to fetch a URL. While this script works with local test pages, it just won't work with the URL that I need the script for.

Here is the code that I wrote for this, following an example file that came with the PHP Simple DOM parser library:

<?php

include('simple_html_dom.php');

$html = file_get_html('http://www.farmersagent.com/Results.aspx?isa=1&name=A&csz=AL');

foreach($html->find('li.name ul#generalListing') as $e)
echo $e->plaintext;  

?>

And this is the error message that I get:

Warning: file_get_contents(http://www.farmersagent.com/Results.aspx?isa=1&name=A&csz=AL) [function.file-get-contents]: failed to open stream: Redirection limit reached, aborting in /home/content/html/website.in/test/simple_html_dom.php on line 70

Please guide me what should be done to make it work. I'm new so please suggest a way that is simple. While reading other questions and their answers on this site, I tried cURL method to create a handle but I failed to make it work. The cURL method that I tried keeps returning "Resources" or "Objects". I don't know how to pass that to Simple HTML DOM Parser to make $html->find() work properly.

Please help! Thanks!

Agnostic

I get 200 OK, no redirection at all when I try to access that file... – Amour

Hi, Kolink! Thanks for your comment. It's weird it shows an Error 500 on my Windows PC and when I run it on my Linux server, that's the error message it shows. – Agnostic

@ChandanMishra it would be good if you choose one of the answers and mark it as the correct one, if it solved your problem, ok? – Greenhead

Answer 1

Had a similar problem today. I was using CURL and it wasn't returning my any error. Tested with file_get_contents() and I got...

failed to open stream: Redirection limit reached, aborting in

Made a few searches and I'v ended with this function that works on my case...

function getPage ($url) {


$useragent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36';
$timeout= 120;
$dir            = dirname(__FILE__);
$cookie_file    = $dir . '/cookies/' . md5($_SERVER['REMOTE_ADDR']) . '.txt';

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt($ch, CURLOPT_ENCODING, "" );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt($ch, CURLOPT_AUTOREFERER, true );
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout );
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout );
curl_setopt($ch, CURLOPT_MAXREDIRS, 10 );
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com/');
$content = curl_exec($ch);
if(curl_errno($ch))
{
    echo 'error:' . curl_error($ch);
}
else
{
    return $content;        
}
    curl_close($ch);

}

The website was checking for a valid user agent and for cookies.

The cookie issue was causing it! :) Peace!

Trumaine

Nice one, works in my case too. You saved me a lot of time, thanks! – Fluorspar

If you return, the curl_close never happens. is this intentional? If so, you should move curl_close into the curl_errno block – Drawplate

Great!! The cookie was the missing parameter for my case. Thanks a lot! – Media

Great job! Thanks. – Deaminate

Thanks a lot ! It works for me too ! :) But the curl_close($ch); should be placed : 1) Before the return $content; 2) And also after the echo 'error:' . curl_error($ch); because return causes the function to end its execution immediately and pass control back to the line from which it was called. I also have to increase the number of redirections : curl_setopt($ch, CURLOPT_MAXREDIRS, 100 ); – Cheyney

Answer 2

Resolved with:

<?php
$context = stream_context_create(
    array(
        'http' => array(
            'max_redirects' => 101
        )
    )
);
$content = file_get_contents('http://example.org/', false, $context);
?>

You can also inform if you have a proxy in the middle:

$aContext = array('http'=>array('proxy'=>$proxy,'request_fulluri'=>true));
$cxContext = stream_context_create($aContext);

More details on: https://cweiske.de/tagebuch/php-redirection-limit-reached.htm (thanks @jqpATs2w)

Greenhead

Response more detailed in cweiske.de/tagebuch/php-redirection-limit-reached.htm – Fabaceous

Answer 3

Using cURL you would need to have the CURLOPT_RETURNTRANSFER option set to true in order to return the body of the request with call to curl_exec like this:

$url = 'http://www.farmersagent.com/Results.aspx?isa=1&name=A&csz=AL';
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
// you may set this options if you need to follow redirects. Though I didn't get any in your case
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
$content = curl_exec($curl);
curl_close($curl);

$html = str_get_html($content);

Redpoll

Hi Mike, Thanks for answering! I've tried that but I just don't know how to pass the handle from cURL to the DOM parser so the find() method starts working. I tried this code but please see how should it be written because this code returns a warning and a fatal error (Fatal error: Call to a member function find() on a non-object): – Agnostic

@ChandanMishra I am not familiar with the library you are using, but in glancing through the documentation there is a function to populate the DOM object from a string, which is what you will get back from cURL. Please see my revised answer above. – Redpoll

After trying str_get_html($content), I get a Fatal error: Fatal error: Call to a member function find() on a non-object in /home/content/html/website.in/test/test.php on line 21 What might be causing this? – Agnostic

@ChandanMishra I don't know have you tried doing variable dumps to see where the process is breaking down (i.e. you did not get a curl result, the DOM object was not created, etc.). – Redpoll

Hi Mike, A var_dump() for $content returns False. – Agnostic

After your curl_exec() call add in this var_dump(curl_error($this->curl)) to see what error curl is getting. – Redpoll

Hi Mike, I tried that but it generates a new fatal error "Using $this when not in object context". Meanwhile, I tried this code and it shows the output as: "Object moved to HERE" and that HERE is linked to "%2fErrorPage.aspx%3faspxerrorpath%3d%2fResults.aspx". link Sorry for an external link. Not quite familiar with comment formatting of this site. – Agnostic

When I try analyzing the page load using Firebug's "Net" tab, The first response that it gets is a 200 but the very second response is a 302 that's for Google Maps. Is it causing the trouble? – Agnostic

@ChandanMishra Sorry I didn't meean to put a $this->curl in there, should just be $curl. I am so used to working with a curl handle stored within a class, I think I typed that out of habit. – Redpoll

No worries. :) I tried it again after correcting the error. var_dump($content) outputs this: "string(31) "Maximum (20) redirects followed" bool(false)" – Agnostic

@ChandanMishra I am not sure why you are getting all those redirects. I do not see it in a browser. There is something strange about that website. – Redpoll

Should I try setting an user agent for cURL? I think the site has different settings for different user agents. – Agnostic

It could be that. Use curl_setopt($curl, CURLOPT_USERAGENT, 'YOUR USER AGENT VALUE') – Redpoll

That seemed to work. The script is still not working but now var_dump($content) shows: string(223513) as well as the HTML document starts loading in the browser window. That happens even if I do a print_r($content). But I tried to use $html = str_get_html($content); and then doing foreach($html->find('li.name ul#generalListing') as $e) echo $e->plaintext still doesn't work. – Agnostic

Answer 4

I'm not sure exactly why you redefined the $html object with a string from get html, The object is meant to be used for searching the string. If you overwrite the object with a string, the object no longer exists and cannot be used.

In any case, to search the string returned from curl.

<?php
$url = 'http://www.example.com/Results.aspx?isa=1&name=A&csz=AL';

include('simple_html_dom.php');

# create object
$html = new simple_html_dom();

#### CURL BLOCK ####

$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
# you may set this options if you need to follow redirects.
# Though I didn't get any in your case
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);

$content = curl_exec($curl);
curl_close($curl);

# note the variable change.
$string = str_get_html($content);

# load the curl string into the object.
$html->load($string);

#### END CURL BLOCK ####

# without the curl block above you would just use this.
$html->load_file($url);

# choose the tag to find, you're not looking for attributes here.
$html->find('a');

# this is looking for anchor tags in the given string.
# you output the attributes contents using the name of the attribute.
echo $html->href;
?>

you might be searching a different tag, the method is the same

# just outputting a different tag attribute
echo $html->class;

echo $html->id;

Teets

Answer 5

I also needed to add this HTTP context options ignore_errors :

see : https://www.php.net/manual/en/context.http.php

$arrContextOptions = array(
    "ssl" => array(
        // skip error "Failed to enable crypto" + "SSL operation failed with code 1."
        "verify_peer" => false,
        "verify_peer_name" => false,
         ),
     // skyp error "failed to open stream: operation failed" + "Redirection limit reached"
     'http' => array(
          'max_redirects' => 101,
          'ignore_errors' => '1'
      ),
           
  );

  $file = file_get_contents($file_url, false, stream_context_create($arrContextOptions));

Obviously, I only use it for quick debugging purpose on my local environment. It is not for production.

Vigilantism