cURL not working sometimes and gives empty result

Question

I used cURL to get data from another website. Sometimes it shows data and sometimes empty result

Here is my Code

    function get_data($url) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        $agent=$_SERVER["HTTP_USER_AGENT"];
        curl_setopt($ch,CURLOPT_USERAGENT, $agent); 
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
         curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false);

        $data = curl_exec($ch);
        curl_close($ch);
        return $data;
    }
    $returned_content = get_data('www.example.com');
    echo $returned_content;

Charmion

Try to check what happens when failing by using curl_error($ch); php.net/curl_error – Concern

For me is working well, maybe you have problem connecting. – Sinker

Probably has something to do with the page you're requesting, but you could try this funct instead cus it has some error handling: geneticcoder.blogspot.com/2015/02/curl-function-for-php.html – Thedathedric

Try adding exit; after echo $returned_content; – Katelynnkaterina

Btw www.example.com is not a URL, you are missing the protocol. – Austrasia

See also the posted answers to PHP cURL, read remote file and write contents to local file – Pibroch

Answer 1

Possibly you call too many connections from your curl_init to one Ip Address, So the server blocks the connection and causes the on/off errors.

Oshiro

Answer 2

Not receiving any content back could be due to one or more out of many different reasons and you need to figure out which.

1. Use curl_error($ch) in your code after the transfer has been performed to see if curl has told you about a problem.

2. Investigate the response headers. A HTTP transfer can be successful and not return any error but might only respond with HTTP headers and no body at all. The HTTP headers may then contain clues as to why. Perhaps it is a redirect you should follow, perhaps it requires HTTP authentication or perhaps the server indicates a temporary server glitch.

Treen

Answer 3

Curl in PHP returns null on AWS EC2 instance

I had similar issue and I fixed it by making sure versions and settings in php.ini file which is in PHP5 and Apache2 folders were same. If its not then Apache2 tries to execute versions set inside the php.ini settings. Also make sure PHP5-libcurl is installed. Its important too.

Harmaning

Answer 4

Possibly the intermittent failures are due to flaky DNS servers, even if it isn't, you'll still benefit from using Google's DNS servers.. See: https://developers.google.com/speed/public-dns/docs/using

Dantzler

Lookups are cached locally. You don't contact a DNS server everytime for every request to the same domain. That's why lookups are served with a TTL. I also doubt it's generally better to use Google's DNS, for privacy reasons. – Austrasia

Answer 5

Solution

Make sure you have no space for any variables or in url. Like "$user_name = "Lokman Hosen"". You have to give urldecode "$user_name = "Lokman_Hosen"" or "$user_name = "Lokman%20Hosen"". You can use

urldecode()

Function to remove space. Now cURL dont allow any space.

Propriety

Answer 6

I ran into this issue trying to connect to an API multiple times in succession.

My answer is pretty much the same as @developer:

Possibly you call too many connections from your one Ip Address, So the server blocks the connection and causes the on/off errors.

This works for me as a work around.

Create a method that attempts the curl request a few times, adding time between each request until it gets data or gives up.

/**
 * Thanks to issues with sometimes not receiving data when making multiple requests
 * here we make a reasonable effort to get data even when a firewall or WAF or ?? maybe
 * throttling request, DDOS, or on fire
 *
 * @param Curl $ch
 *
 * @return bool|string
 */
function curlExecFireWalker($ch)
{
   //In case we fail, let's count our failures
   $wowFailCount = 0;
   do {
      $data = curl_exec($ch);
      if (!$data) {
         //Well this is awkward, we are expecting data and didn't get any :(
         $wowFailCount++;
         //Let's calculate the length of a smoke break
         $smokeBreak = pow($wowFailCount, 2);
         //Smoke 'em if you got 'em
         sleep($smokeBreak);
      }
      //Well let's not allow to many failures and get out of here before the entire thing is on fire
   } while (!$data && $wowFailCount <= 3);

   return $data;

}

Then instead of

$data = curl_exec($ch);

call the fire walker method

$data = curlExecFireWalker($ch);

Quench