How to avoid "HTTP/1.1 999 Request denied" response from LinkedIn?

Question

I'm making request to LinkedIn page and receiving "HTTP/1.1 999 Request denied" response. I use AWS/EC-2 and get this response. On localhost everything works fine.

This is sample of my code to get html-code of the page.

<?php
error_reporting(E_ALL);
$url= 'https://www.linkedin.com/pulse/5-essential-strategies-digital-michelle';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
var_dump($response);
var_dump($info); 

I don't need whole page content, just meta-tags (title, og-tags).

Struve

Answer 1

Note that the error 999 don't exist in W3C Hypertext Transfer Protocol - HTTP/1.1, probably this error is customized (sounds like a joke)

LinkedIn don't allow direct access, the probable reason of them blocking any "url" from others webservers access should be to:

1. Prevent unauthorized copying of information
2. Prevent invasions
3. Prevent abuse of requests.
4. Force use API

Some IP addresses of servers are blocked, as the "IP" from "domestic ISP" are not blocked and that when you access the LinkedIn with web-browser you use the IP of your internet provider.

The only way to access the data is to use their APIs. See:

• Accessing LinkedIn public pages using Python
• Heroku requests return 999

Note: The search engines like Google and Bing probably have their IPs in a "whitelist".

Sunk

Yes Guilherme. It is not related to certificate or User-Agent. Looks like LinkedIn blacklisted AWS. – Struve

How browser gets access without API? The same way, via http. Problem in ban by IP/Subnet from LinkedIn. I tried to contact LinkedIn support and they don't want to do anything with it. They are ignore the problem, pushed me to Forum, which is not working. So I came here, maybe here exists some normal linkedin employee. But it looks like here most of people are bureaucrats and some stupid guys, who don't really understand the problem. Only you tried to help. – Struve

Guilherme, I understand it before asking question here. – Struve

@Struve Do you want to force access, even with the IP of your server on the block list? – Sunk

yes. I can build distributed parsing network, but I don't think it is a good idea. I think LinkedIn shouldn't block access to their content because it is censorship. It can be called in different ways but is exactly censorship. Today they are blocking networks, tomorrow they will block access for countries or specific people. It's stopping progress and has negative impact in common. – Struve

@Struve If you believe that the problem is a censorship imposed by LinkedIn, then your problem is not with the PHP and this is not an issue for StackOverflow, you should talk to a "LinkedIn Adm" need unblocking your IP (note that shared servers usually have "IP" random and therefore unblocking would not be possible). I edited my answer, please read. – Sunk

Found out the same when I was logged in on a VPN. Once I've disabled the VPN it loaded fine. – Marpet

Answer 2

<?php
header("Content-Type: text/plain");

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.linkedin.com/company/technistone-a-s-");

$header = array();
$header[] = "Host: www.linkedin.com";
$header[] = "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0";
$header[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$header[] = "Accept-Language: en-US,en;q=0.5";
$header[] = "Accept-Encoding: gzip, deflate, br";
$header[] = "Connection: keep-alive";
$header[] = "Upgrade-Insecure-Requests: 1";

curl_setopt($ch,CURLOPT_ENCODING , "gzip");
curl_setopt($ch, CURLOPT_HTTPHEADER , $header);
$my_var = curl_exec($ch);

echo $my_var;

Raven

Could you explain briefly how this should help? Code-only answers are rarely very helpful. – Shove

The only important line is the User-Agent, all others headers can be removed and it'll still work. – Beall

User agent is not enought for linked in. I tried it, and important from this headers are these $header[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8"; $header[] = "Accept-Language: en-US,en;q=0.5"; $header[] = "Accept-Encoding: gzip, deflate, br"; – Discussion

Answer 3

LinkedIn is not supporting the default encoding 'identity' , so if you set the header

'Accept-Encoding': 'gzip, deflate'

you should get the response , but you would have to decompress it.

Richards

You also need the Accept and User-Agent headers. – Barbi

I set following headers but yet not working: headers = { 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36', 'referer': 'https://linkedin.com', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'accept-encoding': 'gzip, deflate, br', 'accept-language': 'en-US,en;q=0.8', 'cache-control': 'no-cache', 'pragma': 'no-cache', 'upgrade-insecure-requests': '1', } – Tommy

Answer 4

I ran into this while doing local web development and using the LinkedIn badge feature (profile.js). I was only getting the 999 Request denied in Chrome, so I just cleared my browser cache and localStorage and it started to work again.

UPDATE - Clearing cache was just a coincidence and the issue came back. LinkedIn is having issues with their badge functionality.

I submitted a help thread to their forums. https://www.linkedin.com/help/linkedin/forum/question/714971

Sucy

Thanks, Wiped the cookie and everything is well again. – Pearce