PhantomJS failing to open HTTPS site

Question

I'm using the following code based on loadspeed.js example to open up a https:// site which requires http server authentication as well.

var page = require('webpage').create(), system = require('system'), t, address;

page.settings.userName = 'myusername';
page.settings.password = 'mypassword';

if (system.args.length === 1) {
    console.log('Usage: scrape.js <some URL>');
    phantom.exit();
} else {
    t = Date.now();
    address = system.args[1];
    page.open(address, function (status) {
        if (status !== 'success') {
            console.log('FAIL to load the address');
        } else {
            t = Date.now() - t;
            console.log('Page title is ' + page.evaluate(function () {
                return document.title;
            }));
            console.log('Loading time ' + t + ' msec');
        }
        phantom.exit();
    });
}  

Its failing to load the page all the time. What could be wrong here? Are secured sites to be handled any differently? The site can be accessed successfully from browser though.

I'm just starting with Phantom right now and find it too good to stop playing around even though i'm not moving forward with this issue.

Henrion

Answer 1

I tried Fred's and Cameron Tinker's answers, but only --ssl-protocol=any option seem to help me:

phantomjs --ssl-protocol=any test.js

Also I think it should be way safer to use --ssl-protocol=any as you still are using encryption, but --ignore-ssl-errors=true will ignore (duh) all ssl errors, including malicious ones.

Hoopen

i had to use these 3 arguments and it resolved: "--web-security=false", "--ssl-protocol=any", "--ignore-ssl-errors=true". webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true', "--web-security=false", "--ssl-protocol=any"]) – Heilungkiang

Answer 2

The problem is most likely due to SSL certificate errors. If you start phantomjs with the --ignore-ssl-errors=yes option, it should proceed to load the page as it would if there were no SSL errors:

phantomjs --ignore-ssl-errors=yes [phantomOptions] script.js [scriptOptions]

I've seen a few websites having problems with incorrectly implementing their SSL certificates or they've expired, etc. A complete list of command line options for phantomjs is available here: http://phantomjs.org/api/command-line.html.

Morality

Thank you, this resolved my problem as well. My case, I didn't get Cert error on browser so it was very confusing but I did curl verbose and noticed that one that didn't work was using Wildcard Cert (ie: CN=*.example.com). It would be nice if phantomjs can return verbose reason as to why this is failed. – Beadroll

Holy monkeys. I wish the SSL error were explained, only stack overflow and the pink unicorn helped me track down the root cause. Thanks guys – Equi

Glad this is helping people. Perhaps PhantomJS will need an update to provide verbose SSL errors with another command line argument. I know that SSL error information is available in Qt, but most times people just suppress the errors without handling them explicitly. – Morality

I know I'm late to the game, but I found that adding the option after the script name didn't work - you need to call it in order: phantomjs --ignore-ssl-errors=yes script.js – Dogear

Yes, the --ignore-ssl-errors=yes option should come before the script name. Thank you for pointing this out. – Morality

Does it give you any error message? I haven't tried this method on phantomjs 1.9.7, but it should work since the --ignore-ssl-errors=yes option is still available. You might also change the SSL protocol as suggested by Fred's answer below. – Morality

I tried this and it didn't work but then I realized I was putting the --ignore-ssl-errors=yes option after the script argument (script.js). The order matters! phantomjs --ignore-ssl-errors=yes script.js. It works like a charm when you get the order right. – Glasgow

My example command shows that phantom options must be first. Then you can specify the script and the script options. I'm glad you got it working though! – Morality

I'd love it if one the more specific flags suggested (like --ssl-protocol=tlsv1) would be the RIGHT solution in all cases, but it seems like this one is (currently) the only fix for hosts with the SNI extension. This issue suggests that PhantomJS currently doesn't support those, so --ignore-ssl-errors=yes is the only workable fix suggested here. – Inadvertency

Answer 3

Note that as of 2014-10-16, PhantomJS defaults to using SSLv3 to open HTTPS connections. With the POODLE vulnerability recently announced, many servers are disabling SSLv3 support.

To get around that, you should be able to run PhantomJS with:

phantomjs --ssl-protocol=tlsv1

Hopefully, PhantomJS will be updated soon to make TLSv1 the default instead of SSLv3.

Chanty

Version 1.9.8 switches the default to TLSv1: github.com/ariya/phantomjs/issues/12670 – Faltboat

Fixed it for me, SSLv3 was disabled on the server – Deweese

Note that updating to PhantomJS 1.9.8 leads to a new bug. – Sludge

This should be the accepted answer. Disabling SSL or allowing any protocol are not good solutions in my opinion. Thanks for sharing. – Cheslie

Didn't work by itself for me. I needed to also add --ignore-ssl-errors. But I can't see anything wrong with the certificate -- signed by a valid CA, not expired and matches hostname. (Chrome displays green lock icon without complaint.) – Mccants

How can I get more verbose debugging out of SSL handshake problems? – Mccants

This wasn't enough to get my tests working, I had to add --ignore-ssl-errors=true to it too. – Herculaneum

This worked for me, but I did not have to set --ignore-ssl-errors=true. I was hitting https://*.herokuapp.com – Categorical

Answer 4

experienced same issue...
--ignore-ssl-errors=yes was not enough to fix it for me, had to do two more things:
1) change user-agent
2) tried all ssl-protocols, the only one that worked was tlsv1 for the page in question
Hope this helps...

Manzanilla

Changing ssl-protocols to tlsv1 worked for me as well, thanks! – Genitive

+1 Fixed my problem, lucky you are not here with me as I would kiss you right now lol, cheers – Dylane

This is exactly what I needed. Thank you very much! Instead of tlsv1 I used "any" and this works as well. (seems a bit safer should I visit other ssl websites. – Pettitoes

change user-agent to what? – Supplication

Answer 5

I experienced the same problem (casperjs 1.1.0-beta3/phantomjs 1.9.7). Using --ignore-ssl-errors=yes and --ssl-protocol=tlsv1 solved it. Using only one of the options did not solve it for me.

Ahriman

This trick fixed my problem on codeship CI, running a phantomjs + behat suite (after ~10 hours of debugging and trying out things) – Cornea

yeah. after some silly hours your comment made my day! – Insoluble

FIXED my issues!!! – Exercise

Answer 6

If someone is using Phantomjs with Sahi the --ignore-ssl-errors option needs to go in your browser_types.xml file. It worked for me.

<browserType>
    <name>phantomjs</name>
    <displayName>PhantomJS</displayName>
    <icon>safari.png</icon>
    <path>/usr/local/Cellar/phantomjs/1.9.2/bin/phantomjs</path>
    <options>--ignore-ssl-errors=yes --debug=yes --proxy=localhost:9999 /usr/local/Cellar/phantomjs/phantom-sahi.js</options>
    <processName>"PhantomJS"</processName>
    <capacity>100</capacity>
    <force>true</force>
</browserType>

Spartacus

Answer 7

I was receiving

Error creating SSL context" from phantomJS (running on CentOS 6.6)

Building from source fixed it for me. Don't forget to use the phantomjs that you built. (instead of the /usr/local/bin/phantomjs if you have it)

sudo yum -y install gcc gcc-c++ make flex bison gperf ruby openssl-devel freetype-devel fontconfig-devel libicu-devel sqlite-devel libpng-devel libjpeg-devel
git clone git://github.com/ariya/phantomjs.git
cd phantomjs
git checkout 2.0
./build.sh
cd bin/
./phantomjs <your JS file>

Steeplejack

Answer 8

What about shebang?

If you're using shebang to execute phantomjs scripts, use the following shebang line

#!/usr/bin/phantomjs --ignore-ssl-errors=yes
    
var system = require('system');
var webpage = require('webpage');

// ... rest of your script

Use any of the above answers. i personally like --ignore-ssl-errors=yes since it's irrelevant to validate my loopback web servers' self-signed certificate.

Indulgent

Answer 9

None of the other answers here helped me; it may be that the specific site(s) I was working with were too picky with their HTTP headers. This is what worked:

var page = webpage.create();
page.customHeaders = {
    "Connection": "keep-alive"
};

I found out that PhantomJS was using "Keep-Alive" (capitalized), and the connection was not being kept alive. :)

Yeld

Answer 10

I was getting SSL Handshake Failed yesterday. I tried many combinations of phantomJS options (--ignore-ssl-errors=yes etc.), but none of them worked.

Upgrading to phantomJS 2.1.1 fixed it.

I used the phantomJS installation instructions at https://gist.github.com/julionc/7476620, changing the phantomJS version to 2.1.1.

Rolan

Answer 11

On the machine you are trying to run phantomjs on to connect to a remote server, run "openssl ciphers." Copy and paste the ciphers listed into the --ssl-ciphers="" command line option. This tells the connecting web server which ciphers are available to use to communicate with your client. If you don't set the ones available on your own machine, it can use any cipher your machine does not understand that the default modern browsers do that are used for the default setting.

Cogitation

Answer 12

phantomjs --web-security=false --ignore-ssl-errors=true scripts.js

Luxuriate

Answer 13

The only thing that worked for me was upping phantomjs from 1.9x to 2.x ;)

Chagrin