Ruby: Phantom.js blocked on specific site?

Question

I am using capybara poltergeist to automate a small script on tumblr.com

My script works fine with my chrome driver.. And my poltergeist driver loads all other websites just fine, but for some reason throws a Capybara::Poltergeist::StatusFailError when I try to load tumblr.

Steps for reproduction:

$ brew install phantomjs
$ gem install capybara
$ gem install poltergeist
$ gem install selenium-webdriver
$ irb


require 'capybara/poltergeist'

module Drivers
  class Poltergeist < Capybara::Poltergeist::Driver
    def needs_server?
      false
    end
  end
end

Capybara.register_driver :poltergeist_errorless do |app|
  Drivers::Poltergeist.new(app, js_errors: false, timeout: 10000, phantomjs_options: ['--load-images=no', '--ignore-ssl-errors=yes'])
end

session = Capybara::Session.new(:poltergeist_errorless)
session.visit('https://google.com') # This works fine
session.visit('https://tumblr.com') # This does not work?

I tried to set all of my headers to look my google chrome's request, but that also does not seem to fix it. Does anyone have any suggestions?

Alongside

Stumped. Pretty hard to debug poltergeist but can't figure out why visiting tumblr results in a {'status' => 'fail' }. Works fine in selenium... – Eward

Thanks for looking into it, I really appreciate your time. I can't understand why this is such a hard problem to solve? – Alongside

Answer 1

The problem is related to phantomjs SSL handshake fail. You can take my gist and run with phantomjs, you will see:

[cut]
= onResourceError()
  - unable to load url: "https://www.tumblr.com/"
  - error code: 6, description: SSL handshake failed
= onResourceReceived()
  id: 3, stage: "end", response: {"contentType":null,"headers":[],"id":3,"redirectURL":null,"stage":"end","status":null,"statusText":null,"time":"2014-09-16T12:06:05.547Z","url":"https://www.tumblr.com/"}
= onLoadFinished()
  status: fail
DONE WITH  fail WebPage(name = "WebPage")

Checking a little bit a workaround is to use --ssl-protocol=any in phantom, so your code will become:

Capybara.register_driver :poltergeist_errorless do |app|
  Drivers::Poltergeist.new(app, js_errors: false, timeout: 10000, phantomjs_options: ['--load-images=no', '--ignore-ssl-errors=yes', '--ssl-protocol=any'])
end

To work.

References:

• debugging phantom js page
• ssl-protocol-any

Devise

I still face this problem on my laptop, but on production. It's a random failure. – Discoid

@Discoid what do you mean by "on production"? Have you initialized Poltergeist with '--ignore-ssl-errors=yes' and '--ssl-protocol=any'? – Devise

I mean it's a random failure. Sometimes work on my laptop but not on production and vice versa. Yeah, I initialised with those options already, and also check with ps aux | grep phantomjs. There is an issue on poltergeist too, github.com/EFForg/phantom-of-the-capitol/issues/47. – Discoid