Blacklist URLs with headless Chrome

Question

I'm trying to block URLs in my specs, achieving something like I had when using capybara_webkit:

Capybara::Webkit.configure do |config|
  config.block_url("*google*")
  config.allow_url('*my_website.com')
end

After reading this article, I tried to do something like:

require 'webmock/rspec'

module WebmockConfig
  def self.default_disabled_urls
    [
      '*google*'
    ]
  end
end

WebMock.disable_net_connect!(allow_localhost: true)
WebMock.disable_net_connect!(allow: WebmockConfig.default_disabled_urls)

but I'm getting

Real HTTP connections are disabled. Unregistered request: POST http://127.0.0.1/session

even if that should be solved by WebMock.disable_net_connect!(allow_localhost: true).

When running the specs without WebMock.disable_net_connect!(allow: WebmockConfig.default_disabled_urls), everything is working fine.

Vaporetto

Answer 1

The capybara-webkit white/blacklisting affects the requests made by the browser, whereas WebMock can only affect requests made by your app. This means WebMock is useless for what you want since it wouldn't actually stop your browser from loading anything from google, etc. To do that while using the selenium driver you need to use a programmable proxy like puffing-billy which will allow you to customize the responses for any matching requests the browser makes.

To configure a driver using headless chrome and puffing_billy you could do something like

Capybara.register_driver :headless_chrome do |app|
 browser_options = ::Selenium::WebDriver::Chrome::Options.new
 browser_options.headless!
 browser_options.add_argument("--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}")
 Capybara::Selenium::Driver.new(app, browser: :chrome, options: browser_options)
end

Whether or not you need any other options is dependent on your system config, etc but you should be able to tell by looking at your current driver registration.

Gar

Hey Thomas. From what I can read here github.com/oesmith/puffing-billy#setup-for-capybara it's not possible to use my actual driver (headless_chrome) with puffing-billy fucntionalities? – Vaporetto

@fabersky It can be used with any Capybara driver that supports a proxy - you just need to register your own custom driver configuration - github.com/oesmith/… and github.com/oesmith/puffing-billy/blob/master/lib/billy/browsers/… . I've added a configuration that could be used with headless chrome to my answer – Gar

Answer 2

The allow_localhost: true settings are overwritten by allow: WebmockConfig.default_disabled_urls you have to call WebMock.disable_net_connect! once with both settings or by adding 'localhost', '127.0.0.1' entries into self.default_disabled_urls

Gunnel

Hey @Gunnel I know how to add multiple urls in WebMock.disable_net_connect!(allow: by adding an array of URLs, but how can I combine allow_localhost and allow? – Vaporetto

@fabersky You can try WebMock.disable_net_connect!(allow: [...], allow_localhost: true) and see if it works or not. If not, manually add localhost ips into allow array – Gunnel

anyway, even if i achieve that in this way, I don't think it would be the expected result. What I want to achieve is just blocking requests from blacklisted URLs. This way, my specs will probably fail once they will get a request from one of that URLs. – Vaporetto