How to make Selenium WebDriver select client certificates dynamically without visually detecting the popup

Question

I'm trying to use Java and Selenium to test a website that requires a client certificate. When I browse to my site I get a popup like the one below to select the correct certificate.

My requirements are as follows:

• Select a certificate by name
• On different versions of Windows / IE / Edge

Ideally the popup is never shown; i.e., the solution would involve invoking some API or setting some configuration to pin the correct certificate to use.

---

My own solution ideas:

• I tried a solution based on visually detecting the correct certificate using SikuliX (which does works) but I'm wondering if there is a better solution that does not rely on visually detecting the popup. Something that is less likely to fail across multiple versions of Windows and that is future proof if Microsoft decides to change the appearance of this popup.

• Another idea that I had (but I don't know how/if it is possible) is to remove all installed certificates except for one so that the popup is never shown:

  • Backup the entire store
  • Remove all client certificates that IE could use (except the one I need)
  • Do the login which would no longer require any certificate selection
  • Restore the backed up store

  Does anyone know how to do this (in Java, possibly invoking CLI commands)?

• Is it possible to start (using Selenium Java) an Internet Explorer Window that only knows the single certificate that I need?

• Is it possible in Internet Explorer to set a default certificate for a given domain?

Tsang

Did you take a look to : SSLContexts.custom().loadTrustMaterial(new File(PATH), PASSWORD, new TrustSelfSignedStrategy()).build(); ? This will return a SSLContext. If it's a good way, I can help you on this, I worked on a SSL Self Signed certificate with Selenium. See : programcreek.com/java-api-examples/… (example 2) – Scalping

@Bob How do you register this SSLContext with Selenium? Would this solution make Internet Explorer use the client certificate from the file without asking the popup? – Tsang

Answer 1

The solution we use is quite clean, easy and portable between browsers and operating systems - use proxy server that will handle SSL handshake for you.

You can set up an in-memory Man-In-The-Middle proxy server in the same JVM the tests is running, or even multiple instances on different ports, each assigned to different client certificate. Then, when creating WebDriver instance, use setProxy method suitable for your browser. Note, the browser will be presented with server certificate that is installed on the proxy itself, not the target server, so there might be some invalid certificate errors that should be suppressed in WebDriver settings. Alternatively - the proxy may simply use valid server certificate, if its key is available to you, in which case the connection is fully transparent for the test script.

One simple proxy server that offers what is required in Java is LittleProxy. Perhaps something like BrowserMob offers a more complete solution with readily available API.

An example using LittleProxy just needs a few (dozen) lines of boilerplate:

Step 1:
Extend org.littleshoot.proxy.MitmManager class with something you can plugin into your code, making use of client certificates (e.g. p12 files or PEM files). Working example available publicly at this repo.

Step 2:
Start proxy server using client certificate and server certificate of your choice:

org.littleshoot.proxy.impl.DefaultHttpProxyServer.DefaultHttpProxyServer.bootstrap()
        .withIdleConnectionTimeout(FIVE_MINUTES)
        .withName(clientCertFile.getName())
        .withPort(port)
        .withAllowLocalOnly(localConnectionOnly)
        .withManInTheMiddle(new MutualAuthenticationCapableMitmManager(
                usingPKCS12File(clientCertFile, clientCertPassword),
                usingPemKeyPair(serverKeyPair[0], serverKeyPair[1])))
        .start();

Create another proxy for each client certificate you need reusing the same port or by startnig concurrent instances.

Step 3:
Start WebDriver using the proxy. Major browsers (IE, Firefox, Chrome) support the setting in similar fashion:

org.openqa.selenium.Proxy proxy = new Proxy();
proxy.setSslProxy("127.0.0.1:5555");
proxy.setNoProxy("<-loopback>"); // overwrite the default no-proxy for localhost, 127.0.0.1

FirefoxOptions options = new FirefoxOptions();
options.setProxy(proxy);
WebDriver driver = new FirefoxDriver(options);

Step 4:
When running tests, the browser will never bother you with any certificate prompts. Profit.

If using this technique, please be extra carefull to keep the secrets secure and especially the proxy server itself unreachable to third parties. Exposing keys is never good idea outside of secure corporate network, regardless if they are real (!!!) or fake.

Dru

I've more or less followed your instructions (except I only cared about the client certificate handling so to the server certificate in MitmManager I passed an empty SslContext with the fake 'always trust' managers) and my problem is I'm always getting ERR_EMPTY_RESPONSE from the browser (whichever site i visit). I've verified with curl that the proxy is indeed listening on the specified port but from the network tab in Chrome I can't really tell what the problem is. Did you ever come across something similar? – Actiniform

I did not face anything like that. Empty response usually indicates the misconfiguration of the server, which in your case is of course the proxy, so it does not really narrow down anything, but it seems your browser facing interface of the proxy does not speak SSL at all, and the browser does not understand it. I'd suggest using the exact configuration I have put in the repo linked in the step 1, and only replacing 'client.example.com.p12' with your own file. If you do not care about the server side certificate, you should be good with the example one. – Dru

Besides that - check your antivirus/firewall and chrome policies that may have been applied by your IT department. MITM is a technique commonly used in all sorts of malware - it is not unreasonable to block it with the help of antivirus software. – Dru

Besides the server ssl, I was also using some problematic dependency versions - particularly for littleproxy - I've used 'org.littleshoot:littleproxy:1.1.2'. After switching to 'xyz.rogfam:littleproxy:2.0.9' (same as in your repo), configuring server ssl as in your example everything works as desired. Your answer was really helpful. Thank you :) – Actiniform

None of the browser drivers work very well with client certs in selenium. For example, I can get certs to work with chrome when in head mode, but not in headless mode. The approach from @Dru is so much better than trying to use native selenium cert support. I was able to quickly get this working. Now I can use any browser, head mode or headless mode, and it works perfectly. One small note - I ended up using BrowserMob instead of LittleProxy, but the overall approach worked exactly as described. I only wished I had found this answer sooner!! – Laellaertes

Answer 2

I can't say for sure, but I wouldn't be surprised if this was impossible with selenium.

This question doesn't fill me with hope: Selenium can't deal with Confirm Certificate popup in IE

The further you get away from the webpage and towards native browser and then OS controls, the more impotent Selenium becomes.

As the only answer to the linked question states: you can disable the popup in the browser, which may or may not be a workable solution.

If you don't want to go down that route, people most often fall back to a Java Robot to handle things that the selenium driver can't do e.g. interact with the print dialogs and other such controls.

See mouseMove, mousePress, mouseRelease You'd call each of these, providing the button position to mouseMove. https://docs.oracle.com/javase/7/docs/api/java/awt/Robot.html

Murdock

I was afraid of that. The Robot solution is similar to my SikuliX solution and is a thing I wanted to avoid. I was hoping java had a way of manipulating the certificate store in a clean way that was able to restore the initial state, or ideally have some IE configuration that automatically took the desired certificate. – Tsang