How to Conceal WebDriver in Geckodriver from BotD in Java?

Question

I followed this post on Stackoverflow to disable Firefox WebDriver detection.

Launch Geckodriver:

System.setProperty("webdriver.gecko.driver", geckdriverExecutableFilePath);



File firefoxProfileFile = new File(fullPathOfFirefoxInstallationFolder);



FirefoxProfile firefoxProfile = null;

        
 try {
    firefoxProfile = new FirefoxProfile(firefoxProfileFile);
     } catch (Exception e) {

    e.printStackTrace();
     }

I disabled WebDriver:

WebDriver Disabled

FirefoxOptions firefoxOptions = new FirefoxOptions();

firefoxOptions.setProfile(firefoxProfile);

// Disables WebRTC
firefoxProfile.setPreference("media.peerconnection.enabled", false);

I disabled Automation Extensions:

Automation Extension Disabled

// Disables Automation Extension
firefoxProfile.setPreference("useAutomationExtension", false);

I added Proxy:

    DesiredCapabilities dc = DesiredCapabilities.firefox();
    Proxy proxy = new Proxy();
    proxy.setHttpProxy(ipAddress + ":" + port);
    proxy.setFtpProxy(ipAddress + ":" + port);
    proxy.setSslProxy(ipAddress + ":" + port);

   dc.setCapability(CapabilityType.PROXY, proxy);




   firefoxOptions.merge(dc);

   driver = new FirefoxDriver(firefoxOptions);

Yet BotD still detects my browser as being controlled by automation tool.

BotD Detection

How can I solve this?

Conk

Update the question with your code trials. – Coil

Would you like to evaluate an answer in Python? – Coil

@DebanjanB python is OK IF the code can be easily ported to Java. Meaning, if code is using some special function that is only available in python then it won't be of use to me as my entire code base is in Java. Thanks in advance for your response! You seem to be the most knowledgeable user on this forum regarding Selenium :) – Conk

Answer 1

When using Selenium driven GeckoDriver initiated firefox Browsing Context

The webdriver-active flag is set to true when the user agent is under remote control. It is initially false.

where, webdriver returns true if webdriver-active flag is set, false otherwise.

As:

navigator.webdriver Defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, for example so that alternate code paths can be triggered during automation.

Further @whimboo in his comments confirmed:

This implementation have to be conformant to this requirement. As such we will not provide a way to circumvent that.

---

Conclusion

So, the bottom line is:

Selenium identifies itself

and there is no way to conceal the fact that the browser is WebDriver driven.

---

Recommendations

However some pundits have suggested some different approaches which can conceal the fact that the Mozilla Firefox browser is WebDriver controled through the usage of Firefox Profiles and Proxies as follows:

selenium4 compatible python code

from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options

profile_path = r'C:\Users\Admin\AppData\Roaming\Mozilla\Firefox\Profiles\s8543x41.default-release'
options=Options()
options.set_preference('profile', profile_path)
options.set_preference('network.proxy.type', 1)
options.set_preference('network.proxy.socks', '127.0.0.1')
options.set_preference('network.proxy.socks_port', 9050)
options.set_preference('network.proxy.socks_remote_dns', False)
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = Firefox(service=service, options=options)
driver.get("https://www.google.com")
driver.quit()

---

Potential Solution

A potential solution would be to use the tor browser as follows:

selenium4 compatible python code

from selenium.webdriver import Firefox  
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
import os

torexe = os.popen(r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile_path = r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default'
firefox_options=Options()
firefox_options.set_preference('profile', profile_path)
firefox_options.set_preference('network.proxy.type', 1)
firefox_options.set_preference('network.proxy.socks', '127.0.0.1')
firefox_options.set_preference('network.proxy.socks_port', 9050)
firefox_options.set_preference("network.proxy.socks_remote_dns", False)
firefox_options.binary_location = r'C:\Users\username\Desktop\Tor Browser\Browser\firefox.exe'
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = webdriver.Firefox(service=service, options=firefox_options)
driver.get("https://www.tiktok.com/")

---

References

You can find a couple of relevant detailed discussions in

• How to initiate a Tor Browser 9.5 which uses the default Firefox to 68.9.0esr using GeckoDriver and Selenium through Python
• How to connect to Tor browser using Python
• How to use Tor with Chrome browser through Selenium

Coil

Upvoted for adding solution with TOR and for thinking out of the box.. – Hotbox

+1 thoroughly impressed with your suggestion to bundle TOR with Firefox. A user on SO suggested that it is possible modify chromedriver source code to remove any bot-identifying attributes. Is it possible to achieve a similar outcome w/ Firefox by either modifying Geckodriver or Firefox WebDriver source code? We can then remove bot-identification without needing to bundle TOR with Firefox? You mentioned Selenium identifies itself but surely we can modify source code to remove identification similar to how it's done in chrome driver? – Interfere

@BradfordGriggs If you can raise a question I'll be happy to post an answer. – Coil

@DebanjanB I awarded your response with +50 bounty. I raised another question about removing bot identifying features from Geckodriver / WebDriver source code. I will add another +50 bounty if you can think of a solution to that question! – Interfere

Answer 2

Because my entire code base (thousands of lines of code) is written for Firefox. In the future I hope to add support for Chrome but in either case I do not want to give up using Firefox just because I can't solve one problem. I am hoping to find a way around this.. There must be a way..

Because the OP really wants an answer, I'm going to give an answer.

• Install Violentmonkey addon to your geckodriver
• Install this script:

// ==UserScript==
// @name            Webdriver Get Rekt
// @author          noname
// @namespace       http://www.example.url/to/your-web-site/
// @description     Put a good description in here
// @license         Creative Commons Attribution License
// @version            0.1
// @include         http*://*/*
// @run-at           document-start
// @compatible      Greasemonkey
// ==/UserScript==

Object.defineProperty(navigator, 'webdriver', {get: () => undefined})

Should be able to override navigator.webdriver BEFORE page load, which eliminate driver detection.

Amblyopia

Answer 3

BotD detects you because you do not override navigator.webdriver attribute.

I was able to override it with this code:

((JavascriptExecutor)driver).executeScript("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})");

Re-run your code with this line after driver.get("BotD url") and click on 'Start detect' on the BotD page.

It will no longer show that webdriver is detected.

I understand you are looking for a way to make it work before the initial page load.

But here are 2 things to consider:

1. Webdriver developers want their tool to be detected by browsers.
2. Gecko driver developers are not going to implement an option to disable navigator.webdriver attribute. (This is the official reply from gecko developer.)

Iloilo

I understand you are looking for a way to make it work before the initial page load Correct. Overriding attribute after the page already loaded is of no value as the browser will already be detected as a bot. I know gecko driver developers did not natively incorporate this option but I believe it's still possible to achieve this if we tinker with the underlying code - similar to how it can be achieved with Chrome. – Conk

So you are looking for some advice on how to hack the driver? Why you don't want to switch to Chrome if it has, what you need? – Iloilo

Because my entire code base (thousands of lines of code) is written for Firefox. In the future I hope to add support for Chrome but in either case I do not want to give up using Firefox just because I can't solve one problem. I am hoping to find a way around this.. There must be a way.. – Conk