Can a website detect when you are using Selenium with chromedriver?

Asked 20/10, 2015 at 0:8 Answered 11/5, 2023 at 19:23

Solved javascript python google-chrome selenium selenium-chromedriver

628

I've been testing out Selenium with Chromedriver and I noticed that some pages can detect that you're using Selenium even though there's no automation at all. Even when I'm just browsing manually just using Chrome through Selenium and Xephyr I often get a page saying that suspicious activity was detected. I've checked my user agent, and my browser fingerprint, and they are all exactly identical to the normal Chrome browser.

When I browse to these sites in normal Chrome everything works fine, but the moment I use Selenium I'm detected.

In theory, chromedriver and Chrome should look literally exactly the same to any web server, but somehow they can detect it.

If you want some test code try out this:

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=1, size=(1600, 902))
display.start()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--disable-extensions')
chrome_options.add_argument('--profile-directory=Default')
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-plugins-discovery");
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.delete_all_cookies()
driver.set_window_size(800,800)
driver.set_window_position(0,0)
print 'arguments done'
driver.get('http://stubhub.com')

If you browse around stubhub you'll get redirected and 'blocked' within one or two requests. I've been investigating this and I can't figure out how they can tell that a user is using Selenium.

How do they do it?

I installed the Selenium IDE plugin in Firefox and I got banned when I went to stubhub.com in the normal Firefox browser with only the additional plugin.

When I use Fiddler to view the HTTP requests being sent back and forth I've noticed that the 'fake browser's' requests often have 'no-cache' in the response header.

Results like this Is there a way to detect that I'm in a Selenium Webdriver page from JavaScript? suggest that there should be no way to detect when you are using a webdriver. But this evidence suggests otherwise.

The site uploads a fingerprint to their servers, but I checked and the fingerprint of Selenium is identical to the fingerprint when using Chrome.

This is one of the fingerprint payloads that they send to their servers:

{"appName":"Netscape","platform":"Linuxx86_64","cookies":1,"syslang":"en-US","userlang":"en-
US","cpu":"","productSub":"20030107","setTimeout":1,"setInterval":1,"plugins":
{"0":"ChromePDFViewer","1":"ShockwaveFlash","2":"WidevineContentDecryptionMo
dule","3":"NativeClient","4":"ChromePDFViewer"},"mimeTypes":
{"0":"application/pdf","1":"ShockwaveFlashapplication/x-shockwave-
flash","2":"FutureSplashPlayerapplication/futuresplash","3":"WidevineContent
DecryptionModuleapplication/x-ppapi-widevine-
cdm","4":"NativeClientExecutableapplication/x-
nacl","5":"PortableNativeClientExecutableapplication/x-
pnacl","6":"PortableDocumentFormatapplication/x-google-chrome-
pdf"},"screen":{"width":1600,"height":900,"colorDepth":24},"fonts":
{"0":"monospace","1":"DejaVuSerif","2":"Georgia","3":"DejaVuSans","4":"Trebu
chetMS","5":"Verdana","6":"AndaleMono","7":"DejaVuSansMono","8":"LiberationM
ono","9":"NimbusMonoL","10":"CourierNew","11":"Courier"}}

It's identical in Selenium and in Chrome.

VPNs work for a single use, but they get detected after I load the first page. Clearly some JavaScript code is being run to detect Selenium.

Magavern answered 20/10, 2015 at 0:8 Comment(19)

I would suggest using a local proxy to take a look at the web traffic going from your request to the server and back. You should be able to tell from there. fiddler, burp, ZED proxy any of them will do the trick. I prefer Burp – Schizogenesis 20/10, 2015 at 0:43

I did this and there's no difference between the requests at all. How are they possibly doing this? – Magavern 21/10, 2015 at 0:41

@RyanWeinstein: It is not traffic. My guess is that Selenium needs to expose some JavaScript hooks which can be detected on the client-side JavaScript. – Augmentation 21/10, 2015 at 19:4

Or if it is traffic then it is a traffic pattern.... you are browsing pages too fast. – Augmentation 21/10, 2015 at 19:6

I'm not browsing too fast. I only load a single page and I navigate through it normally using my mouse and keyboard. Also it doesn't make sense that Selenium needs to expose hooks, because its literally running chrome.exe. It just runs normal chrome and allows you to get data from it. Any other ideas? I was thinking maybe it has something to do with cookies. This is driving me crazy. – Magavern 21/10, 2015 at 19:12

Mikko's idea sounds pretty plausible to me. Selenium needs to be able to detect events so that it can respond to them. So it must be injecting some javascript code which is detected even when you're not actually automating anything. – Ecstatics 21/10, 2015 at 19:38

Its possible, but if that's the case then it would go against everything that anyone knows about Selenium. There is nowhere online where anyone knows a way to detect if a user is using chromedriver. So if there's a way to detect it using frontend javascript then I want to know what it is. #3614972 – Magavern 21/10, 2015 at 19:43

Can you post the request headers (except the sensitive info, of course)? That's one... There's also probably some user interaction that the site is expecting which is not happening. – Briarroot 22/10, 2015 at 5:33

This site uses distill bot detection technology and delivers content using akamaitechnologies.com CDN from diffrent ips e.g. 95.100.59.245 , 104.70.243.66 , 23.202.161.241 – Tho 22/10, 2015 at 10:12

Same happens to 411.com – Tho 22/10, 2015 at 13:31

I am able to hit stubhub.com and 411.com with chromedriver/selenium and your listed settings. Reading on distil from SIslam it appears as though they are keeping a fingerprint in memory, which is probably triggered based on behaviour. – Joannejoannes 23/10, 2015 at 20:18

The fingerprint contains some browser information mostly. You say you are able to go to stubhub and navigate around in chromedriver? Are you sure you're using exactly my settings? I can't navigate around using chromedriver so there must be some difference. @Joannejoannes – Magavern 23/10, 2015 at 20:43

I am experiencing the same issue with Selenium and the firefox driver. The interesting thing to note is I am running Selenium in a VMWare Workstation Virtual Machine that is accessing the internet through a NAT. The host machine is able to access stubhub, while the VM is unable to access when using Selenium, or even the browser instance Selenium launched. I had the VM Browser instance Blocked and stubhub still recognizes the machine and has it blocked. So it must be performing a fingerprint of the browser and machine in some manner. – Anselmo 23/10, 2015 at 21:34

@RyanWeinstein Yes fp is generated from about 40 variables that encompasses client and network info! What i read in distill! – Tho 25/10, 2015 at 13:16

Without using pyvirtualdisplay, I can connect to stubhub.com and browse around just fine; I don't get blocked – Septuagenarian 27/10, 2015 at 21:0

@cwa: Can you post your code? Noone else here seems to have gotten it to work. – Rn 19/12, 2015 at 13:46

Did you try to connect via remote debugging instead of starting a new instance ? sites.google.com/a/chromium.org/chromedriver/help/… The webdriver will be invisible for the website. – Novick 28/9, 2016 at 15:14

Case of me it is working. I can open stubhub.com with Selenium and navigate items later. – Ridgley 13/6, 2017 at 0:13

Relevant Q&A here with working solutions: #53040051 – Coffelt 2/8, 2022 at 20:26

186

Replacing `cdc_` string

You can use Vim or Perl to replace the cdc_ string in chromedriver. See the answer by @Erti-Chris Eelmaa to learn more about that string and how it's a detection point.

Using Vim or Perl prevents you from having to recompile source code or use a hex editor.

Make sure to make a copy of the original chromedriver before attempting to edit it.

Our goal is to alter the cdc_ string, which looks something like $cdc_lasutopfhvcZLmcfl.

The methods below were tested on chromedriver version 2.41.578706.

Using Vim

vim -b /path/to/chromedriver

After running the line above, you'll probably see a bunch of gibberish. Do the following:

Replace all instances of cdc_ with dog_ by typing :%s/cdc_/dog_/g.
- dog_ is just an example. You can choose anything as long as it has the same amount of characters as the search string (e.g., cdc_), otherwise the chromedriver will fail.
To save the changes and quit, type :wq! and press return.
- If you need to quit without saving changes, type :q! and press return.

The -b option tells vim upfront to open the file as a binary, so it won't mess with things like (missing) line endings (especially at the end of the file).

Using Perl

The line below replaces all cdc_ occurrences with dog_. Credit to Vic Seedoubleyew:

perl -pi -e 's/cdc_/dog_/g' /path/to/chromedriver

Make sure that the replacement string (e.g., dog_) has the same number of characters as the search string (e.g., cdc_), otherwise the chromedriver will fail.

Wrapping Up

To verify that all occurrences of cdc_ were replaced:

grep "cdc_" /path/to/chromedriver

If no output was returned, the replacement was successful.

Go to the altered chromedriver and double click on it. A terminal window should open up. If you don't see killed in the output, you've successfully altered the driver.

Make sure that the name of the altered chromedriver binary is chromedriver, and that the original binary is either moved from its original location or renamed.

My Experience With This Method

I was previously being detected on a website while trying to log in, but after replacing cdc_ with an equal sized string, I was able to log in. Like others have said though, if you've already been detected, you might get blocked for a plethora of other reasons even after using this method. So you may have to try accessing the site that was detecting you using a VPN, different network, etc.

Enlistee answered 31/8, 2018 at 3:49 Comment(25)

@LekaBaper Thanks for the heads up. The chromedriver version that I used was version 2.41.578706. – Enlistee 12/11, 2018 at 4:47

Did not worked even when I used this chromedriver.exe modification on new physical computer on different network. – Amaris 24/6, 2019 at 4:40

it's give an error says, this version cannot work in this computer :( – Playful 21/7, 2019 at 22:59

@Enlistee Is there any undetectable open-source fork which you are aware of ? – Malkamalkah 27/9, 2019 at 8:59

@Malkamalkah No, not that I’m aware of. – Enlistee 27/9, 2019 at 15:39

Note that the chromedriver people have declared this issue won't-fix, so you can expect to have to use a fork or edit the binary for the indefinite future. bugs.chromium.org/p/chromedriver/issues/detail?id=3220 – Bala 15/11, 2019 at 15:37

hi, @colossatr0n. I've tried with your method and also change my ip, but it still not worked as to this site: authstl.alipay.com/login/index.htm, is there any new magic that i could use. – Monotint 29/12, 2019 at 2:2

TLDR; open binary in hex editor, change string starting with $cdc to some other string of same length, then save and run modified binary. – Triphylite 15/6, 2020 at 5:32

I have seven instances of cdc_ in my Chromedriver. Do I need to replace all or just one? Your Vim/perl methods differs on this point. – Shrievalty 4/12, 2020 at 11:38

@Shrievalty Replace all of them. I'll update the Vim instructions. – Enlistee 8/12, 2020 at 23:35

After modification, I cannot open the chrome driver. Mac shouted that the original file has been altered that's why for security reasons, it did not start. – Paredes 15/5, 2021 at 0:56

for weeks now, I've been finding solutions on why chromedriver keeps giving me 403 error and such. But this where I found my heaven. Thanks for this solution on just editing the chromdriver source and everything works for me. – Photima 1/11, 2021 at 20:27

sed -b "s/cdc_/dog_/g" chromedriver.exe > chromedriver_patched.exe For those with GNU sed on Windows. – Inodorous 30/11, 2021 at 16:20

this breaks my chromedriver.. once I do perl -pi -e 's/cdc_/dog_/g' it breaks the chromedriver and I must restore it using perl -pi -e 's/dog_/cdc_/g' to make it running again.. I tried doing perl -pi -e 's/$cdc_/$dog_/g' but I'm not sure what the effect of it.. – Flexor 26/2, 2022 at 12:23

what about geckodriver? – Sulphuric 10/6, 2022 at 18:11

This was THE ONLY (and I mean, really, literally, the only) method that worked for me. Thank you so much. – Laspisa 26/6, 2022 at 13:21

@JugertMucoimaj No, this method is specific for Chrome. GeckoDriver is a completely different binary. – Laspisa 26/6, 2022 at 13:22

@Laspisa you know sum about geckodriver – Sulphuric 27/6, 2022 at 12:26

when i use vim in Mac, it shows: E486: Pattern not found: cdc_ – Smallage 19/7, 2022 at 15:8

Note that modified chromedriver won't run on M1 Mac. Here's a workaround: https://mcmap.net/q/65248/-chromedriver-killed-on-apple-silicon-when-cdc_-modified – Electrograph 31/8, 2022 at 1:46

My Chromedriver doesn't have the cdc string – Comber 3/9, 2022 at 14:46

Have an example video to do that? I am new, I don't understand to do that. – Sheepdip 11/9, 2022 at 9:44

Seems like a lot of manual work, but still it wouldn't cover all eventualities according to other posts in this thread. Wouldn't it be more effective and time-efficient to employ a full-blown package such as undetected-chromedriver or selenium-stealth? – Akkadian 30/10, 2022 at 10:24

Wont work for sites using datadom datadome.co/bot-management-protection/… – Intranuclear 23/11, 2022 at 19:3

Just as a reminder for beginners like me: 1. Remember to wrap the path in double quotes (") in both vim or pearl when modifying, with the administrator mode on. 2. cmd does not have an original command like grep, use findstr instead. – Tartrazine 26/1, 2023 at 18:2

263

Basically, the way the Selenium detection works, is that they test for predefined JavaScript variables which appear when running with Selenium. The bot detection scripts usually look anything containing word "selenium" / "webdriver" in any of the variables (on window object), and also document variables called $cdc_ and $wdc_. Of course, all of this depends on which browser you are on. All the different browsers expose different things.

For me, I used Chrome, so, all that I had to do was to ensure that $cdc_ didn't exist anymore as a document variable, and voilà (download chromedriver source code, modify chromedriver and re-compile $cdc_ under different name.)

This is the function I modified in chromedriver:

File call_function.js:

function getPageCache(opt_doc) {
  var doc = opt_doc || document;
  //var key = '$cdc_asdjflasutopfhvcZLmcfl_';
  var key = 'randomblabla_';
  if (!(key in doc))
    doc[key] = new Cache();
  return doc[key];
}

(Note the comment. All I did I turned $cdc_ to randomblabla_.)

Here is pseudocode which demonstrates some of the techniques that bot networks might use:

runBotDetection = function () {
    var documentDetectionKeys = [
        "__webdriver_evaluate",
        "__selenium_evaluate",
        "__webdriver_script_function",
        "__webdriver_script_func",
        "__webdriver_script_fn",
        "__fxdriver_evaluate",
        "__driver_unwrapped",
        "__webdriver_unwrapped",
        "__driver_evaluate",
        "__selenium_unwrapped",
        "__fxdriver_unwrapped",
    ];

    var windowDetectionKeys = [
        "_phantom",
        "__nightmare",
        "_selenium",
        "callPhantom",
        "callSelenium",
        "_Selenium_IDE_Recorder",
    ];

    for (const windowDetectionKey in windowDetectionKeys) {
        const windowDetectionKeyValue = windowDetectionKeys[windowDetectionKey];
        if (window[windowDetectionKeyValue]) {
            return true;
        }
    };
    for (const documentDetectionKey in documentDetectionKeys) {
        const documentDetectionKeyValue = documentDetectionKeys[documentDetectionKey];
        if (window['document'][documentDetectionKeyValue]) {
            return true;
        }
    };

    for (const documentKey in window['document']) {
        if (documentKey.match(/\$[a-z]dc_/) && window['document'][documentKey]['cache_']) {
            return true;
        }
    }

    if (window['external'] && window['external'].toString() && (window['external'].toString()['indexOf']('Sequentum') != -1)) return true;

    if (window['document']['documentElement']['getAttribute']('selenium')) return true;
    if (window['document']['documentElement']['getAttribute']('webdriver')) return true;
    if (window['document']['documentElement']['getAttribute']('driver')) return true;

    return false;
};

According to answer, there are multiple methods to remove them. One of them is simply opening chromedriver.exe with a HEX-editor and removing all occurences of $cdc_

Esplanade answered 19/12, 2016 at 10:14 Comment(31)

This is interesting, you tested it and it worked even across multiple requests? – Magavern 19/12, 2016 at 17:43

yes it worked without probs, note one problem is if you fell into the "blacklist" BEFORE this change, it's quite hard to get out. if you want to get out of the existing black list, you need to implement fake canvas fingerprinting, disable flash, change IP, and change request header order (swap language and Accept headers). Once you fell into the blacklist, they have very good measures to track you, even if you change IP, even if you open chrome in incognito, etc – Esplanade 19/12, 2016 at 19:23

Just tested and it worked. Be prepared that at least one hour is needed though, to clone chromium repository and build it... – Woke 24/5, 2017 at 11:49

Makalele: I clarified: change the $cdc_ part to something completely random, eg change it to 'MAKALELE_' – Esplanade 9/6, 2017 at 21:14

@Arya chromium.googlesource.com/chromium/src/+/master/docs/… – Molotov 26/6, 2017 at 4:3

@Molotov Thanks. After I follow the instructions (until "fetch chromium") in the url, I can't find "/js" or "/js/call_function.js" file from "chromium/src/" to remove such "$cdc_". Do you know where it is located? – Plenty 1/7, 2017 at 1:6

I found the file "/Users/your_username/chromium/src/chrome/test/chromedriver/js" – Plenty 1/7, 2017 at 9:8

In my case the webdriver is apparently still detected, even though this $cdc_ variable had its name changed. Any other suggestions on what can be done to make Chrome look "100% natural"? – Layne 18/11, 2017 at 14:0

@DennisThrysøe did you see the later part of my post? – Esplanade 21/11, 2017 at 8:24

@Erti-ChrisEelmaa: Yes, none of these seem to be present. – Layne 22/11, 2017 at 20:17

compile older version of Chromium. See when my post was written and take chrome version that was about then roughly. A lot can change in a year. – Esplanade 23/11, 2017 at 11:47

There is a simple, and automatable way to do that, which is to replace cdc_ with a 4 character other string in the chromedriver binary. The way to do that is to use a similar perl command : perl -pi -e 's/cdc_/abc_/g' /path/to/chromedriver. As soon as someone else confirms, feel free to edit the original answer – Barbarese 30/1, 2018 at 21:24

I also find that there is only "cdc_" to replace on Mac and Linux 64, and on Linux 32 one should replace both "wdc_" and "selenium". Again as soon as someone confirms I'll add that to the post – Barbarese 1/2, 2018 at 23:36

I simply replaced $cdc with xxxx in chromedriver.exe in a hex editor and it worked! I also noticed that if you maximize the browser window (rather than use a predefined size) it's detected less often. – Month 25/2, 2018 at 17:59

@RafaelAlmeida ChromeDriver 2.35.528139 – Month 15/6, 2018 at 17:21

was this on windows, osx, or linux? Hex editing on osx doesn't seem to work. – Alliteration 23/7, 2018 at 22:28

hex-editted with $zzz_zzzzzzzzzzzzzzzzzzzzzz_ (same amount of characters) but didn't work. – Evade 9/8, 2018 at 20:54

anyone having problems take my JS code that detects bots and upload it to somewhere, then visit that page and see what is being detected. – Esplanade 10/8, 2018 at 7:25

@Alliteration on mac I get

selenium.common.exceptions.WebDriverException: Message: Service /Users/ishandutta2007/Downloads/chromedriver2 unexpectedly exited. Status code was: -9

– Malkamalkah 23/8, 2018 at 3:38

Tried this with windows 2.44 exe version, only one place has 'cdc_', and replace it seems not working, still can be detected by websites. – Acierate 8/12, 2018 at 6:28

Replacing $cdc_ worked for a couple of times, then got blocked again. It happens only on Linux though, in windows my script works fine. any other solution alongside this one will be of great help! – Piperpiperaceous 24/1, 2019 at 1:47

I'm assuming that Selenium/Chrome71 no longer exposes these vars - I've just tried putting them into the console and they didn't exist: Uncaught ReferenceError: $cdc_asdjflasutopfhvcZLmcfl_ is not defined – Salty 31/1, 2019 at 5:36

@NinoŠkopac you should be trying the example of "how bot networks might use to detect" script I gave at the bottom of the post. the suffix after cdc_ is not constant, and changes between builds. The CDC variable also doesn't appear immediately, it appears when you've been browsing a little on web. – Esplanade 31/1, 2019 at 22:47

I just tried executing your function in Chrome 72 via BrowserStack - it returned false every time (which is good I think). – Salty 4/2, 2019 at 14:10

I used a hex editor (iHex) to edit the chromedriver with version 2.35.528139. I only found one instance of cdc_ which i replaced with xxxx (like that szx did). How can I tell if it worked properly? – Piaffe 17/6, 2019 at 3:42

so in summary there is no way of getting past a website supported by distil networks.... is this the end? no more webscraping :). even if it is not selenium it apperars they detect puthon, so does this mean that scrapy, PySpider, etc.... are also hopeless? i wonder if somebody should open up a SUPER BOUNTY on this one as there seems to be no definitve way forward.. – Chile 28/6, 2019 at 15:20

@Erti-ChrisEelmaa Is there any undetectable open-source fork which you are aware of ? – Malkamalkah 27/9, 2019 at 9:2

I suppose the developers of the browsers can add a new tricky flag in future, without our knowledge. I wonder if it has ever happened since this answer was written? – Inveterate 30/1 at 13:51

@Inveterate that can happen, but it is extremely unlikely they would touch an old version, that's why it is important to pay attention to the dates / version numbers. Eg if you want to try this you can go back and get chromedrivers source code at 2016 and build it yourself. – Esplanade 30/1 at 18:20

@Erti-ChrisEelmaa I see. Probably using a very old browser makes you an outlier. But then, websites might loose many real users if they block old browsers. So the solution would be to update the version of the browser periodically, together with new tricky flags – Inveterate 3/2 at 21:38

186