How to Prevent Selenium 3.0 (Geckodriver) from Creating Temporary Firefox Profiles?
Asked Answered
S

6

15

I'm running the latest version of Selenium WebDriver with Geckodriver. I want to prevent Selenium from creating temporary Firefox Profiles in the temporary files directory when launching a new instance of WebDriver. Instead I want to use the original Firefox Profile directly. This has double benefit. First, it saves time (it takes significant amount of time for the profile to be copied to the temporary directory). Second, it ensures that cookies created during session are saved to the original profile. Before Selenium started relying on Geckodriver I was able to solve this problem by editing the class FirefoxProfile.class in SeleniumHQ as seen below:

public File layoutOnDisk() {

 File profileDir;

 if (this.disableTempProfileCreation) {
  profileDir = this.model;
  return profileDir;

  } else {

   try {
    profileDir = TemporaryFilesystem.getDefaultTmpFS().createTempDir("ABC", "XYZ");
    File userPrefs = new File(profileDir, "user.js");
    this.copyModel(this.model, profileDir);
    this.installExtensions(profileDir);
    this.deleteLockFiles(profileDir);
    this.deleteExtensionsCacheIfItExists(profileDir);
    this.updateUserPrefs(userPrefs);
    return profileDir;
    } catch (IOException var3) {
   throw new UnableToCreateProfileException(var3);
  }
 }
}

This would stop Selenium from creating a temporary Firefox Profile when the parameter disableTempProfileCreation was set to true.

However, now that Selenium is being controlled by Geckodriver this solution no longer works as the creation (and launch) of Firefox Profile is controlled by Geckodriver.exe (which is written in Rust language). How can I achieve the same objective with Geckodriver? I don't mind editing the source code. I'm using Java.

Thanks

Important Update:

I would like to thank everyone for taking the time to respond to this question. However, as stated in some of the comments, the first 3 answers do not address the question at all - for two reasons. First of all, using an existing Firefox Profile will not prevent Geckodriver from copying the original profile to a temporary directory (as indicated in the OP and clearly stated by one or more of the commentators below). Second, even if it did it is not compatible with Selenium 3.0.

I'm really not sure why 3 out of 4 answer repeat the exact same answer with the exact same mistake. Did they read the question? The only answer the even attempts to address the question at hand is the answer by @Life is complex however it is incomplete. Thanks.

Sparkman answered 16/5, 2021 at 20:53 Comment(5)
After doing a lot of code and issue reviews for Geckodriver, Firefox and Selenium. I have determined that the effort to do what you need is difficult. You could edit the source code capabilities.rs to remove the .temp_dir call, but it seems that you might have to change code in Firefox also to prevent the .temp_dir* from being created. I also noted that Selenium is moving away from copying and using profiles in ver 4, which is in beta. So fixing the issue in the current version might require more effort when the new versions are released.Immune
I also noted that multiple users have complained to Selenium about the issue that you're trying to resolve. The overseers at Selenium either tell the users to contact Mozilla for support or tell them to try ver 4, which doesn't use profiles. I have found that there needs to better documentation on setting preferences under Selenium for Firefox, Chrome and Edge.Immune
My best recommendation would be to either move to Selenium ver 4 (BETA), which doesn't use profiles. (I'm unsure about the temp_dir thing in that code base) OR downgrade your Geckodriver, Firefox and Selenium to what worked before you upgraded.Immune
@Lifeiscomplex thanks so much for your effort! The thing about Selenium version 4 is that I think it was in beta since 2018 so I would be concerned whether we we'll see official release any time in near future. Regarding profiles being depreciated in Selenium 4 that is interesting and I would love to learn more. I could not find official documentation on this specific feature and whether there is any copying of original profiles in v4. Downgrading Selenium is not an option because a big chunk of the code is written around version 3 that if I downgrade it will cause much more issues. Thanks!Sparkman
So I assume that your issues happened when you upgraded either Geckodriver or Firefox. I also assume that you cannot downgrade either of these applications to their previous state. Concerning Selenium 4 the release notes for the latest have this - (1) Only give deprecation warning if Profile is being used in options (2) Deprecate using a Firefox profile in Options. I cannot find anything on creating the temp_dir in the release notes.Immune
I
8

UPDATED POST 05-30-2021


This is the hardest question that I have every tried to answer on Stack Overflow. Because it involved the interactions of several code bases written in multiple languages (Java, Rust and C++). This complexity made the question potentially unsolvable.

My last crack at this likely unsolvable question:

Within the code in your question you are modifying the file user.js This file is still used by Selenium.

public FirefoxProfile() {
    this(null);
  }

  /**
   * Constructs a firefox profile from an existing profile directory.
   * <p>
   * Users who need this functionality should consider using a named profile.
   *
   * @param profileDir The profile directory to use as a model.
   */
  public FirefoxProfile(File profileDir) {
    this(null, profileDir);
  }

  @Beta
  protected FirefoxProfile(Reader defaultsReader, File profileDir) {
    if (defaultsReader == null) {
      defaultsReader = onlyOverrideThisIfYouKnowWhatYouAreDoing();
    }

    additionalPrefs = new Preferences(defaultsReader);

    model = profileDir;
    verifyModel(model);

    File prefsInModel = new File(model, "user.js");
    if (prefsInModel.exists()) {
      StringReader reader = new StringReader("{\"frozen\": {}, \"mutable\": {}}");
      Preferences existingPrefs = new Preferences(reader, prefsInModel);
      acceptUntrustedCerts = getBooleanPreference(existingPrefs, ACCEPT_UNTRUSTED_CERTS_PREF, true);
      untrustedCertIssuer = getBooleanPreference(existingPrefs, ASSUME_UNTRUSTED_ISSUER_PREF, true);
      existingPrefs.addTo(additionalPrefs);
    } else {
      acceptUntrustedCerts = true;
      untrustedCertIssuer = true;
    }

    // This is not entirely correct but this is not stored in the profile
    // so for now will always be set to false.
    loadNoFocusLib = false;

    try {
      defaultsReader.close();
    } catch (IOException e) {
      throw new WebDriverException(e);
    }
  }

So in theory you should be able to modify capabilities.rs in the geckodriver source code. That file contains the temp_dir.

As I stated this in only a theory, because when I looked at the Firefox source, which has temp_dir spread throughout the code base.

ORIGINAL POST 05-26-2021


I'm not sure that you can prevent Selenium from creating a temporary Firefox Profile.

From the gecko documents:

"Profiles are created in the systems temporary folder. This is also where the encoded profile is extracted when profile is provided. By default geckodriver will create a new profile in this location."

The only solution that I see at the moment would require you modify the Geckodriver source files to prevent the creation of temporary folders/profiles.

I'm currently looking at the source. These files might be the correct ones, but I need to look at the source more:

Here are some other files that need to be combed through:

https://searchfox.org/mozilla-central/search?q=tempfile&path=


This looks promising:

https://searchfox.org/mozilla-central/source/testing/geckodriver/doc/Profiles.md

"geckodriver uses [profiles] to instrument Firefox’ behaviour. The user will usually rely on geckodriver to generate a temporary, throwaway profile. These profiles are deleted when the WebDriver session expires.

In cases where the user needs to use custom, prepared profiles, geckodriver will make modifications to the profile that ensures correct behaviour. See [Automation preferences] below on the precedence of user-defined preferences in this case.

Custom profiles can be provided two different ways:

1. by appending --profile /some/location to the [args capability], which will instruct geckodriver to use the profile in-place;

I found this question on trying to do this: how do I use an existing profile in-place with Selenium Webdriver?

Also here is an issue that was raised in selenium on Github concerning the temp directory. https://github.com/SeleniumHQ/selenium/issues/8645


Looking through the source of geckodriver v0.29.1 I found a file where the profile is loaded.

source: capabilities.rs

   fn load_profile(options: &Capabilities) -> WebDriverResult<Option<Profile>> {
        if let Some(profile_json) = options.get("profile") {
            let profile_base64 = profile_json.as_str().ok_or_else(|| {
                WebDriverError::new(ErrorStatus::InvalidArgument, "Profile is not a string")
            })?;
            let profile_zip = &*base64::decode(profile_base64)?;

            // Create an emtpy profile directory
            let profile = Profile::new()?;
            unzip_buffer(
                profile_zip,
                profile
                    .temp_dir
                    .as_ref()
                    .expect("Profile doesn't have a path")
                    .path(),
            )?;

            Ok(Some(profile))
        } else {
            Ok(None)
        }
    }

source: marionette.rs

    fn start_browser(&mut self, port: u16, options: FirefoxOptions) -> WebDriverResult<()> {
        let binary = options.binary.ok_or_else(|| {
            WebDriverError::new(
                ErrorStatus::SessionNotCreated,
                "Expected browser binary location, but unable to find \
             binary in default location, no \
             'moz:firefoxOptions.binary' capability provided, and \
             no binary flag set on the command line",
            )
        })?;

        let is_custom_profile = options.profile.is_some();

        let mut profile = match options.profile {
            Some(x) => x,
            None => Profile::new()?,
        };

        self.set_prefs(port, &mut profile, is_custom_profile, options.prefs)
            .map_err(|e| {
                WebDriverError::new(
                    ErrorStatus::SessionNotCreated,
                    format!("Failed to set preferences: {}", e),
                )
            })?;

        let mut runner = FirefoxRunner::new(&binary, profile);

        runner.arg("--marionette");
        if self.settings.jsdebugger {
            runner.arg("--jsdebugger");
        }
        if let Some(args) = options.args.as_ref() {
            runner.args(args);
        }

        // https://developer.mozilla.org/docs/Environment_variables_affecting_crash_reporting
        runner
            .env("MOZ_CRASHREPORTER", "1")
            .env("MOZ_CRASHREPORTER_NO_REPORT", "1")
            .env("MOZ_CRASHREPORTER_SHUTDOWN", "1");

        let browser_proc = runner.start().map_err(|e| {
            WebDriverError::new(
                ErrorStatus::SessionNotCreated,
                format!("Failed to start browser {}: {}", binary.display(), e),
            )
        })?;
        self.browser = Some(Browser::Host(browser_proc));

        Ok(())
    }

    pub fn set_prefs(
        &self,
        port: u16,
        profile: &mut Profile,
        custom_profile: bool,
        extra_prefs: Vec<(String, Pref)>,
    ) -> WebDriverResult<()> {
        let prefs = profile.user_prefs().map_err(|_| {
            WebDriverError::new(
                ErrorStatus::UnknownError,
                "Unable to read profile preferences file",
            )
        })?;

        for &(ref name, ref value) in prefs::DEFAULT.iter() {
            if !custom_profile || !prefs.contains_key(name) {
                prefs.insert((*name).to_string(), (*value).clone());
            }
        }

        prefs.insert_slice(&extra_prefs[..]);

        if self.settings.jsdebugger {
            prefs.insert("devtools.browsertoolbox.panel", Pref::new("jsdebugger"));
            prefs.insert("devtools.debugger.remote-enabled", Pref::new(true));
            prefs.insert("devtools.chrome.enabled", Pref::new(true));
            prefs.insert("devtools.debugger.prompt-connection", Pref::new(false));
        }

        prefs.insert("marionette.log.level", logging::max_level().into());
        prefs.insert("marionette.port", Pref::new(port));

        prefs.write().map_err(|e| {
            WebDriverError::new(
                ErrorStatus::UnknownError,
                format!("Unable to write Firefox profile: {}", e),
            )
        })
    }
}

After looking through the gecko source it looks like mozprofile::profile::Profile is coming from FireFox and not geckodriver


It seems that you might have issues with profiles when you migrate to Selenium 4.

ref: https://github.com/SeleniumHQ/selenium/issues/9417

For Selenium 4 we have deprecated the use of profiles as there are other mechanisms that we can do to make the start up faster. Please use the Options class to set preferences that you need and if you need to use an addon use the driver.install_addon("path/to/addon") you can install selenium 4, which is in beta, via pip install selenium --pre


I noted in your code you were writing to user.js, which is a custom file for FireFox. Have you considered creating on these files manually outside of Gecko?

Also have you looked at mozprofile?

Immune answered 26/5, 2021 at 0:48 Comment(6)
Thank you for your response. Yours actually the only answer that addresses the question. However, it is still incomplete as I would like to which files must be edited to achieve stated goal. Thanks!Sparkman
@Sparkman I'm still looking for the correct file. There is a ton of testing code in mozilla central that I have to separate from the base code. I will keep looking, but likely won't get anything posted before this bounty question expires. This is where I'm currently looking searchfox.org/mozilla-central/source/browser/appImmune
thanks for the update. Once we con pinpoint the files that need to be edited will accept as answer. However, one thing I'm trying to understand is: the files you are pointing to are part of Mozilla Firefox? But I was under impression that we need to modify source of: github.com/mozilla/geckodriver/releases/tag/v0.29.1 (but this does not contain any "Profile" class) am I wrong? Thanks!Sparkman
@Sparkman I'm looking through the source code of that release now. I believe that the Profile class is under mozprofile::profile::Profile;. I think that the mozprofile::profile::Profile is coming from FireFox and not geckodriver.Immune
thanks again for your effort and comprehensive response! I'm going to look at the source code you mentioned to see if I could get this working with Selenium V3. One thing I'm not sure about: you mentioned the copying might be coming from Firefox not Geckdodriver. But if this is the case why doesn't FF copy permanent profile to temporary directory every time FF is launched (without WebDriver)? From my understanding the copying only happens when we launch FF via WebDriver / Selenium.. Thank you!Sparkman
@Lifeiscomplex Thank you for your excellent response. I added a bounty of +50 to the question. If it's possible to modify the response to include working Rust code that will accept any profile path as an argument at runtime (so not hard coded) where WebDriver won't create a temp copy of the original profile and instead use the original profile directly..Bandeau
B
3

Thanks to source code provided in answer of Life is complex in link!. I have the chance to look through geckodriver source.

EXPLANATION

I believe that the reason you could not find out any rust_tmp in source because it is generated randomly by Profile::new() function.

When I look deeper in code structure, I saw that browser.rs is the place where the browser is actually loaded which is called through marionette.rs. If you noticing carefully, LocalBrowser::new method will be called whenever a new session is initialized and the profile will be loaded in that state also. Then by checking browser.rs file, there will be a block code line 60 - 70 used to actually generate profile for new session instance. Now, what need to do is modifying this path to load your custom profile.

SHORT ANSWER

Downloading zip file of geckodriver-0.30.0, extracting it by your prefer zip program :P

Looking on src/browser.rs of geckodriver source, in line 60 - 70, hoping you will see something like this:

        let is_custom_profile = options.profile.is_some();

        let mut profile = match options.profile {
            Some(x) => x,
            None => Profile::new()?,
        };

Change it to your prefer folder ( hoping you know some rust code ), example:

        /*
        let mut profile = match options.profile {
            Some(x) => x,
            None => Profile::new()?,
        };
        */
        let path = std::path::Path::new("path-to-profile");
        let mut profile = Profile::new_from_path(path)?;

Re-compile with prefer rust compiler, example:

Cargo build

NOTE

Hoping this info will help you someway. This is not comprehensive but hoping it is good enough hint for you like it is possible to write some extra code to load profile from env or pass from argument, it is possible but I'm not rust developer so too lazy for providing one in here.

The above solution is work fine for me and I could load and use directly my profile from that. Btw, I work on Archlinux with rust info: cargo 1.57.0.

TBH, this is first time I push comment on stackoverflow, so feel free to correct me if I'm wrong or produce unclear answer :P

Update

  1. I worked in geckodriver 0.30.0 which will not be the same as geckodriver 0.29.1 mentioned by Life is complex. But the change between 2 versions just be split action, so the similar modify path in version 0.29.1 will be included in method MarionetteHandler::start_browser in file src/marionette.rs.

  2. Since my starting point is Life is complex answer, please looking through it for more information.

Bonucci answered 27/12, 2021 at 20:58 Comment(2)
Thank you for your excellent response. I added a bounty of +50 to the question. If it's possible to modify the response to include working Rust code that will accept any profile path as an argument at runtime (so not hard coded) where WebDriver won't create a temp copy of the original profile and instead use the original profile directly..Bandeau
@BradfordGriggs see my post here with your solution https://mcmap.net/q/783198/-how-to-prevent-selenium-3-0-geckodriver-from-creating-temporary-firefox-profilesVindication
V
1

I've come up with a solution that 1) works with Selenium 4.7.0--however, I don't see why it wouldn't work with 3.x as well, 2) allows the user to pass in an existing Firefox profile dynamically via an environment variable--and if this environment variable doesn't exist, simply acts "normally", and 3) if you do not want a temporary copy of the profile directory, simply do not pass the source profile directory to Selenium.

I downloaded Geckodriver 0.32.0 and made it so that you simply need to provide the Firefox profile directory via the environment variable FIREFOX_PROFILE_DIR. For example, in C#, before you create the FirefoxDriver, call:

Environment.SetEnvironmentVariable("FIREFOX_PROFILE_DIR", myProfileDir);

The change to Rust is in browser.rs, line 88, replacing:

    let mut profile = match options.profile {
        ProfileType::Named => None,
        ProfileType::Path(x) => Some(x),
        ProfileType::Temporary => Some(Profile::new(profile_root)?),
    };

with:

    let mut profile = if let Ok(profile_dir) = std::env::var("FIREFOX_PROFILE_DIR") {
        Some(Profile::new_from_path(Path::new(&profile_dir))?)
    } else {
        match options.profile {
            ProfileType::Named => None,
            ProfileType::Path(x) => Some(x),
            ProfileType::Temporary => Some(Profile::new(profile_root)?),
        }
    };

You may refer to my Git commit to see the diff against the original Geckodriver code.

Vindication answered 14/1, 2023 at 6:10 Comment(0)
L
0

The new driver by default creates a new profile if no options are set. To use a existing profile, one way to do this is to set the system property webdriver.firefox.profile before creating the firefox driver. A small code snippet that can create a firefox driver (given you have locations for geckodriver, and the firefox profile):

System.setProperty("webdriver.gecko.driver","path_to_gecko_driver");
System.setProperty("webdriver.firefox.profile", "path_to_firefox_profile");
WebDriver driver = new FirefoxDriver();

You could even set these system properties using the env. variables and skip defining them everywhere.

Another way to do this is to use the FirefoxOptions class which allows you to configure a lot of options. To start with, take a look at org.openqa.selenium.firefox.FirefoxDriver and org.openqa.selenium.firefox.FirefoxOptions. A small example:

FirefoxOptions options = new FirefoxOptions();
options.setProfile(new FirefoxProfile(new File("path_to_your_profile")));
WebDriver driver = new FirefoxDriver(options);

Hope this is helpful.

Liberec answered 19/5, 2021 at 15:49 Comment(2)
Unfortunately, this does not address the OPs problem. The OP knows how to launch Firefox with a specific Profile. The problem is that FF still copies the original Profile to a temporary folder which 1) increases the amount of time it takes to launch FF and 2) causes cookies to be deleted once WebDriver is closed. So the answer addresses neither of the two issues in the OP.Talkative
Preventing Selenium (Geckodriver) from creating temporary Firefox Profiles will require some kind of edit to the source files.Talkative
A
0

You can create firefox profile which will be clean and name it as SELENIUM

So When initializing the Webdriver get the profile which you have already created through the code, so that it wont create any new temp profiles all the time.

ProfilesIni allProfiles = new ProfilesIni();
FirefoxProfile desiredProfile = allProfiles.getProfile("SELENIUM");
WebDriver driver = new FirefoxDriver(desiredProfile);

That way, you assure that this profile will be used anytime you do the tests.

-Arjun

Amazed answered 20/5, 2021 at 13:38 Comment(1)
This won't prevent Selenium from creating a temporary Firefox Profile. Selenium always copies the profile to a temporary directory even when supplied with a custom profile. Please read the question before responding. That is besides the fact that I believe ProfilesIni is not compatible with Selenium 3.0.Impressment
P
0

You can handle this by using --

    FirefoxProfile profile = new FirefoxProfile(new File("D:\\Selenium Profile..."));                  

    WebDriver driver = new FirefoxDriver(profile);

There is one more option but it inherits all the cookies, cache contents, etc. of the previous uses of the profile let’s see how it will be --

    System.setProperty("webdriver.firefox.profile", "MySeleniumProfile");

    WebDriver driver = new FirefoxDriver(...);

Hope this answers your question in short.

Penile answered 23/5, 2021 at 6:19 Comment(2)
Did you got a chance to check this solution ? or you were looking for some other way to approach the issue?Penile
Thanks for your response. However, as stated in the OP, this does not address the question. Kindly see the update in OP. Thanks.Sparkman

© 2022 - 2024 — McMap. All rights reserved.