HTMLUnit: change User Agent string

Question

I'm using HtmlUnit in my Java project in order to test a web page that has got Javascript inside. My code clicks a button that calls a Javascript function, wich redirect the user to another page (like link shortener services). This is the code:

public void click()
{
    WebClient webClient = new WebClient();
    HtmlPage page = webClient.getPage("http://mywebsite.com");
    HtmlImage a = page.getHtmlElementById("my_button");
    page = (HtmlPage) a.click();
}

The problem is that HTMLUnit uses the default User Agent (Internet Explorer 8) and there are a only few to set (Firefox 17 and Chrome). The behavior of mywebsite.com doesn't change if it detects another browser/user agent. By the way, the user agent string is stored by the website for statistical purposes and I need to change it every time I visit it.

I've tried to change the User Agent by creating a new BrowserVersion object in this way:

BrowserVersion bv = new BrowserVersion("Chrome", "Mozilla/5.0", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1468.0 Safari/537.36", 28);

By the way, when I instantiate a Webclient object passing my bv object, my code doesn't work anymore. From what I've understood, HtmlUnit documentation says that I've got to check if the user agent specified in my BrowserVersion has proper features to run Javascript.

However, note that the constants are not enough to fully customize the browser, you also need to look into the BrowserVersionFeatures and the classes inside "javascript" package.

What does it mean? Why HtmlUnit doesn't work anymore? My goal is only to change the User Agent string. How can I achieve this? Note that I've tried Selenium too, without success. Thank you for your help.

EDIT 1:

Found this trick. If I instantiate the BrowserVersion as follow:

BrowserVersion bv = new BrowserVersion("Netscape", "blablabla", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1468.0 Safari/537.36", 0);

it works, but I don't understand WHY. I have to set the first string as Netscape (tried Chrome and Mozilla but it doesn't work). The 2nd string is random, I can put anything if Netscape is set as first parameter. The third string is the well formatted User Agent and the 4th parameter is an integer indicating the version. Can you explain me why it works only if Netscape is passed as first parameter and random to others (except the 2nd)?

UPDATES:

Sometimes it doesn't work (as described above). For some User Agents strings, the page is not correctly loading. I cannot understand why the User Agent should modify the behaviour of HtmlUnit as I'm pretty sure that Javascript it's quite easy and should be run by all browser versions. So, my final question is: how can I change the user agent string in HtmlUnit without altering its behaviour when executing Javascript?

Absolution

Answer 1

Using Htmlunit 2.28 you can set it like this example below

final BrowserVersion myChrome = new BrowserVersion.BrowserVersionBuilder(BrowserVersion.CHROME)
    // do your setup here
    .setXXX(..)
    .build();

Flagwaving

Answer 2

You can pass it to the WebClient constructor:

WebClient webClient = new WebClient(BrowserVersion.CHROME);

Supported user agents are listed here: http://htmlunit.sourceforge.net/apidocs/com/gargoylesoftware/htmlunit/BrowserVersion.html

Fritter

Answer 3

Don't worry about creating a new BrowserVersion object. You can change the user agent after creating the driver without messing with all the versioning mess:

String DEFAULT_MOBILE_USER_AGENT_STRING = "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25";

HtmlUnitDriver driver = new HtmlUnitDriver(); //Or insert a capabilities object
driver.getBrowserVersion().setUserAgent(DEFAULT_MOBILE_USER_AGENT_STRING);
driver.get("http://facebook.com");

This will correctly visit http://facebook.com and get redirected to https://m.facebook.com/?_rdr&refsrc=https://www.facebook.com/ because the user agent string tells facebook the browser is an iPhone.

Elation

Isn't this related to Selenium? I'm talking about HTMLUnit here – Absolution

It seems like the API has changed with Selenium 3 - the BrowserVersion is kind of immutable – Springhalt

Answer 4

On the Selenium 3 you can do:

val browser = BrowserVersionBuilder(BrowserVersion.CHROME).build()
val driver = HtmlUnitDriver(browser);

The BrowserVersionBuilder brings a lot of customizations.

Biological

Answer 5

I was able to set the user agent to a random string such as 'Selenium' when constructing the Chrome Driver object.

ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--start-maximized");
chromeOptions.addArguments("user-agent=Selenium");
chromeOptions.setExperimentalOption("useAutomationExtension", false);
aDriver = new ChromeDriver(chromeOptions);
aDriver.manage().timeouts().implicitlyWait(Const.Chrome_TIMEOUT, TimeUnit.SECONDS);

Indole