Using Selenium RemoteWebDriver in C#

Question

I'm trying to use the Selenium RemoteWebDriver in C#, basically all I want to do is to programmatically scrape a webpage's html after the javascript has completed manipulating the DOM (without a browser window popping up.)

First I started the selenium-server.jar like so:

C:\Program Files\selenium-server>java -jar "C:\Program Files\selenium-server\selenium-server.jar"

13:34:46.163 INFO - Java: Sun Microsystems Inc. 19.1-b02
13:34:46.166 INFO - OS: Windows 7 6.1 amd64
13:34:46.174 INFO - v2.0 [a2], with Core v2.0 [a2]
13:34:46.277 INFO - RemoteWebDriver instances should connect to: **http://127.0.0.1:4444/wd/hub**
13:34:46.278 INFO - Version Jetty/5.1.x
13:34:46.279 INFO - Started HttpContext[/selenium-server/driver,/selenium-server
/driver]
13:34:46.280 INFO - Started HttpContext[/selenium-server,/selenium-server]
13:34:46.280 INFO - Started HttpContext[/,/]
13:34:46.311 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler@6019d0
a1
13:34:46.312 INFO - Started HttpContext[/wd,/wd]
13:34:46.316 INFO - Started SocketListener on 0.0.0.0:4444
13:34:46.316 INFO - Started org.openqa.jetty.jetty.Server@199a0c7c

Next I tried to run this line from a test case:

var driver = new RemoteWebDriver(new Uri("http://127.0.0.1:4444/wd/hub"), DesiredCapabilities.Chrome());

This line errors:

Test 'Housters.Test.ScrapeTest.TestSelenium' failed: OpenQA.Selenium.WebDriverException : Unexpected error. {"message":"java.lang.NullPointerException","localizedMessage":"java.lang.NullPointerException","cause":{"class":"java.lang.NullPointerException","stackTrace":[{"fileName":"DriverFactory.java","class":"java.lang.StackTraceElement","lineNumber":43,"className":"org.openqa.selenium.remote.server.DriverFactory","methodName":"getBestMatchFor","nativeMethod":false},{"fileName":"DriverFactory.java","class":"java.lang.StackTraceElement","lineNumber":76,"className":"org.openqa.selenium.remote.server.DriverFactory","methodName":"newInstance","nativeMethod":false},{"fileName":"Session.java","class":"java.lang.StackTraceElement","lineNumber":48,"className":"org.openqa.selenium.remote.server.Session$1","methodName":"call","nativeMethod":false},{"fileName":"Session.java","class":"java.lang.StackTraceElement","lineNumber":46,"className":"org.openqa.selenium.remote.server.Session$1","methodName":"call","nativeMethod":false},{"class":"java.lang.StackTraceElement","lineNumber":-1,"className":"java.util.concurrent.FutureTask$Sync","methodName":"innerRun","nativeMethod":false},{"class":"java.lang.StackTraceElement","lineNumber":-1,"className":"java.util.concurrent.FutureTask","methodName":"run","nativeMethod":false},{"class":"java.lang.StackTraceElement","lineNumber":-1,"className":"java.util.concurrent.ThreadPoolExecutor$Worker","methodName":"runTask","nativeMethod":false},{"class":"java.lang.StackTraceElement","lineNumber":-1,"className":"java.util.concurrent.ThreadPoolExecutor$Worker","methodName":"run","nativeMethod":false},{"class":"java.lang.StackTraceElement","lineNumber":-1,"className":"java.lang.Thread","methodName":"run","nativeMethod":false}]},"class":"java.util.concurrent.ExecutionException","stackTrace":[{"class":"java.lang.StackTraceElement","lineNumber":-1,"className":"java.util.concurrent.FutureTask$Sync","methodName":"innerGet","nativeMethod":false},{"class":"java.lang.StackTraceElement","lineNumber":-1,"className":"java.util.concurrent.FutureTask","methodName":"get","nativeMethod":false},{"fileName":"Session.java","class":"java.lang.StackTraceElement","lineNumber":68,"className":"org.openqa.selenium.remote.server.Session","methodName":"execute","nativeMethod":false},{"fileName":"Session.java","class":"java.lang.StackTraceElement","lineNumber":54,"className":"org.openqa.selenium.remote.server.Session","methodName":"<init>","nativeMethod":false},{"fileName":"DriverSessions.java","class":"java.lang.StackTraceElement","lineNumber":76,"className":"org.openqa.selenium.remote.server.DriverSessions","methodName":"newSession","nativeMethod":false},{"fileName":"NewSession.java","class":"java.lang.StackTraceElement","lineNumber":46,"className":"org.openqa.selenium.remote.server.handler.NewSession","methodName":"handle","nativeMethod":false},{"fileName":"ResultConfig.java","class":"java.lang.StackTraceElement","lineNumber":144,"className":"org.openqa.selenium.remote.server.rest.ResultConfig","methodName":"handle","nativeMethod":false},{"fileName":"DriverServlet.java","class":"java.lang.StackTraceElement","lineNumber":271,"className":"org.openqa.selenium.remote.server.DriverServlet","methodName":"handleRequest","nativeMethod":false},{"fileName":"DriverServlet.java","class":"java.lang.StackTraceElement","lineNumber":256,"className":"org.openqa.selenium.remote.server.DriverServlet","methodName":"doPost","nativeMethod":false},{"fileName":"HttpServlet.java","class":"java.lang.StackTraceElement","lineNumber":727,"className":"javax.servlet.http.HttpServlet","methodName":"service","nativeMethod":false},{"fileName":"HttpServlet.java","class":"java.lang.StackTraceElement","lineNumber":820,"className":"javax.servlet.http.HttpServlet","methodName":"service","nativeMethod":false},{"fileName":"ServletHolder.java","class":"java.lang.StackTraceElement","lineNumber":428,"className":"org.openqa.jetty.jetty.servlet.ServletHolder","methodName":"handle","nativeMethod":false},{"fileName":"ServletHandler.java","class":"java.lang.StackTraceElement","lineNumber":677,"className":"org.openqa.jetty.jetty.servlet.ServletHandler","methodName":"dispatch","nativeMethod":false},{"fileName":"ServletHandler.java","class":"java.lang.StackTraceElement","lineNumber":568,"className":"org.openqa.jetty.jetty.servlet.ServletHandler","methodName":"handle","nativeMethod":false},{"fileName":"HttpContext.java","class":"java.lang.StackTraceElement","lineNumber":1530,"className":"org.openqa.jetty.http.HttpContext","methodName":"handle","nativeMethod":false},{"fileName":"HttpContext.java","class":"java.lang.StackTraceElement","lineNumber":1482,"className":"org.openqa.jetty.http.HttpContext","methodName":"handle","nativeMethod":false},{"fileName":"HttpServer.java","class":"java.lang.StackTraceElement","lineNumber":909,"className":"org.openqa.jetty.http.HttpServer","methodName":"service","nativeMethod":false},{"fileName":"HttpConnection.java","class":"java.lang.StackTraceElement","lineNumber":820,"className":"org.openqa.jetty.http.HttpConnection","methodName":"service","nativeMethod":false},{"fileName":"HttpConnection.java","class":"java.lang.StackTraceElement","lineNumber":986,"className":"org.openqa.jetty.http.HttpConnection","methodName":"handleNext","nativeMethod":false},{"fileName":"HttpConnection.java","class":"java.lang.StackTraceElement","lineNumber":837,"className":"org.openqa.jetty.http.HttpConnection","methodName":"handle","nativeMethod":false},{"fileName":"SocketListener.java","class":"java.lang.StackTraceElement","lineNumber":245,"className":"org.openqa.jetty.http.SocketListener","methodName":"handleConnection","nativeMethod":false},{"fileName":"ThreadedServer.java","class":"java.lang.StackTraceElement","lineNumber":357,"className":"org.openqa.jetty.util.ThreadedServer","methodName":"handle","nativeMethod":false},{"fileName":"ThreadPool.java","class":"java.lang.StackTraceElement","lineNumber":534,"className":"org.openqa.jetty.util.ThreadPool$PoolThread","methodName":"run","nativeMethod":false}]}
    at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)
    at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(DriverCommand driverCommandToExecute, Dictionary`2 parameters)
    at OpenQA.Selenium.Remote.RemoteWebDriver.StartSession(ICapabilities desiredCapabilities)
    at OpenQA.Selenium.Remote.RemoteWebDriver..ctor(Uri remoteAddress, ICapabilities desiredCapabilities)
    ScrapeTest.cs(36,0): at Housters.Test.ScrapeTest.TestSelenium()

In the service console window, it shows this error:

13:44:55.558 INFO - WebDriver remote server: Executing: [new session: null] at URL: /session)
13:44:55.560 INFO - WebDriver remote server: Exception: java.lang.NullPointerException

I'm trying to do this from Windows 7 x64. What am I doing wrong? This seems like a lot of work for what I want to do...

Nonu

If you don't want a "browser window popping up", Selenium may not be the right tool for the job. It will always instantiate a browser window (though there is the HtmlUnit driver, which handles things in memory). What happens if you use DesiredCapabilities.Firefox()? – Langmuir

Firefox pops up a new browser window. I spent many hours with the HtmlUnit driver ported over from Java to a .NET dll, however half the time it would throw an exception with no error message, the other half of the time it'd take 15-30 seconds to make a web request. – Nonu

Answer 1

If you are attempting to run Selenium2 on localhost, you don't need to use RemoveWebDriver() and Selenium Server - You can use the following:

WebDriver driver = new ChromeDriver();

I have found less issues running locally in this way than when using RemoveWebDriver, and you should find you get more information readily available if there is a problem.

Alternatively you can use HtmlUnit directly as described at http://blog.stevensanderson.com/2010/03/30/using-htmlunit-on-net-for-headless-browser-automation/

Adrien

That pops up a new browser window. – Nonu

If you need it to be completely hidden then your only option with Selenium would be to use HtmlUnit through RemoteWebDriver. Use DesiredCapabilities.HtmlUnitWithJS instead of DesiredCapabilities.Chrome. – Adrien

Answer 2

I see that you already have an accepted answer, but because it answers completely different question, I will add my 5 cents.

So if you want to connect to a remove web driver, instead of the line:

var driver = new FirefoxDriver();

You have to do:

var driver = new RemoteWebDriver(new Uri("http://localhost:4444/wd/hub"), DesiredCapabilities.Firefox()); // instead of this url you can put the url of your remote hub

Do not forget to include remote namespace before the previous line:

using OpenQA.Selenium.Remote;

Smolensk

I'm always seeing this "Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:4444". Pls help.. – Pigskin

I am trying to use RemoteWebDriver with Multilogin service. But I get "session not created: Missing or invalid capabilities (Driver info: chromedriver=73.0.3683.103,platform=Windows NT 10.0.17763 x86_64)" – Githens

Answer 3

Getting the HTML of the page you're interested in is easy, particularly in .NET (use HttpWebRequest). Parsing it into a DOM-like structure and applying the transforms indicated by the JavaScript on the page, not so much. It requires a browser, or at the very least, an HTML parser with a DOM construction engine, as well as a JS script engine to manipulate the resulting DOM. It's not a trivial problem. At this point, your choices are: (1) use HtmlUnit, env.js, or one of the other "headless" browser projects, none of which require WebDriver to do what you want to do; or (2) live with a browser window popping up. And in the case of (2), using the raw InternetExplorerDriver or FirefoxDriver (or ChromeDriver in the upcoming 2.0b4) is a simpler choice than using the remote server.

Langmuir

Answer 4

Since Chrome 59 came available you are able to run the browser in headless mode. See this thread

Pelagic