How to fix 'Address already in use' error with browsermob-proxy?

Question

I am still trying to use browsermob-proxy (here) to try to monitor the network traffic during a python selenium test (see browsermobproxypy). I am using the following script to test it:

from browsermobproxy import Server
server = Server("/home/adietz/Projects/Invest/browsermob-proxy/browsermob-proxy-2.1.4/bin/browsermob-proxy")
server.start()
proxy = server.create_proxy()

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)


proxy.new_har("google")
driver.get("http://www.google.co.in")
proxy.har

server.stop()
driver.quit()

I somehow managed to install a 'browsermob.jar' file (which is not really nicely described in the documentation here; you have to download the jar files from here). I am starting the proxy as follows:

java -jar ./browsermob-proxy-2.1.4/lib/browsermob-dist-2.1.4.jar --port 9090
Running BrowserMob Proxy using LittleProxy implementation. To revert to the legacy implementation, run the proxy with the command-line option '--use-littleproxy false'.
[INFO  2018-01-18T07:47:20,761 net.lightbody.bmp.proxy.Main] (main) Starting BrowserMob Proxy version 2.1.4 
[INFO  2018-01-18T07:47:20,782 org.eclipse.jetty.util.log] (main) jetty-7.x.y-SNAPSHOT 
[INFO  2018-01-18T07:47:20,802 org.eclipse.jetty.util.log] (main) started o.e.j.s.ServletContextHandler{/,null} 
[INFO  2018-01-18T07:47:20,885 org.eclipse.jetty.util.log] (main) Started [email protected]:9090 

Then I am running the test script as

python tester1.py

However, in the logs I see errors like follows:

[WARN  2018-01-18T07:28:30,265 org.eclipse.jetty.util.log] (main) FAILED [email protected]:8080: java.net.BindException: Address already in use 
[WARN  2018-01-18T07:28:30,266 org.eclipse.jetty.util.log] (main) FAILED org.eclipse.jetty.server.Server@4b41e4dd: java.net.BindException: Address already in use 
[ERROR 2018-01-18T07:28:30,266 net.lightbody.bmp.proxy.Main] (main) Failed to start Jetty server. Aborting. java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_151]
    at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_151]
    at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_151]
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:1.8.0_151]
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) ~[?:1.8.0_151]
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:162) ~[browsermob-dist-2.1.4.jar:?]
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:356) ~[browsermob-dist-2.1.4.jar:?]
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:240) ~[browsermob-dist-2.1.4.jar:?]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) ~[browsermob-dist-2.1.4.jar:?]
    at org.eclipse.jetty.server.Server.doStart(Server.java:269) ~[browsermob-dist-2.1.4.jar:?]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:55) ~[browsermob-dist-2.1.4.jar:?]
    at net.lightbody.bmp.proxy.Main.main(Main.java:64) [browsermob-dist-2.1.4.jar:?]

My best guess is that the port is already in use, or something else is not working correctly. Maybe I need to explicitly specify the port in the script? But how to do that? A python documentation is virtually not-existing ...

Addendum

• Looking at the source code(!) I figured out how to set the port. I am using the following line to define the server and the port in the python script:

  server = Server("/home/adietz/Projects/Invest/browsermob-proxy/browsermob-proxy-2.1.4/bin/browsermob-proxy", {'port':9090})

But no matter what I set as the port I always get the following error:

Traceback (most recent call last):
  File "tester1.py", line 5, in <module>
    proxy = server.create_proxy()
  File "/home/adietz/Projects/Invest/browsermob-proxy/venv_browsermob/local/lib/python2.7/site-packages/browsermobproxy/server.py", line 40, in create_proxy
    client = Client(self.url[7:], params)
  File "/home/adietz/Projects/Invest/browsermob-proxy/venv_browsermob/local/lib/python2.7/site-packages/browsermobproxy/client.py", line 37, in __init__
    "Another server running on this port?\n%s..." % content[:512])
Exception: Could not read Browsermob-Proxy json
Another server running on this port?
java.lang.RuntimeException: java.net.BindException: Address already in use
    at org.littleshoot.proxy.impl.DefaultHttpProxyServer.doStart(DefaultHttpProxyServer.java:545)
    at org.littleshoot.proxy.impl.DefaultHttpProxyServer.start(DefaultHttpProxyServer.java:489)
    at org.littleshoot.proxy.impl.DefaultHttpProxyServer.access$700(DefaultHttpProxyServer.java:69)
    at org.littleshoot.proxy.impl.DefaultHttpProxyServer$DefaultHttpProxyServerBootstrap.start(DefaultHttpProxyServer.java:858)
    at net.lightbody.bmp.Brows...

So there must be something else working incorrectly...

Maybe I will just code such a proxy on my own. Seems to be the faster way...

Bullpup

Check what other processes might be using the port. You can do this with lsof -- lsof -i:9090 then kill the process. If the socket is not opened with SO_REUSEADDR the OS may not let you use that port for a while. On Windows it can take several minutes to free up the port again, even after stopping the process. All that said, I'm not sure that you need to do java -jar ./browsermob-proxy-2.1.4..... -- To my recollection, you can use BMP with the Python code alone, you point it to the binary and it does the right thing. That may also be you're problem and would make sense given the issue – Ultrastructure

I checked that there is no process running using that port, i.e. there is no output of the command lsof -i:9090. However, running just the python script gives the Another server running on this port? error, when using the port 9090 in the python script... – Bullpup

Looking at some internal documentation for a retired project we have that uses BMP, there is no mention of doing java -jar anywhere. The BMP server should be started by the Python code alone. Not sure why you're having an address-in-use error if no processes are running on the port. My only guess is, like I mentioned, it can take some time for the address to be reusable again and varies by OS. It may be worthwhile to give your system a reboot if you can. You may be able to browse SO for other 'address in use' issues; I don't think it's specific to BMP or Python in this case. – Ultrastructure

But that explanation cannot explain why I get the SAME error when using many different ports (without using java -jar at any time!). Tried with 9080, 9070, 1111, 11111, 12345, 54321... Always the same error. – Bullpup

Yeah, you're right. That's very odd to me. My only other guess would be something along the lines of a security policy, especially if you're using an OS enforcing SELinux. Possibly an issue with ulimits? I might suggest trying to root around in other logs if a reboot doesn't clear it up. Sorry I know that doesn't help much. If you do eventually figure it out, please do update. I'm curious to what the issue might be. – Ultrastructure

@sytech: Nevertheless thanks for your help. I will check regarding the security policies... – Bullpup

Answer 1

We can start the BMP server with required port as given below. server accepts dictionary object for server arguments.

from browsermobproxy import Server
dict={'port':8090}
server = Server(path="C:\\Projects\\BrowserMobProxy\\bin\\browsermob-proxy",options=dict)
server.start()
proxy = server.create_proxy()

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)


proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob

server.stop()
driver.quit()

Dialectical

I just mentioned this in the comments. Last I remember, you don't need to do java -jar ....... to use BMP. If that's the case, sounds like OP starting the java server and trying to use python code to do the same thing on the same port is causing the address-in-use issue. – Ultrastructure

What does BMP mean? – Bullpup

@Alex, BMP means browser mob proxy – Dialectical

@Ultrastructure yes, he may be trying to start on same port. – Dialectical

I guess I give up and write my own proxy. Seems to be faster – Bullpup

Still not working sometimes. I tried many many ports, still get the error. I check the ports they are free! Still I get this error... – Bullpup

Answer 2

Initialize the server with this -

proxyserver =  Server(
       path="/root/tests/bsp_usecase_tests/bin/browsermob-proxy",
       options={'existing_proxy_port_to_use': 8090})

The internal code in Browsermob-Proxy is looking for key in 'options' dictionary named 'existing_proxy_port_to_use', else it uses default i.e. empty dictionary.

Dongola