How to use browsermob with python-selenium?

Asked 11/1, 2018 at 7:35 Answered 4/5, 2019 at 21:48

I want to use browsermob to monitor the network connections when doing a GUI test with selenium. I have found some information and documentation here and here and here, but its absolutely unclear how to really use it.

In the documentation it reads:

server = Server("path/to/browsermob-proxy")

But what is that path? Where to find it?

Also I see

java -jar browsermob.jar --port 9090

but no explanation at all as to what this jar file is, if it is part of the browsermob installation, or something unrelated.

I would appreciate if someone can provide a COMPLETE and WORKING example on how to use browsermob, and what ALL I need to install...

Katey answered 11/1, 2018 at 7:35 Comment(6)

the docs are pretty good and give complete examples: github.com/lightbody/browsermob-proxy#using-with-selenium – Lemaceon 15/1, 2018 at 13:38

When I try to start the proxy with ./browsermob-proxy -port 8080I get an error ` ./browsermob-proxy: line 12: $BASEDIR/lib/browsermob-dist-${project.version}.jar: bad substitution` – Katey 15/1, 2018 at 13:46

Also, do I need to start that proxy for the selenium tests, or is that in the tests itself. How? The python implementation is not mentioned... – Katey 15/1, 2018 at 13:47

github.com/AutomatedTester/browsermob-proxy-py – Lemaceon 15/1, 2018 at 13:55

server = Server("/Users/adietz/Projects/Invest/browsermob/browsermob-proxy/browsermob-dist/src/main/scripts/browsermob-proxy") server.start()

gives error: browsermobproxy.exceptions.ProxyServerError: Can't connect to Browsermob-Proxy – Katey 15/1, 2018 at 14:2

Also there is no documentation to install browsermob-proxy-py, and there is no single jarfile in the entire repository. Do I miss something here? – Katey 15/1, 2018 at 14:3

You can try the code below:

    from browsermobproxy import Server
    import psutil
    import time
    
    for proc in psutil.process_iter():
        # check whether the process name matches
        if proc.name() == "browsermob-proxy":
            proc.kill()
    
    dict = {'port': 8090}
    server = Server(path="./BrowserMobProxy/bin/browsermob-proxy", options=dict)
    server.start()
    time.sleep(1)
    proxy = server.create_proxy()
    time.sleep(1)
    from selenium import webdriver
    profile = webdriver.FirefoxProfile()
    selenium_proxy = proxy.selenium_proxy()
    profile.set_proxy(selenium_proxy)
    driver = webdriver.Firefox(firefox_profile=profile)
    
    
    proxy.new_har("google")
    driver.get("http://www.google.co.uk")
    print (proxy.har) # returns a HAR JSON blob
    
    server.stop()
    driver.quit()

Two things, if your code fails the process could be left open sometimes. So I added the code below for closing the duplicate instances.

import psutil
import time

for proc in psutil.process_iter():
    # check whether the process name matches
    if proc.name() == "browsermob-proxy":
        proc.kill()

Also a sleep of 1 second before and after creating the proxy.

server.start()
time.sleep(1)
proxy = server.create_proxy()
time.sleep(1)

This helps in getting rid of some intermittent issues which can be faced due to the server taking some time to start.

Cornwallis answered 22/1, 2018 at 17:47 Comment(4)

Thanks a lot. Your code suggestion actually did work! – Katey 23/1, 2018 at 7:8

@Tarun. i have my doubt what if browsermob-proxy process is running and capturing for a user in my website and another user start doing it. would not it kill user A process and data? – Piccalilli 6/10, 2018 at 10:14

@DhneshDhingra, when you do server.start() it will choose a free port and two process will come. But if a browser is opened manually then this will have no impact – Cornwallis 7/10, 2018 at 7:53

@Tarun actually if proc.name() == "browsermob-proxy": proc.kill(), woulndt be all the processes killed? – Piccalilli 7/10, 2018 at 18:40

`BrowserMob Proxy`

BrowserMob Proxy is an open source tools which is used to capture performance data for a web applications in an HAR format. It also allows to manipulate browser behavior and traffic, such as simulating network traffic, rewriting HTTP requests and responses etc and manipulate network traffic from their AJAX applications. In short, BrowserMob proxy helps us to capture client side performance data for a web application using Selenium WebDriver automated tests.

You can find more details about BrowserMob Proxy from the Python Documentation and this tutorial.

Demonstration of BrowserMob Proxy 2.0 with Python client on Windows

Install browsermob-proxy through the CLI :

C:\Users\your_user>pip install browsermob-proxy
Collecting browsermob-proxy
  Downloading browsermob-proxy-0.8.0.tar.gz
Collecting requests>=2.9.1 (from browsermob-proxy)
  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 92kB 148kB/s
Collecting idna<2.7,>=2.5 (from requests>=2.9.1->browsermob-proxy)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 61kB 639kB/s
Collecting urllib3<1.23,>=1.21.1 (from requests>=2.9.1->browsermob-proxy)
  Downloading urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 133kB 250kB/s
Collecting certifi>=2017.4.17 (from requests>=2.9.1->browsermob-proxy)
  Downloading certifi-2017.11.5-py2.py3-none-any.whl (330kB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 337kB 401kB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.9.1->browsermob-proxy)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 143kB 296kB/s
Installing collected packages: idna, urllib3, certifi, chardet, requests, browse
rmob-proxy
  Running setup.py install for browsermob-proxy ... done
Successfully installed browsermob-proxy-0.8.0 certifi-2017.11.5 chardet-3.0.4 id
na-2.6 requests-2.18.4 urllib3-1.22

Download the browsermob-proxy binaries browsermob-proxy-2.1.4-bin form the following url :
```
https://bmp.lightbody.net/
```
Extract and Save the directory within C:\Utility

Launch the Browsermobproxy Server manually through the CLI command :

C:\Utility\browsermob-proxy-2.1.4\lib>java -jar browsermob-dist-2.1.4.jar --port 9090
Running BrowserMob Proxy using LittleProxy implementation. To revert to the legacy implementation, run the proxy with the command-line option '--use-littleproxy false'.
[INFO  2018-01-17T19:01:30,276 net.lightbody.bmp.proxy.Main] (main) Starting BrowserMob Proxy version 2.1.4
[INFO  2018-01-17T19:01:30,388 org.eclipse.jetty.util.log] (main) jetty-7.x.y-SNAPSHOT
[INFO  2018-01-17T19:01:30,471 org.eclipse.jetty.util.log] (main) started o.e.j.s.ServletContextHandler{/,null}
[INFO  2018-01-17T19:01:30,871 org.eclipse.jetty.util.log] (main) Started [email protected]:9090

You can also launch the Browsermobproxy Server through your code as below.

Create a new PyDev module (if using Eclipse) and write a basic program through your IDE as follows :

from browsermobproxy import Server
server = Server("C:\\Utility\\browsermob-proxy-2.1.4\\bin\\browsermob-proxy")
server.start()
proxy = server.create_proxy()

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)


proxy.new_har("google")
driver.get("http://www.google.co.in")
proxy.har # returns a HAR JSON blob

server.stop()
driver.quit()

Snapshot :

Execute your Test as a Python Run
While your Program executes you will observe Firefox Quantum Browser gets initialized and the url http://www.google.co.in opens up and gets closed at the end of the test.
On completion of the Test Execution you will find the following files within your work space which will give you all the details of the Test Execution :
```
bmp.log
geckodriver.log
server.log
```
Snapshot :

Demetria answered 15/1, 2018 at 16:5 Comment(6)

Why is all of this not part of the original 'browsermob-proxy' documentation? How should I know what and how to install??? – Katey 16/1, 2018 at 6:29

In 'server.log'/'bmp.log' I got an error: Failed to start Jetty server. – Katey 16/1, 2018 at 6:56

And what are the three log files for? Where can I see a list of connections being made? – Katey 16/1, 2018 at 6:56

Thanks again for the extensive answer, but its still not working for me. See #48315938 for a complete description of the error I get. – Katey 18/1, 2018 at 7:4

Do you have any idea on how to capture responses? – Constantan 6/9, 2018 at 9:19

@undetectedSelenium how to use this code with chrome? – Hallucination 14/1, 2023 at 14:56

You can try the code below:

    from browsermobproxy import Server
    import psutil
    import time
    
    for proc in psutil.process_iter():
        # check whether the process name matches
        if proc.name() == "browsermob-proxy":
            proc.kill()
    
    dict = {'port': 8090}
    server = Server(path="./BrowserMobProxy/bin/browsermob-proxy", options=dict)
    server.start()
    time.sleep(1)
    proxy = server.create_proxy()
    time.sleep(1)
    from selenium import webdriver
    profile = webdriver.FirefoxProfile()
    selenium_proxy = proxy.selenium_proxy()
    profile.set_proxy(selenium_proxy)
    driver = webdriver.Firefox(firefox_profile=profile)
    
    
    proxy.new_har("google")
    driver.get("http://www.google.co.uk")
    print (proxy.har) # returns a HAR JSON blob
    
    server.stop()
    driver.quit()

Two things, if your code fails the process could be left open sometimes. So I added the code below for closing the duplicate instances.

import psutil
import time

for proc in psutil.process_iter():
    # check whether the process name matches
    if proc.name() == "browsermob-proxy":
        proc.kill()

Also a sleep of 1 second before and after creating the proxy.

server.start()
time.sleep(1)
proxy = server.create_proxy()
time.sleep(1)

This helps in getting rid of some intermittent issues which can be faced due to the server taking some time to start.

Cornwallis answered 22/1, 2018 at 17:47 Comment(4)

Thanks a lot. Your code suggestion actually did work! – Katey 23/1, 2018 at 7:8

@DhneshDhingra, when you do server.start() it will choose a free port and two process will come. But if a browser is opened manually then this will have no impact – Cornwallis 7/10, 2018 at 7:53

@Tarun actually if proc.name() == "browsermob-proxy": proc.kill(), woulndt be all the processes killed? – Piccalilli 7/10, 2018 at 18:40

This problem is related to the fact, that installation of browsermob-proxy is not just doing:

pip install browsermob-proxy

After you executed the code above, you need to go to https://bmp.lightbody.net, download the zip, unzip it and then when calling Server() in your python script, identify path to the executable from the zip you just created. It is located in "bin" folder and is called browsermob-proxy.

In my case it was:

server = Server("/anaconda3/lib/python3.7/site-packages/browsermobproxy/browsermob-proxy-2.1.4/bin/browsermob-proxy")

Dorelle answered 4/5, 2019 at 21:48 Comment(0)

You need to configure the driver to use BMP as a proxy so it can record the network activity. Here is an example....

from browsermobproxy import Server
from selenium import webdriver

server = Server('/path/to/bmp/bin/browsermob-proxy') #Local path to BMP
server.start()
proxy = server.create_proxy() #Proxy is used to generate a HAR file containing the connection URLS that the MP3s are loaded from.
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy)) #Configure chrome options
driver = webdriver.Chrome(chrome_options=chrome_options)
proxy.new_har('filename')

Then all the activity will be recorded to that file.

In my case, the path for the binary file was C:\Python27\Lib\site-packages\browsermobproxy\browsermob-proxy-2.1.0-beta-3\bin\browsermob-proxy on Windows with Python 2.7

Unaesthetic answered 11/1, 2018 at 7:44 Comment(7)

What does '/path/to/bmp/bin/browsermob-proxy' mean? What is that? How to figure out what to insert here? – Katey 11/1, 2018 at 7:45

Updated my answer with where I found it. I believe it was put there when I did pip install. – Unaesthetic 11/1, 2018 at 7:47

Ah so the documentation is incomplete. It does not say how to install that 'browsermob-proxy' not does it tell you you need to install something in addition – Katey 11/1, 2018 at 7:47

Could be. I recall this not being an easy thing to figure out. I pulled this out of some really old code I wrote that did this. – Unaesthetic 11/1, 2018 at 7:49

ProxyServerError: The Browsermob-Proxy server process failed to start. Check <open file '/home/adietz/Projects/Jenkins/bsp-usecase-tests/server.log', mode 'w' at 0x7fc9d3012db0>for a helpful error message. – Katey 11/1, 2018 at 7:50

But I have nothing else – Katey 11/1, 2018 at 7:51

Its a shell script – Katey 11/1, 2018 at 7:52

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

BrowserMob Proxy

Demonstration of BrowserMob Proxy 2.0 with Python client on Windows

Recommended topics

Hot tags

`BrowserMob Proxy`