How to access Network panel on google chrome developer tools with selenium?

Question

I want to get the output that is shown on the network panel of the developer tools.

[Network panel --> Name, Method, Status, Type, Initiator, Size, Time, Timeline]

I need this information.

Platinum

Use browsermop-proxy for this – Mantle

Answer 1

This possible via Selenium WebDriver. For this you should do the following:

1. Download selenium language-specific client drivers from - http://docs.seleniumhq.org/download/ and add apropriate jar files to your project build path.

2. To run a test with Chrome/Chromium you will also need chromdriver binary which you can download from - http://chromedriver.storage.googleapis.com/index.html

3. Create a test case like this:

    // specify the path of the chromdriver binary that you have downloaded (see point 2)
    System.setProperty("webdriver.chrome.driver", "/root/Downloads/chromedriver");
    ChromeOptions options = new ChromeOptions();
    // if you like to specify another profile
    options.addArguments("user-data-dir=/root/Downloads/aaa"); 
    options.addArguments("start-maximized");
    DesiredCapabilities capabilities = DesiredCapabilities.chrome();
    capabilities.setCapability(ChromeOptions.CAPABILITY, options);
    WebDriver driver = new ChromeDriver(capabilities);
    driver.get("http://www.google.com");
    String scriptToExecute = "var performance = window.performance || window.mozPerformance || window.msPerformance || window.webkitPerformance || {}; var network = performance.getEntries() || {}; return network;";
    String netData = ((JavascriptExecutor)driver).executeScript(scriptToExecute).toString();

Executing javascript on Chrome/Chromium will help you to get the networking (not only) info. The resulting string 'netData' will contain the required data in JSONArray format.

Hope this will help.

Om

thanks it works pretty well. Is there any option which could enable us to get the content transferred size from network panel ? – Botvinnik

How can you get the total size transferred? Adding up transferSize is inaccurate – Surrebuttal

Beware this value may not be accurate due to CORS, developer.mozilla.org/en-US/docs/Web/API/Resource_Timing_API/… – Surrebuttal

There is not network. There is performance. What about the information about blocked resources? – Suspender

When i try this i can able to get the first 150 network responses, how i can get entire response details. – Practice

Answer 2

From this answer.

You can use the LoggingPreferences to get the Performance logs. It returns the data in json format. Here is a sample java code. Tested this with selenium 2.53, chromedriver 2.20, Chrome 50 on Ubuntu 14.04. This should work on windows also.

    DesiredCapabilities d = DesiredCapabilities.chrome();
    LoggingPreferences logPrefs = new LoggingPreferences();
    logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
    d.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
    WebDriver driver = new ChromeDriver(d);
    driver.get("http://www.google.com");
    LogEntries les = driver.manage().logs().get(LogType.PERFORMANCE);
    for (LogEntry le : les) {
        System.out.println(le.getMessage());
    }

Here is a sample output. It is formatted manually. The actual ouput is in a single line.

{
    "message": {
        "method": "Network.requestWillBeSent",
        "params": {
            "documentURL": "https://www.google.co.in/?gfe_rd=cr&ei=gpwxV4OSKMmR2ASEg6-YCg&gws_rd=ssl",
            "frameId": "31172.2",
            "initiator": {
                "stack": {
                    "callFrames": [
                        {
                            "columnNumber": 11511,
                            "functionName": "",
                            "lineNumber": 55,
                            "scriptId": "50",
                            "url": "https://www.google.co.in/?gfe_rd=cr&ei=gpwxV4OSKMmR2ASEg6-YCg&gws_rd=ssl"
                        }
                    ]
                },
                "type": "script"
            },
            "loaderId": "31172.3",
            "request": {
                "headers": {
                    "Accept": "*/*",
                    "Referer": "https://www.google.co.in/",
                    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
                },
                "initialPriority": "Low",
                "method": "GET",
                "mixedContentType": "none",
                "url": "https://www.google.co.in/xjs/_/js/k=xjs.s.en.VTDhrkH4c9U.O/m=sx,c,sb,cdos,cr,elog,jsa,r,hsm,qsm,j,p,d,csi/am=AJQ0CwoS8fchIGwhrCA1YGBR/rt=j/d=1/t=zcms/rs=ACT90oGi2YIjVL5cBzOc1-MD37a1NqZ1jA"
            },
            "requestId": "31172.3",
            "timestamp": 251208.074288,
            "type": "Other",
            "wallTime": 1462869123.92204
        }
    },
    "webview": "8AF4A466-8027-4340-B9E9-CFEBDA769C50"
}

Dehiscence

Is there a way to get the response body? – Grandiose

im looking for a way to get response body too. seems like its impossible. – Lincoln

Answer 3

Let's say you want to load a page (e.g. google.com) and extract array of resource timing objects (i.e. window.performance.getEntries()):

import time
from selenium import webdriver
driver = webdriver.Chrome('/path/to/chromedriver)
driver.get('https://www.google.com');
time.sleep(5)
timings = driver.execute_script("return window.performance.getEntries();")
print timings

Aim

Answer 4

As noted in other answers, you need to utilize window.performance methods.

getEntries()
getEntriesByType()
getEntriesByName()
» Entry Types

For instance, I've used the following snippet within a nodejs Selenium-WebDriver: Chromedriver test to collect google analytics network calls:

driver
  .executeScript( "return window.performance.getEntriesByType('resource');" )
  .then( (perfEntries)=> {
    let gaCalls = perfEntries.filter(function(entry){
      return /collect\?/i.test(entry.name);
    });
    console.log(gaCalls);
});

If you used getEntries() instead of getEntriesByType('resource'), it would return...all entries.

Unready

Thanks for this! I used the script window.performance.getEntriesByName('<exact request name>') to count the particular number of API calls going out. This was the most helpful! – Colincolinson

Answer 5

With Python, one way to do it is:

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import chromedriver_binary # If you're using conda like me.

yoururl = "www.yoururl.com"

caps = DesiredCapabilities.CHROME
caps['goog:loggingPrefs'] = {'performance': 'ALL'}
driver = webdriver.Chrome(desired_capabilities=caps)

driver.get(yoururl)
time.sleep(10) # wait for all the data to arrive. 
perf = driver.get_log('performance')

perf is a list of dictionaries and you'll be able to find the item you're looking for in this list. i.e, dictionaries are the things that you see in Chrome's dev tool network tab.

Boanerges

One more thing: From Selenium 4.0 onward it will officially support accessing Chrome devtool network tab information but it's still in beta state. Check out Selenium repo for more info and if you want, try using the beta for the same purpose. – Boanerges

selenium4 encourages you to use the bidi api: selenium.dev/documentation/webdriver/bidirectional/bidi_api needs lots of improvement – Lavinia