How to get transferred size of a complete page load?

Question

With Selenium or JavaScript how could you get the (over the network) transferred size (bytes) of the loaded page including all the content, images, css, js, etc?

The preferred size is that of what goes over the network, that is compressed, only for the requests that are made, etc.

This is what you usually can see in dev tools, to the right in the network status bar:

 

If that's not possible, could one just get a total size of all the loaded resources (without compression, etc)? That would be an acceptable alternative.

The browser is Firefox, but if it could be done with some other Selenium compatible browser that would be acceptable also.

 

I guess this could be done using a proxy, but is there any JS or Selenium way to get such information?

If proxy is the only way, which one would one use (or implement) to keep things simple for such a task? Just implementing something in Java before setting up the driver?

(The solution should work at least on Linux, but preferably on Windows also. I'm using Selenium WebDriver via Java.)

Bellringer

Neither Selenium nor JS have access to the wires (network). They only get to see the rendering engine (DOM). You will need something else to get that information. – Gerhard

Answer 1

For future reference, it is possible to request this information from the browser by javascript. However, at the time of writing no browser supports this feature for this specific data yet. More information can be found here.

In the mean time, for Chrome you can parse this information from the performance log.

    //Enable performance logging
    LoggingPreferences logPrefs = new LoggingPreferences();
    logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
    capa.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);

    //Start driver
    WebDriver driver = new ChromeDriver(capa);

You can then get this data like this

for (LogEntry entry : driver.manage().logs().get(LogType.PERFORMANCE)) {
        if(entry.getMessage().contains("Network.dataReceived")) {
            Matcher dataLengthMatcher = Pattern.compile("encodedDataLength\":(.*?),").matcher(entry.getMessage());
            dataLengthMatcher.find();
            //Do whatever you want with the data here.
        }

If, like in your case, you want to know the specifics of a single page load, you could use a pre- and postload timestamp and only get entries within that timeframe.

Schwaben

Do you think it will give the exact total page size which listed in chrome dev tools network section as .... byte transferred? – Asdic

Answer 2

The performance API mentioned in Hakello's answer is now well supported (on everything except IE & Safari), and is simple to use:

return performance
  .getEntriesByType("resource")
  .map((x) => x.transferSize)
  .reduce((a, b) => (a + b), 0);

You can run that script using executeScript to get the number of bytes downloaded since the last navigation event. No setup or configuration is required.

Lope

no performance api never returns the right answer , you can not get the exact info with dev tools. Never apply this method. – Asdic

@Asdic may be right - however, that caveat aside, this only returns the actual bytes transferred and does not include resources that were cached previously. If your goal is to check the ensite page weight of the page as if nothing has been cached, replace transferSize with decodedBodySize. This will ensure that all of the entires calculated are as if a user navigated to the page for the first time. – Weston

Code should be like that, it gives the exact content size which is displayed on developer tools: @Weston const entry = performance.getEntriesByType("navigation") .find(({ name }) => name === location.href); var pageSizeLength = entry.decodedBodySize; console.log(pageSizeLength); – Asdic

Answer 3

Yes you can do it using BrowserMobProxy. This is a java jar which use selenium Proxy to track network traffic from client side. like page load time duration, Query string to different services etc. you can get it bmp.lightbody.net . This api will create .har files which will contain all these information in json format which you can read using an online tool http://www.softwareishard.com/har/viewer/

Epanaphora

Answer 4

I have achieved this in Python, which might save people some time. To setup the logging:

logging_prefs = {'performance' : 'INFO'}    
caps = DesiredCapabilities.CHROME.copy()
caps['loggingPrefs'] = logging_prefs
driver = webdriver.Chrome(desired_capabilities=caps)

To calculate the total:

total_bytes = []
for entry in driver.get_log('performance'):
        if "Network.dataReceived" in str(entry):
            r = re.search(r'encodedDataLength\":(.*?),', str(entry))
            total_bytes.append(int(r.group(1)))
            mb = round((float(sum(total_bytes) / 1000) / 1000), 2)

Naaman

Do you think accumulation will be equal to data showed on network tab as transferred size? (total transferred size) F.e I see 50 entry in driver.get_log('performance'): but I see 20 request on network tab in dev tools. How this difference come from? so we can not rely on this code? – Asdic

Answer 5

I found the solution (chrome + firefox):

const entry = performance.getEntriesByType("navigation")
  .find(({ name }) => name === location.href);

  var pageSizeLength = entry.decodedBodySize;
  console.log(pageSizeLength);

Asdic

This only includes the document and not any of the resources when tested in Chrome. The question was how to get the size including all of the loaded resources over the network. – Bellringer

Yes you are right, I asked the question in fact but the code I have sent as answer is the one looking for, I asked the question wrongly. – Asdic