Using Selenium get network request not working properly [closed]

Question

I want to take all the network requests using selenium. I am not getting any way to find this solution if anyone can suggest to me or provide a code or library that would be appreciated.

This is my code:

ChromeOptions options = new ChromeOptions();
// Setting some chrome features here

ProxyServer proxyServer = new ProxyServer(4444);
proxyServer.start();

Proxy proxy = proxyServer.seleniumProxy();

DesiredCapabilities capabilities = DesiredCapabilities.chrome();

capabilities.setCapability(ChromeOptions.CAPABILITY, options);
capabilities.setCapability(CapabilityType.PROXY, proxy);

WebDriver driver = new ChromeDriver(capabilities); // Error happens here

Weakkneed

are you looking for this: openhub.net/p/selenium-profiler ? – Ellison

another way - using proxy for capture all network activity, in fast search I found this: github.com/lightbody/browsermob-proxy – Ellison

example for python gist.github.com/mkaz/3047779 and docs w3.org/TR/navigation-timing/#processing-model – Priam

@AlexeyShrub any way to get status code from XHR using python? – Alliber

@zakaria-shahed - Any reason you have removed by accepted answer after 7 years since 2017? – Praetor

Answer 1

Not exactly open by dev tools but found some network, performance and other results.

Yes you can do that using JavascriptExecutor

Below Code will give you all performance, network etc entries:-

ChromeOptions options = new ChromeOptions();
options.addArguments("start-maximized");
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(capabilities);
driver.get("http://www.google.com");
String scriptToExecute = "var performance = window.performance || window.mozPerformance || window.msPerformance || window.webkitPerformance || {}; var network = performance.getEntries() || {}; return network;";
String netData = ((JavascriptExecutor)driver).executeScript(scriptToExecute).toString();
System.out.println(netData);

OR Below Code will give you specific performance entries:-

DesiredCapabilities d = DesiredCapabilities.chrome();
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
d.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
WebDriver driver = new ChromeDriver(d);
driver.get("https://www.google.co.in/");
driver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
LogEntries les = driver.manage().logs().get(LogType.PERFORMANCE);
for (LogEntry le : les) {
    System.out.println(le.getMessage());
}

The first code retrun network return network;" because of this JS tag. You can remove JS code of entity which you don't require

The second code return perfromance

Hope it will help you :)

Praetor

The output from the first code example isn't valid JSON (it's missing quotation marks, mostly). In order to fix that and get valid JSON output, instead of return network; from within the Javascript string, return JSON.stringify(network); will output valid JSON with the quotes and all. – Dartboard

Thanks for sharing @George Pantazes ... I will try and update it soon – Praetor

Another caveat with the Javascript solution: getting the network requests via performance entries will only give the network requests at/up to page load and cannot poll for subsequent async/ajax calls. I believe it is the same for the logging solution since it's also logging Performance (although I haven't tried the logging code). @ShubhamJain for my own education, can you confirm if the two snippets both get Performance requests? I think given the limitations of using Performance, I'm going to use BrowserMobProxy instead. – Dartboard

I tried to use this method but I think it's not a complete log. Is there any method to have a more complete network log? for example the type of requests, if the responses were successful ... ? – Azurite

Answer 2

It's working for me

ChromeOptions options = new ChromeOptions();
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable( LogType.PERFORMANCE, Level.ALL );
options.setCapability( "goog:loggingPrefs", logPrefs );
WebDriverManager.chromedriver().setup();
WebDriver driver = new ChromeDriver(options);
driver.get("http://www.google.com");
    
List<LogEntry> entries = driver.manage().logs().get(LogType.PERFORMANCE).getAll();
System.out.println(entries.size() + " " + LogType.PERFORMANCE + " log entries found");
 for (LogEntry entry : entries) {
   System.out.println(entry.getMessage());
 }

Bosh

Answer 3

1. You can use "browsermob-proxy", "LoggingPreferences", "CloseableHttpClient", "HttpURLConnection" for getting the logs
2. If you wish not to use browser and want to get the response, then I would suggest to go for "CloseableHttpClient".
3. Copy the URI ("www.somewebsite.com/v1/api/sign-in?"). Get the request payload(which will be available in that particular API URI). Pass all the parameters with "&" like this "www.somewebsite.com/v1/api/sign-in?&username=xyz&password=1234566&app_id=12123214324234134&app_secret=213242345345345" (Remember app id and app secret is very unique and do not expose it anywhere)
4. Once you get the URI, this code will give you JSON format response

            HttpPost request = new HttpPost(str);
            request.setHeader("content-type", "application/json");
            HttpResponse response = client.execute(request);
            BufferedReader bufReader = new BufferedReader(new InputStreamReader(
                    response.getEntity().getContent()));
            while ((line = bufReader.readLine()) != null) {
                builder=String.valueOf(line);
            }

            System.out.println(builder);
        }

Sarene

The browsermob proxy hasn't been maintained for a couple years. We just released an actively maintained fork named the BrowserUp Proxy. It has the same API as the BrowserMob proxy, but also has Java 11 support, HTTP/2 support, updated dependencies, and people caring for it. – Knitted