How to use Chrome DevTools protocol in Selenium (using Python) for capturing HTTP requests and responses?

Question

I know that Fetch Domain is used for this purpose but I do not know how exactly I can implement it. In Selenium python, I used the following code to enable issuing of requestPaused events.

driver.execute_cdp_cmd("Fetch.enable",{})
driver.get('https://www.example.com')

But I do not know how can I handle requestPaused event (I need to call one fulfillRequest or continueRequest/continueWithAuth). As a result, my program stops working. I really appreciate it if anyone could provide me an example to help me understand how it works.

Tamarau

I'm not sure it can be done. Anyway most requests will still use xhr rather than fetch. – Cobos

see also Selenium 4: how add event listeners in CDP – Shock

the problem is that CDP does not expose streams, so you cannot build a generic http client, see also cdp: Fetch.fulfillRequest and (very) long body and selenium_driverless: network interception with Fetch.enable breaks cloudflare and my aiohttp_chromium – Shock

Answer 1

Yes, you saw it right.

As per the release notes of Selenium v4.0.0-alpha-3:

* Expose devtools APIs from chromium derived drivers.
* Expose presence of devtools support on a role-based interface

As per the release notes of Selenium v4.0.0.0-alpha-1:

* Basic support for CDP landed via the "DevTools" interface.

So chrome-devtools-protocol is all set to be available with selenium4 which will allow for tools to instrument, inspect, debug and profile Chromium, Chrome and other Blink-based browsers. In the discussion Controlling Chrome Devtools with Selenium Webdriver @AdiOhana mentions of the example usage of a few commands from the Profiler Domain as follows:

    driver.getDevTools().createSession();
    driver.getDevTools().send(new Command("Profiler.enable", ImmutableMap.of()));
    driver.getDevTools().send(new Command("Profiler.start", ImmutableMap.of()));
    //register to profiler events
    driver.getDevTools().addListener(new Event("Profiler.consoleProfileStarted", ConsoleProfileStarted.class), new Consumer<Object>() {
        @Override
        public void accept(Object o) {
            //do something
        }
    });

Note: Until the Profiler domain will added to Selenium java client, you will have to supply your Mapper.

---

Fetch Domain

Fetch Domain will enable clients substitute browser's network layer with client code.

• The Fetch Domain methods are as follows:

  • Fetch.disable: Disables the fetch domain.
  • Fetch.enable: Enables issuing of requestPaused events. A request will be paused until client calls one of failRequest, fulfillRequest or continueRequest/continueWithAuth.
  • Fetch.failRequest: Causes the request to fail with specified reason.
  • Fetch.fulfillRequest: Provides response to the request.
  • Fetch.continueRequest: Continues the request, optionally modifying some of its parameters.
  • Fetch.continueWithAuth: Continues a request supplying authChallengeResponse following authRequired event.
  • Fetch.getResponseBody: Causes the body of the response to be received from the server and returned as a single string. May only be issued for a request that is paused in the Response stage and is mutually exclusive with takeResponseBodyForInterceptionAsStream. Calling other methods that affect the request or disabling fetch domain before body is received results in an undefined behavior.
  • Fetch.takeResponseBodyAsStream: Returns a handle to the stream representing the response body. The request must be paused in the HeadersReceived stage. Note that after this command the request can't be continued as is -- client either needs to cancel it or to provide the response body. The stream only supports sequential read, IO.read will fail if the position is specified. This method is mutually exclusive with getResponseBody. Calling other methods that affect the request or disabling fetch domain before body is received results in an undefined behavior.

• The Fetch Domain events are as follows:

  • Fetch.requestPaused: Issued when the domain is enabled and the request URL matches the specified filter. The request is paused until the client responds with one of continueRequest, failRequest or fulfillRequest. The stage of the request can be determined by presence of responseErrorReason and responseStatusCode -- the request is at the response stage if either of these fields is present and in the request stage otherwise.
  • Fetch.authRequired: Issued when the domain is enabled with handleAuthRequests set to true. The request is paused until client responds with continueWithAuth.

---

References

You can find a couple of revelant discussions in:

• Can Selenium WebDriver (java) interact with the Browser's inspect tool element selector?
• What is the difference between WebDriver and DevTool protocol

Jespersen

4.0 is still in beta as of now so you can try logging performance instead if you're using Senium 3.0. Related answer: https://mcmap.net/q/276815/-how-to-access-network-panel-on-google-chrome-developer-tools-with-selenium – Suzettesuzi

How is this selected as right answer? the question asks "using python" but the answer is in java. – Fibrinogen