C# Selenium 4: Setup request interception

Question

I'm trying to use Selenium 4 to log requests during manual usage of Chrome browser.

The issue is that request interception stops after around 40 seconds of usage (approximately).

I've tried to change commandTimeout but it didn't change anything. Also I've tried to look into chromedriver logs but I didn't find anithing there.

Here's my code:

static void Main(string[] args)
    {
        // Enable chromedriver logging
        var service = ChromeDriverService.CreateDefaultService();
        service.LogPath = AppDomain.CurrentDomain.BaseDirectory + "chromedriver.log";
        service.EnableVerboseLogging = true;
        var options = new ChromeOptions();

        var webDriver = new ChromeDriver(service, options);
        var devToolsSession = webDriver.CreateDevToolsSession();
        devToolsSession.Network.Enable(new EnableCommandSettings());

        EventHandler<RequestInterceptedEventArgs> requestIntercepted = (sender, e) =>
        {
            Console.WriteLine(e.Request.Url);
        };

        RequestPattern requestPattern = new RequestPattern();
        requestPattern.InterceptionStage = InterceptionStage.Request;
        requestPattern.ResourceType = ResourceType.Image;
        var setRequestInterceptionCommandSettings = new SetRequestInterceptionCommandSettings();
        setRequestInterceptionCommandSettings.Patterns = new RequestPattern[] { requestPattern };
        devToolsSession.Network.SetRequestInterception(setRequestInterceptionCommandSettings);
        devToolsSession.Network.RequestIntercepted += requestIntercepted;

        while (true)
        {
            webDriver.Url = "https://translate.google.com/";
            Thread.Sleep(5000);
            webDriver.Navigate().Refresh();
        }
    }

Tucana

Ever solve this? I'm not seeing what I would normally see in the Network tab within the chromedriver.log but maybe it's the same 40 second thing you are running into. – Sandasandakan

No luck from my side for now. I've created issue on Selenium github: github.com/SeleniumHQ/selenium/issues/8960 Maybe they will tell us something – Tucana

There is some working solutions on GitHub (link above) but I don't see them useful because of lack of api documentation and chome version dependency. For now I'm using python and pypi.org/project/selenium-wire. It's really easy to reach my goal with it. – Tucana

Answer 1

As of Selenium 4 beta-1, you can use the Fetch API and retrieve the same resluts (SetRequestIntercepted is deprecated in ChromeTools as late as December 29, 2020).

In either case, the key point for both RequestIntercepted (prior to deprecation) and Fetch to be able to continue after intercepting the request is to capture the RequestId.

With Fetch this can be done in ContinueRequest():

fetch.ContinueRequest(new Fetch.ContinueRequestCommandSettings()
{
   RequestId = e.RequestId
});

Below is your code updated to use Fetch which allowed the query to run for 2+ minutes and 5+ minutes before I manually stopped it:

(Note: ensure all of your DevTools as using the same version (i.e. 89 in my case) or you will get errors):

using V89 = OpenQA.Selenium.DevTools.V89;
using V89Net = OpenQA.Selenium.DevTools.V89.Network;
using OpenQA.Selenium.DevTools.V89.Log;

var service = ChromeDriverService.CreateDefaultService();
service.LogPath = AppDomain.CurrentDomain.BaseDirectory + "chromedriver.log";
service.EnableVerboseLogging = true;
var options = new ChromeOptions();

new DriverManager().SetUpDriver(new ChromeConfig(), VersionResolveStrategy.MatchingBrowser);

var driver = new ChromeDriver(service, options);

IDevTools devTools = driver as IDevTools;

var devToolsSession = devTools.GetDevToolsSession();

var fetch = devToolsSession.GetVersionSpecificDomains<V89.DevToolsSessionDomains>()
    .Fetch;
    
var enableCommandSettings = new V89.Fetch.EnableCommandSettings();

var requestPattern = new V89.Fetch.RequestPattern();
requestPattern.RequestStage = V89.Fetch.RequestStage.Response;
requestPattern.ResourceType = V89Net.ResourceType.Document;

enableCommandSettings.Patterns = new V89.Fetch.RequestPattern[] { requestPattern };

fetch.Enable(enableCommandSettings);

void RequestIntercepted(object sender, V89.Fetch.RequestPausedEventArgs e)
{

    e.Request.Url.Dump();

    fetch.ContinueRequest(new V89.Fetch.ContinueRequestCommandSettings()
    {
        RequestId = e.RequestId
    });
}

fetch.RequestPaused += RequestIntercepted;

while (true)
{
    driver.Url = "https://translate.google.com/";
    Thread.Sleep(5000);
    driver.Navigate().Refresh();
}

Screenshot (From left-to-right): LinqPad output Console.Write.Url equivalent, Above code in LINQPad, and view of Chrome log file)

Phosphorus

In My instance I needed to receive the Websocket connection and frames. I ended up using Session.GetVersionSpecificDomains<OpenQA.Selenium.DevTools.V119.DevToolsSessionDomains>().Network which contained the API's for WebSocketCreated and WebSocketFrameReceived. – Matriarchy

fetch.ContinueRequest is an awaitable. Can not it cause concurrency issues we do not await for it to be executed? – Flay