How do I get the response headers in Selenium?

Question

How can I get request/response logs in selenium? I have an ajax call that returns login information, and whenever I try to capture it via:

 selenium.captureNetworkTraffic("json");

it returns only client-side items (like images .pn), but not the actual JSON response I'm interested in.

Turbellarian

can any one help me,i thought selenium should obviously have clicks made inside and response from it?? didnt any one tried this?? – Turbellarian

Answer 1

Unfortunately, it doesn't look like you can do that with selenium alone in an automated fashion, as this is against their design principles for selenium. I don't agree that it is (since it supports getting cookie information -- what user actually goes through and parses through their cookies?).

Their 'schtick' is that selenium helps you automate what a user does; since a user doesn't care about the response headers, why would this framework? For me, this is disappointing because I need to ensure the content type being returned is what I believe it should be (to test an API). It appears another Stack Overflow question has come to a similar conclusion.

Samoyedic

Answer 2

In case someone is still looking for an answer, there is a way to get the response headers using Selenium 4 and Chrome developer tools API

DevTools devTools = driver.getDevTools();
devTools.createSession();
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));
devTools.addListener(Network.responseReceived(),
        responseReceived -> {
            Headers headers = responseReceived.getResponse().getHeaders();
            if (!headers.isEmpty()){
                System.out.println("Headers:");
                headers.forEach((key,value) -> {
                    System.out.println("    " + key + "="+ value);
                });
            }
        });

This was tested using Selenium 4 alpha 6

Retrad

Not just ChromeDriver... Technically any of the drivers implementing HasDevTools (which appears to at least include firefox and edge). – Sudorific

oh petassakes, add the corresponding imports !!!!!! – Courtroom

Answer 3

If you are looking for other way round solution it could be also done using proxy server which will output log file including headers for each request/response which go through the proxy.

I have used Titanium Web Proxy and registered proxy for driver options like this:

ChromeOptions chromeOptions = new ChromeOptions
{
    Proxy = new Proxy
    {
        Kind = ProxyKind.Manual,
        HttpProxy = "localhost:8888"
    }
};

Bramwell

I upvoted, I did not check whether it actually works. But I think the idea is worth a vote nonetheless. – Quartis

Answer 4

I realize that this question is old as hell, but I've developed a way to do this for firefox + selenium by modifying webdriver.xpi and creating driver.http_logger.variousLogMethods(). Please reply to this post if there's interest in this, and I can post the code to github.

Westbrook

I am interested. Please post the github link. – Orthodontia

@user2426679 I'm interested as well, could you please show me on Github how to execute this? – Tyburn

@Tyburn -- I finally got around to uploading this little project. I did this a while back, so the documentation is sparse. bitbucket.org/stbrownlow/seleniumwithfirefoxhttplogging/src – Westbrook

Interesting. Has anyone been able to get this to work on newer versions of FF? – Baese

@Baese it won't work without rewriting the patch within webdriver.xpi. See the readme in the BitBucket for locating the edits I made to the .xpi selenium extension, as a starting point for porting to a newer version of Firefox. – Westbrook

Answer 5

You can do it by injecting a XMLHttpRequest and waiting for the result:

Object response = ((JavascriptExecutor) driver).executeAsyncScript(
       "var callback = arguments[arguments.length - 1];" +
       "var xhr = new XMLHttpRequest();" +
       "xhr.open('GET', '/resource/data.json', true);" +
       "xhr.onreadystatechange = function() {" +
       "  if (xhr.readyState == 4) {" +
       "    callback(xhr.getResponseHeader());" +
       "  }" +
       "};" +
       "xhr.send();");
   JsonObject json = new JsonParser().parse((String) response);
   assertEquals("cheese", json.get("food").getAsString());

You can modify the xhr callback to get what you want.

Mangosteen

Answer 6

I found a trick if you use jquery you can register a global listener :

First you need to declare a global variable in your main script :

var globalVariableForSelenium = "";

In your selenium you register the listener :

if (driver instanceof JavascriptExecutor) {
                ((JavascriptExecutor) driver)
                .executeScript("$(document).ajaxSuccess(function(event, xhr, settings) {  globalVariableForSelenium = settings.headers    });");
            }

Then you execute your action

Finaly you get the value of the global variable :

if (driver instanceof JavascriptExecutor) {
            Object res = ((JavascriptExecutor) driver)
            .executeScript("return  serializeGlobalvariable();");

}

serializeGlobalvariable() is a function that will serialize globalVariableForSelenium which contains an array of header.

ps: my need was to access to the ajax response so I set the global variable with xhr.responseJSON.mypath

Prolusion