Selenium-wire response object - Way to get response body as string rather than bytes

Question

I want to fetch a response body as string in selenium-wire which I will eventually parse as JSON.

response.body in selenium-wire gives bytes string. I tried decoding it as response.body.decode('utf-8') but this gives decoding error.

Can someone help me with this? I am fine with both solutions:

1. Either a way to decode the bytes string as normal string
2. A way to get response body as normal string in the first place

Nitride

Answer 1

I figured out a way to do this (not an ideal approach) While making the selenium webdriver object, you can pass a parameter options in which you can tell it explicitly to give decoded request and response objects and not as bytes.

Nitride

Hi! I have the same problem with decoding the responses. May I ask, what parameter in options did you use/set so that the WebDriver would pass the objects ? – Ithyphallic

options = {'disable_encoding': True} – Nitride

Answer 2

By default selenium-wire returns body response as bytes.

The documentation says:

"The response body as bytes. If the response has no body the value of body will be empty, i.e. b''. Sometimes the body may have been encoded by the server - e.g. compressed. You can prevent this with the disable_encoding option. To manually decode an encoded response body you can do:

from seleniumwire.utils import decode

body = decode(response.body, response.headers.get('Content-Encoding', 'identity'))

And it works for me.

Predikant

Answer 3

put this:

decode(request.response.body, request.response.headers.get('Content-Encoding', 'identity'))

before you put the statement:

response.body.decode('utf-8')

the full code:

from seleniumwire import webdriver
from seleniumwire.utils import decode as sw_decode

browser = webdriver.Chrome()

browser.get(url)

for request in browser.requests:
    if request.url == url:
        data = sw_decode(request.response.body, request.response.headers.get('Content-Encoding', 'identity'))
        data = data.decode("utf8")
        print(type(data))
        break

browser.quit()

the output:

<class 'str'>

Walt

Answer 4

I would like to share my solution, It's work for me.

In python >= 3.5.x

from seleniumwire import webdriver
import chromedriver_autoinstaller
import brotli

chromedriver_autoinstaller.install()
driver = webdriver.Chrome()

driver.get('https://www.facebook.com')

for request in driver.requests:
    if request.url == "https://www.facebook.com/":
        resp = request.response.body
        resp = brotli.decompress(resp)
        print(resp[0:200].decode("utf-8"))

driver.quit()

Schubert

I don't want to hardcode brotlyi into my code. What if it was gzip for some reason? – Bloodhound

Answer 5

None of these are needed.

Add a translation to the end of the variable you get and it's done.

My code ex:

requestBody = ''
for request in driver.requests:
    if request.response:
        if request.url == 'https://api.ex.com/v1':
            requestBody = request.body.decode("utf-8")

My solution:

request.body.decode("utf-8")

Dygall