How can I see the entire HTTP request that's being sent by my Python application?

Question

In my case, I'm using the requests library to call PayPal's API over HTTPS. Unfortunately, I'm getting an error from PayPal, and PayPal support cannot figure out what the error is or what's causing it. They want me to "Please provide the entire request, headers included".

How can I do that?

Ratable

Answer 1

A simple method: enable logging in recent versions of Requests (1.x and higher.)

Requests uses the http.client and logging module configuration to control logging verbosity, as described here.

Demonstration

Code excerpted from the linked documentation:

import requests
import logging

# These two lines enable debugging at httplib level (requests->urllib3->http.client)
# You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# The only thing missing will be the response.body which is not logged.
try:
    import http.client as http_client
except ImportError:
    # Python 2
    import httplib as http_client
http_client.HTTPConnection.debuglevel = 1

# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

requests.get('https://httpbin.org/headers')

Example Output

$ python requests-logging.py 
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): httpbin.org
send: 'GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nAccept-Encoding: gzip, deflate, compress\r\nAccept: */*\r\nUser-Agent: python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-48-generic\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Date: Sat, 29 Jun 2013 11:19:34 GMT
header: Server: gunicorn/0.17.4
header: Content-Length: 226
header: Connection: keep-alive
DEBUG:requests.packages.urllib3.connectionpool:"GET /headers HTTP/1.1" 200 226

Marzipan

Thanks, @EmmettJ.Butler =) Though I'm not sure this info was available at the time of the original inquiry. – Marzipan

Note that httplib isn't available on Python 3. To make the code portable, replace import httplib with import requests.packages.urllib3.connectionpool as httplib or use six and from six.moves import http_client as httplib. – Apnea

In requests 2.18.1 and Python 3, the logger logging.getLogger("requests.packages.urllib3") doesn't exist or has no effect. – Elva

for Python3 see here - docs.python-requests.org/en/latest/api/?highlight=debug from http.client import HTTPConnection – Sporogonium

Unfortunately "send:" "reply:" and "header:" lines are not actually logged, but just printed to stdout. But I want to have this info in the log files! – Nitwit

The link posted by @Sporogonium no longer works. This appears to be the current replacement: requests.readthedocs.io/en/master/api/… – Flycatcher

to have complete logs, use 'hooks', see my answer further below – Florenceflorencia

Thank you!! You saved me. I needed this to replicate the functionality in a different environment. Nothing worked!! I was soo frustrated I even went through the source code of the requests library but I got lost in all the function calls. – Epicycloid

Answer 2

r = requests.get('https://api.github.com', auth=('user', 'pass'))

r is a response. It has a request attribute which has the information you need.

r.request.allow_redirects  r.request.headers          r.request.register_hook
r.request.auth             r.request.hooks            r.request.response
r.request.cert             r.request.method           r.request.send
r.request.config           r.request.params           r.request.sent
r.request.cookies          r.request.path_url         r.request.session
r.request.data             r.request.prefetch         r.request.timeout
r.request.deregister_hook  r.request.proxies          r.request.url
r.request.files            r.request.redirect         r.request.verify

r.request.headers gives the headers:

{'Accept': '*/*',
 'Accept-Encoding': 'identity, deflate, compress, gzip',
 'Authorization': u'Basic dXNlcjpwYXNz',
 'User-Agent': 'python-requests/0.12.1'}

Then r.request.data has the body as a mapping. You can convert this with urllib.urlencode if they prefer:

import urllib
b = r.request.data
encoded_body = urllib.urlencode(b)

depending on the type of the response the .data-attribute may be missing and a .body-attribute be there instead.

Berhley

Which of these gives me "the entire request, headers included"? – Ratable

added some more. What else do you need besides the headers and the body? – Berhley

I'm not entirely sure what they're looking for. I was hoping to capture everything that went over the wire for them, in that exact format, byte-for-byte. – Ratable

Same feeling as Chris. And I end up using @Marzipan 's answer. It is good. Upvoted. – Grison

I think this doesn't work if the initial response is a redirect. – Aircondition

I think for most cases, this is the right way to go. If you just want to see the headers, print the headers. But for the OP's particular needs, debug tracing might be justified. – Cephalochordate

This is the preferred way of doing it in my case. Only one note: the response.request seems to be a PreparedRequest in my case; it doesn't have .data but .body instead. – Wry

r.request.path_url worked best for me (full_url no longer seems to exist). – Degas

for the full URL (with the querystring parameters) you can also use response.url (which is a bit different in that it's not response.request... – Molech

How come I can't see all the request attribute you shown @Skylar ? vars(r.request) {'_body_position': None, '_cookies': <RequestsCookieJar[]>, 'body': None, 'headers': {'User-Agent': 'python-requests/2.18.4', 'Connection': 'keep-alive', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate'}, 'hooks': {'response': []}, 'method': 'GET', 'url': 'myUrl..'} requests==2.18.4I am looking if actually sent request via specified proxy – Bigoted

Answer 3

You can use HTTP Toolkit to do exactly this.

It's especially useful if you need to do this quickly, with no code changes: you can open a terminal from HTTP Toolkit, run any Python code from there as normal, and you'll be able to see the full content of every HTTP/HTTPS request immediately.

There's a free version that can do everything you need, and it's 100% open source.

I'm the creator of HTTP Toolkit; I actually built it myself to solve the exact same problem for me a while back! I too was trying to debug a payment integration, but their SDK didn't work, I couldn't tell why, and I needed to know what was actually going on to properly fix it. It's very frustrating, but being able to see the raw traffic really helps.

Contestant

amazing work buddy! this just helped me a lot after hours of testing and unsuccessfully fiddling with fiddler... – Cleromancy

Answer 4

A much simpler way to debug HTTP local requests is to use netcat. If you run

nc -l 1234

you'll start listening on port 1234 for HTTP connections. You can access it via http://localhost:1234/foo/foo/....

On the terminal, you'll see whatever raw data you sent to the endpoint. For example:

POST /foo/foo HTTP/1.1
Accept: application/json
Connection: keep-alive
Host: example.com
Accept-Language: en-en
Authorization: Bearer ay...
Content-Length: 15
Content-Type: application/json

{"test": false}

Vender

With gnu-netcat I had to use nc -l -p 1234. – Sherborne

Answer 5

No logging system completely works, (as of requests 2.26 anyway, very old versions had maybe another behaviour)

The good solution is to use 'hooks' and print details as they happen.

This is fairly well explained here : https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/

under "printing everything",

but in case the link dies here is the important parts

import requests
from requests_toolbelt.utils import dump

def logging_hook(response, *args, **kwargs):
    data = dump.dump_all(response)
    print(data.decode('utf-8'))

http = requests.Session()
http.hooks["response"] = [logging_hook]

http.get("https://api.openaq.org/v1/cities", params={"country": "BA"})

The result this time will be a complete trace of sent query and received response.

I've tried it successfully with POST and lots of headers : it works. Don't forget to pip install requests_toolbelt.

# Output
< GET /v1/cities?country=BA HTTP/1.1
< Host: api.openaq.org

> HTTP/1.1 200 OK
> Content-Type: application/json; charset=utf-8
> Transfer-Encoding: chunked
> Connection: keep-alive
>
{
   "meta":{
      "name":"openaq-api",
      "license":"CC BY 4.0",
      "website":"https://docs.openaq.org/",
      "page":1,
      "limit":100,
      "found":1
   },
   "results":[
      {
         "country":"BA",
         "name":"Goražde",
         "city":"Goražde",
         "count":70797,
         "locations":1
      }
   ]
}

Florenceflorencia

Answer 6

A previous answer seems to have been downvoted because it started with "nothing completely works" and then provides this perfect solution:

1. Install the requests_toolbelt collection of utilities with pip install requests-toolbelt.
2. Use it like so:

   import requests
   from requests_toolbelt.utils import dump
   
   response = requests.get("https://v2.jokeapi.dev/joke/Any?safe-mode")
   print(dump.dump_all(response).decode("utf-8"))
   

Acclaim

Thanks so much for highlighting that previous answer. I'd skimmed past it because it looked like more than what I wanted! – Horsey

Answer 7

If you're using Python 2.x, try installing a urllib2 opener. That should print out your headers, although you may have to combine that with other openers you're using to hit the HTTPS.

import urllib2
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1)))
urllib2.urlopen(url)

Kinard

Answer 8

The verbose configuration option might allow you to see what you want. There is an example in the documentation.

NOTE: Read the comments below: The verbose config options doesn't seem to be available anymore.

Zingg

There is? Can't really find it. – Plaza

@badcat. There was a "Verbose Logging" section at the time. It seems it was removed in December. – Zingg

Ah, that would explain that. :) Still, now this question is kind of valid again, because I couldn't find a way to print the whole traffic between server and client for debugging. – Plaza

Is there a recommended "new way" to achieve the same effect as verbose logging? – Cerated

My answer demonstrates the correct method for Requests 1.x and higher. – Marzipan

Answer 9

As others have noted there is a nice requests-toolbelt module with convenient functions to dump request and response content using requests hooks. Unfortunately (as of now) there is only a hook to be invoked on a successful completion of a request. It's not always a case. I.e. request could end up with ConnectionError or Timeout exceptions.

The requests-toolbelt module on it's own also provides public functions to dump completed requests only. However using a bit of non-public APIs and Session sub-classing it's possible to implement logging of requests before send and logging of responses after receive.

NOTE: code relies on implementation details/non-public APIs of requests-toolbelt module and thus make unexpectedly break in the future:

import requests

from requests_toolbelt.utils import dump

class MySession(requests.Session):
    def send(self, req, *args, **kwargs):
        prefixes = dump.PrefixSettings(b'< ', b'> ')

        data = bytearray()

        try:
            dump._dump_request_data(req, prefixes, data)
            resp = super().send(req, *args, **kwargs)
            dump._dump_response_data(resp, prefixes, data)
        finally:
            print(data.decode('utf-8'))

        return resp

and here is an example of usage:

>>> MySession().get('https://httpbin.org/headers')
< GET /headers HTTP/1.1
< Host: httpbin.org
< User-Agent: python-requests/2.25.1
< Accept-Encoding: gzip, deflate
< Accept: */*
< Connection: keep-alive
< 

> HTTP/1.1 200 OK
> Date: Fri, 19 Aug 2022 10:43:51 GMT
> Content-Type: application/json
> Content-Length: 225
> Connection: keep-alive
> Server: gunicorn/19.9.0
> Access-Control-Allow-Origin: *
> Access-Control-Allow-Credentials: true
> 
{
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.25.1"
  }
}

>>> MySession().get('https://non.existent')
< GET / HTTP/1.1
< Host: non.existent
< User-Agent: python-requests/2.25.1
< Accept-Encoding: gzip, deflate
< Accept: */*
< Connection: keep-alive
< 


Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
...

Usurer