Getting a raw, unparsed HTTP response

Question

Are there any straightforward ways to make a HTTP request and get at the raw, unparsed response (specifically the headers)?

Helsinki

What exactly are you hoping to do with it that you can't do with the more user-friendly ways of receiving the data? – Botti

I want to see if the server is using \n instead of \r\n in its responses. – Helsinki

Answer 1

Using the socket module directly:

import socket

CRLF = "\r\n"

request = [
    "GET / HTTP/1.1",
    "Host: www.example.com",
    "Connection: Close",
    "",
    "",
]

# Connect to the server
s = socket.socket()
s.connect(('www.example.com', 80))

# Send an HTTP request
s.send(CRLF.join(request))

# Get the response (in several parts, if necessary)
response = ''
buffer = s.recv(4096)
while buffer:
    response += buffer
    buffer = s.recv(4096)

# HTTP headers will be separated from the body by an empty line
header_data, _, body = response.partition(CRLF + CRLF)

print header_data

HTTP/1.0 302 Found
Location: http://www.iana.org/domains/example/
Server: BigIP
Connection: Keep-Alive
Content-Length: 0

Hybris

Wow, thanks for the fully working solution! Does this get hugely more complicated if you need to use HTTPS? – Helsinki

Thanks, Jeremy -- I just logged in again to add the response splitting, and you beat me to it :) – Hybris

@Acorn: If you want to do SSL this way, you will need to impor the ssl module, and use an SSLSocket rather than a regular socket. I haven't used it myself, so there may be other differences. Sounds like a good topic for another SO question, though :) – Hybris