Scripting an HTTP header request with netcat

Question

I'm trying to play around with netcat to learn more about how HTTP works. I'd like to script some of it in bash or Perl, but I've hit upon a stumbling block early on in my testing.

If I run netcat straight from the prompt and type in a HEAD request, it works and I receive the headers for the web server I'm probing.

This works:

    [romandas@localhost ~]$ nc 10.1.1.2 80
    HEAD / HTTP/1.0

    HTTP/1.1 200 OK
    MIME-Version: 1.0
    Server: Edited out
    Content-length: 0
    Cache-Control: public
    Expires: Sat, 01 Jan 2050 18:00:00 GMT

    [romandas@localhost ~]$

But when I put the same information into a text file and feed it to netcat through a pipe or via redirection, in preparation for scripting, it doesn't return the headers.
The text file consists of the HEAD request and two newlines:

HEAD / HTTP/1.0

Sending the same information via echo or printf doesn't work either.

$ printf "HEAD / HTTP/1.0\r\n"; |nc -n 10.1.1.2 80
$ /bin/echo -ne 'HEAD / HTTP/1.0\n\n' |nc 10.1.1.2 80

Any ideas what I'm doing wrong? Not sure if it's a bash problem, an echo problem, or a netcat problem.

I checked the traffic via Wireshark, and the successful request (manually typed) sends the trailing newline in a second packet, whereas the echo, printf, and text file methods keep the newline in the same packet, but I'm not sure what causes this behavior.

Arianna

FYI, the \r\n in the printf is purposeful; I switched between \r and \n in various combinations to see if it changed anything -- to no effect. – Arianna

Answer 1

You need two pairs of "\r\n", and also to tell netcat to wait for a response.

The following or something similar should work:

printf "HEAD / HTTP/1.0\r\n\r\n" | nc -n -i 1 10.1.1.2 80

Broider

This doesn't work for me. I copied your line directly, but the connection still terminates before transmitting the header. – Arianna

Then the problem is at your server end. printf "HEAD / HTTP/1.0\r\n\r\n" | nc www.toothycat.net 80 works here. – Broider

The server isn't terminating the request; it's initiated by the client. Wireshark shows my client sending a FIN packet right after the HEAD request packet, then the server ACKs and terminates gracefully. Not sure what causes the difference. What version of netcat are you using? – Arianna

1.10-33 Oh, try adding a -q 10 to make netcat wait for a response after it gets the EOF on STDIN. – Broider

Ah, that did it. I'm using GNU netcat (v1.84), so the option is -i 1 (to wait between successive lines of input). That caused the second newline to be sent later, and produced the desired HEAD response. Edit that into your answer, and I'll go ahead and accept it. :) Thanks! – Arianna

FYI, just the two newlines matter \n\n works (for me) just as well as \r\n\r\n. – Arianna

The RFC requires CRLF pairs; best to be safe if you can. – Broider

Answer 2

Another way is to use what is called the 'heredoc' convention.

$ nc -n -i 1 10.1.1.2 80 <<EOF
> HEAD / HTTP/1.0
>
> EOF

Wallis

Answer 3

Another way to get nc to wait for the response is to add a sleep to the input. e.g.

(printf 'GET / HTTP/1.0\r\n\r\n'; sleep 1) | nc HOST 80

Cilla

I am a fan of this approach because you have to specify an integer number of seconds with nc's -i flag; however, with sleep you can specify decimals, and a much smaller interval seems to work for me (using a local server), like 0.01 seconds. – Houseraising

Answer 4

You can use below netcat command to make your instance webserver:

MYIP=$(ifconfig eth0|grep 'inet addr'|awk -F: '{print $2}'| awk '{print $1}')
while true; do echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to $MYIP" | sudo nc -l -p 80 ; done&

Unearned

IPv4 IP which is used for default gateway (no matter what interface; Linux): ip route show | grep -o 'src [\.0-9]\+' | uniq | cut -d ' ' -f2 – Whiffler

Answer 5

This line will also work as equivalent:

echo -e "HEAD / HTTP/1.1\nHost: 10.1.1.2\nConnection: close\n\n\n\n" | netcat 10.1.1.2 80

Santos