How to read from a TCPServer socket in ruby using read, readpartial and read_nonblock

Question

I have a 2 part question on reading from sockets and how is it managed on Ruby servers like Unicorn or Mongrel

1. I've learnt that to read from a socket is different from reading a file and that there are no distinct EOF message sent and the data is an endless stream. So how do you know when to stop reading? My TCPServer for example in this case when I hit my server by accessing http://localhost:9799 from a browser, it hangs after there is no more data to read and it won't throw the EOFError either.

require 'socket'

READ_CHUNK = 1024
socket = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM)
addr = Socket.pack_sockaddr_in(9799, '127.0.0.1')
socket.bind(addr)
socket.listen(Socket::SOMAXCONN)
socket.setsockopt(:SOCKET, :REUSEADDR, true)

puts "Server is listening on port = 9799"
loop do 
    connection, addr_info = socket.accept
    data_buffer = ""

    loop do
        begin
            connection.read_nonblock(READ_CHUNK, data_buffer)
            puts "Buffer = #{data_buffer}"
        rescue Errno::EAGAIN => e
            IO.select([connection])         
            retry
        rescue EOFError
            break
        end
    end
    connection.write("HTTP/1.1 200 \r\n")
    connection.write("Content-Type: text/html\r\n")
    connection.write("Status 200 \r\n")
    connection.write("Connection: close \r\n")
    connection.write("Hello World \r\n")
    connection.close
end

I'd like to know whats the best practice/standard approach used by Ruby servers. I see the Unicorn uses read_nonblock from kgio library and mongrel uses readpartial (I'm not sure about these but going through the code this is what I feel is the approach adopted.) Even with checks for \r\n how does the server know the input is complete. Could explain how this should be done (and I think gets is not the approach - its with read, readpartial, read_nonblock).

2). I would really appreciate a few lines on how this is achieved in servers like unicorn or passenger

Thank you.

Pinite

Answer 1

It's done in unicorn here https://github.com/defunkt/unicorn/blob/master/lib/unicorn/http_request.rb#L69-L71

There is add_parse method(read the comments above methods) https://github.com/defunkt/unicorn/blob/master/ext/unicorn_http/unicorn_http.rl#L760-L778

Also take a look at some explanations here http://www.ruby-forum.com/topic/2267632#1014288

Here is your working code using http_parser.rb https://gist.github.com/4136962

gem install http_parser.rb

require 'socket'
require "http/parser"


READ_CHUNK = 1024 * 4
socket = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM)
addr = Socket.pack_sockaddr_in(9799, '127.0.0.1')
socket.bind(addr)
socket.listen(Socket::SOMAXCONN)
socket.setsockopt(:SOCKET, :REUSEADDR, true)

puts "Server is listening on port = 9799"
loop do
    connection, addr_info = socket.accept

    parser = Http::Parser.new
    begin
      data = connection.readpartial(READ_CHUNK)
      puts "Buffer = #{data}"
      parser << data
    end until parser.headers

    connection.write("HTTP/1.1 200 \r\n")
    connection.write("Content-Type: text/html\r\n")
    connection.write("Status 200 \r\n")
    connection.write("Connection: close \r\n")
    connection.write("\r\n\r\n")
    connection.write("Hello World \r\n")
    connection.close
end

Selfexplanatory

Thank you for response and introducing me to http_parser.rb. Know of any servers which don't use the parser library and opt for some other approach? – Pinite

I don't know servers that don't use parsers, what kind of another approach? – Selfexplanatory

Im sorry what I meant was servers that don't use the http_parser.rb library. – Pinite

Yes, for example unicorn uses own parser maybe based on mongrel, they are all use custom high-performance HTTP request parser implemented using Ragel. – Selfexplanatory

Why loop is waiting for headers? I want to obtain a body if it exists. – Mot