Getting Headers from a Ruby Net::HTTP Request before making the request

Question

In Ruby, how can I get hold of the HTTP Request Headers that will be sent by a net/http(s) or open-uri request BEFORE it actually makes the request.

In some cases, headers are used when creating a signed string in a URI. Surely there is some way to acquire the request headers that will be sent. These should include the "Host:" header for example.

Bock

Answer 1

see http://ruby-doc.org/stdlib-2.0/libdoc/net/http/rdoc/Net/HTTP.html#label-Setting+Headers

Works well in ruby 2.0.0 - but you are correct, different behavior in 1.9.3

Ruby 2.0.0

require 'net/http'
uri = URI('http://github.com/ruby')

http_request = Net::HTTP::Get.new(uri)
http_request.each_header { |header| puts header }

# => accept-encoding
# => accept
# => user-agent
# => host    

http_response = Net::HTTP.start(uri.hostname, uri.port) do |http|
  http.request(http_request)
end

Ruby 1.9.3

require 'uri'
require 'net/http'
uri = URI.parse('http://github.com/ruby')

http_request = Net::HTTP::Get.new(uri.path)
http_request.each_header { |header| puts header }

# => accept
# => user-agent

http_response = Net::HTTP.start(uri.hostname, uri.port) do |http|
  http.request(http_request)
end

http_request.each_header { |header| puts header }

# => accept
# => user-agent
# => host

Precipitation

interesting works great in ruby 2.0.0 - updated answer but probably doesn't help – Precipitation

Thanks :) but unfortunately, as you presumed, I need that host header pre-request hahaha :( – Bock

Answer 2

The Net::HTTP classes all seem to use Net::HTTPHeader as a mixin. You should be able to use to_hash() on the request object to get all headers at once, or each_header() / each() to iterate one header at a time.

Tripping

While the above should work to get a list of explicitly set headers, I'm not sure if it will include in advance any additional headers Net::HTTP might add at the moment it goes to send the request ('Host' being the perfect example... it's entirely possible this header is only auto-generated at the last second). – Tripping

Hmm.. Yeah, that's what concerned me. I'd found each_header already. I'm happy to assume the "Host:" header from the URI string, but I'm worried there might be other headers created automatically that I'll miss. – Bock

Answer 3

Here's some code I wrote to help.

def get_request_headers(request)
  http_method = request.method
  path = request.path

  request.to_hash.merge("method" => [http_method]).merge("path" => [path])
end

So now, you can run something like this.

url = URI("http://www.google.com")

request, response = Net::HTTP.start(uri(ico).host) do |http|
  request = Net::HTTP::Get.new(uri(ico))
  response = http.request request

  [request, response]
end

get_request_headers(request)
=> {"accept-encoding"=>["gzip;q=1.0,deflate;q=0.6,identity;q=0.3"], "accept"=>["*/*"], "user-agent"=>["Ruby"], "host"=>["www.google.com"], "method"=>["GET"], "path"=>["/"]}

request.to_hash give us a few headers for free, but there's more information stored in the instance variables for the request and response classes.

With the following code, you can check if there's anything else you'd like to merge into the basic request hash.

request.instance_variables.each do |variable|
  puts "#{variable}: #{request.instance_variable_get(variable)}"
end

=> @method: GET
=> @request_has_body: false
=> @response_has_body: true
=> @uri: http://www.google.com
=> @path: /
=> @decode_content: true
=> @header: {"accept-encoding"=>["gzip;q=1.0,deflate;q=0.6,identity;q=0.3"], "accept"=>["*/*"], "user-agent"=>["Ruby"], "host"=>["www.google.com"]}
=> @body: 
=> @body_stream: 
=> @body_data: 
=> [:@method, :@request_has_body, :@response_has_body, :@uri, :@path, :@decode_content, :@header, :@body, :@body_stream, :@body_data]

Note that I've pulled out the method and path for the get_request_headers method.

Finally, you can do the same for the response.

def get_response_headers(response)
  code = response.code
  http_version = response.http_version
  message = response.message

  response.to_hash.merge("code" => [code]).merge("http_version" => [http_version]).merge("message" => [message])
end

get_response_headers(response)
=> {"date"=>["Thu, 06 May 2021 14:34:27 GMT"], "expires"=>["-1"], "cache-control"=>["private, max-age=0"], "content-type"=>["text/html; charset=ISO-8859-1"], "p3p"=>["CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\""], "server"=>["gws"], "content-length"=>["6067"], "x-xss-protection"=>["0"], "x-frame-options"=>["SAMEORIGIN"], "set-cookie"=>["NID=215=dYowhmNSD9_CnKYLtsFI3uWVGy8ca8PKJTE8VY6_92q7tU5Y_AOWLsaabXxlSPBjc2QjOr4xXVX5SGMCrccTCnBR9pbdsKkrpVTV5TMqrV6H09ChxGjBr5mHVdZkgjOxswiXu72TF3eAX0uhXqloDb-5gmZ6NJ4w1YDKQKNoDp4; expires=Fri, 05-Nov-2021 14:34:27 GMT; path=/; domain=.google.com; HttpOnly"], "code"=>["200"], "http_version"=>["1.1"], "message"=>["OK"]}

Lina