What’s the best way to handle exceptions from Net::HTTP?

Question

What’s the best way to rescue exceptions from Net::HTTP?

Exceptions thrown are described in Ruby’s socket.c, like Errno::ETIMEDOUT, Errno::ECONNRESET, and Errno::ECONNREFUSED. The base class to all of these is SystemCallError, but it feels weird to write code like the following because SystemCallError seems so far removed from making an HTTP call:

begin
  response = Net::HTTP.get_response(uri)
  response.code == "200"
rescue SystemCallError
  false
end

Is it just me? Is there a better way to handle this beyond fixing Net::HTTP to handle the Errno exceptions that would likely pop up and encapsulate them in a parent HttpRequestException?

Disturbing

Thanks everyone! I ended up packaging the answers in the form of a Ruby Gem I can use in the future to deal with this situation: net_http_exception_fix [github.com/edward/net_http_exception_fix] – Disturbing

Correct link: github.com/edward/net_http_exception_fix (Edward’s comment slurps an extra "]" into the URL). – Rotary

I've written another library that solves the same problem in a different way: github.com/barsoom/net_http_timeout_errors – Faceless

Cool! Nice work, Henrik. – Disturbing

"net_http_exception_fix" doesn't catch Errno::EHOSTUNREACH. but it is still worth to use. – Cabdriver

Answer 1

I agree it is an absolute pain to handle all the potential exceptions. Look at this to see an example:

Working with Net::HTTP can be a pain. It's got about 40 different ways to do any one task, and about 50 exceptions it can throw.

Just for the love of google, here's what I've got for the "right way" of catching any exception that Net::HTTP can throw at you:

begin
  response = Net::HTTP.post_form(...) # or any Net::HTTP call
rescue Timeout::Error, Errno::EINVAL, Errno::ECONNRESET, EOFError,
       Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError => e
  ...
end

Why not just rescue Exception => e? That's a bad habit to get into, as it hides any problems in your actual code (like SyntaxErrors, whiny nils, etc). Of course, this would all be much easier if the possible errors had a common ancestor.

The issues I've been seeing in dealing with Net::HTTP have made me wonder if it wouldn't be worth it to write a new HTTP client library. One that was easier to mock out in tests, and didn't have all these ugly little facets.

What I've done, and seen most people do, is move away from Net::HTTP and move to 3rd party HTTP libraries such as:

httparty and faraday

Estival

Or Ruby's own Open-URI or HTTPClient or Typhoeus. Open-URI is a much simpler interface for quick tasks. HTTPClient and Typhoeus are industrial strength, handling threading and all sorts of other heavy lifting. – Yeld

HTTPClient is hot! I think I’ll end up using that in the future. Thanks for the reference. – Disturbing

Your Google-fu is superior and I’m glad to have landed on Tammer’s work. That’s just what I was looking for. – Disturbing

Errno::ECONNREFUSED worth including. See here list of errors for Unix bases systems – Elderly

Errno::ENETUNREACH - just got another one after my target VM died... – Roving

Answer 2

I experienced the same problem, and after a lot of research, I realized the best way to to handle all exceptions Net::HTTP methods would throw is to rescue from StandardError.

As pointed by Mike Lewis's answer, Tammer Saleh blog post proposes rescuing from a lot exceptions, but it is still flaw. There are some exceptions he does not rescue from, like Errno::EHOSTUNREACH, Errno::ECONNREFUSED, and possible some socket exceptions.

So, as I found out in tenderlove's translation of an old ruby-dev thread, the best solution is rescuing from StandardError, unfortunately:

begin
  response = Net::HTTP.get_response(uri)
rescue StandardError
  false
end

It is awful, but if you want your system does not break because of these other exceptions, use this approach.

Ayrshire

Beware, this will silence errors like VCR::Errors::UnhandledHTTPRequestError as well – Watersoak

Answer 3

Another approach is to aggregate all these exceptions in a constant, and then re-use this constant, e.g.:

ALL_NET_HTTP_ERRORS = [
  Timeout::Error, Errno::EINVAL, Errno::ECONNRESET, EOFError,
  Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError
]

begin
  your_http_logic()
rescue *ALL_NET_HTTP_ERRORS
  …
end

It is far more maintainable and cleaner.

However, word of warning. I've copied the possible exceptions list from the aforementioned Tammer Saleh's blog post, and I know that his list is incomplete. For instance, Net::HTTP.get(URI("wow")) raises Errno::ECONNREFUSED which is not listed. Also, I wouldn't be surprised if the list should be modified for different Ruby versions.

For this reason, I recommend sticking to rescue StandardError in most cases. In order to avoid catching too much, move as much as possible outside the begin-rescue-end block, preferably leave only a call to one of the Net::HTTP methods.

Wye

Answer 4

Your intuition on this is right, for the most robust solution, I'd probably rescue each one individually (or in small groups) and take the appropriate action, like trying the connection again, or abandoning the request all together. I like to avoid using a very high-level/generic rescue because it might catch exceptions that I'm not prepared for or didn't expect.

Synchrotron

Ok, cool – glad to know I’m not the only one :) I ended up packaging your advice + @MikeLewis into a gem called net_http_exception_fix – Disturbing