Python's urllib2.urlopen() hanging with local connection to a Java Restlet server

Question

I'm trying to connect to a local running Restlet server from python, but the connection hangs infinitely (or times out if I set a timeout).

import urllib2
handle = urllib2.urlopen("http://localhost:8182/contact/123") # hangs

If I use curl from a shell to open the above URL, the results return quickly. If I use urllib2 to open a different local service (e.g. a Django web server on port 8000), urllib2 works fine.

I've tried disabling firewall (I'm doing this on OS X). I've tried changing localhost to 127.0.0.1. The logs from Restlet for both the curl and urllib2 connection appear the same aside from the user-agent.

My workaround would be to just call curl via subprocess, but I'd rather understand why this is failing.

Here's how my Restlet Resource looks:

public class ContactResource extends ServerResource {

  @Get
  public String represent() throws Exception {
    return "<contact details>";
  }
  //....
}

Let me know if you want more info/code

Functional

What happens if you try different timeout values? urllib2.urlopen("...", timeout=1)? – Lamphere

With timeout=1 or 2, I get "urllib2.URLError: <urlopen error timed out>" and handle.read() returns empty string – Functional

Is your Restlet application designed to change behavior based on User-Agent or Accept headers? Also, your comment that you see urllib2 in the log implies the server does see it connect and responds to it - urllib2 just doesn't think the response ever comes through? – Twoseater

My Restlet server is very basic (i.e. followed some of the tutorials out there), so I didn't add any user-agent/header behavior changes. I should also mention that aside from my "/contact/123" url not working, the 404 catch-all page (i.e. "requested action not found") has the same issue with urllib (hangs indefinitely). – Functional

Try watching what goes across the network (using a wire sniffer, netcat) with something which does work and something which doesn't. Then repeat those by hand ("telnet localhost 8182" and type in the request) until you figure out what triggers the difference. – Yumuk

@AndrewDalke - thanks, I'll try this and follow-up soon – Functional

Is the dns resolver confused? Try using 127.0.0.1 instead of localhost – Hypoglossal

Answer 1

I encountered the similar issues and ended up using the Requests package.

Berrios

Answer 2

there is ProxyHandler ( http://docs.python.org/library/urllib2.html#urllib2.ProxyHandler ) in urllib2

try to pass empty dictionary to it before urlopen

urllib2.ProxyHandler([])
handle = urllib2.urlopen("http://localhost:8182/contact/123")

Tempe

The above code generates an error from the first line: "AssertionError: proxies must be a mapping" (python 2.7) – Functional

@dolan it is my mistake, sorry. The right code is urllib2.ProxyHandler({}) as said in the docs "To disable autodetected proxy pass an empty dictionary." – Tempe

That simply spit out "<urllib2.ProxyHandler instance at 0x1006370e0>", then next line still hung. Maybe I'm supposed to use the proxy instance in some manner? – Functional