HTTPException: Deadline exceeded while waiting for HTTP response from URL: #deadline

Question

We are using the developers python guide with Python data 2.15 library and as per example stated for app engine. createSite("test site one", description="test site one", source_site =("https://sites.google.com/feeds/site/googleappsforus.com/google-site-template" ))

We are getting an un-predictable response every time we use.

Exception: HTTPException: Deadline exceeded while waiting for HTTP response from URL: https://sites.google.com/feeds/site/googleappsforyou.com

Did someone experience the same issue? Is it AppEngine or Sites API related?

Regards,

Gothicism

FYI: You should tag this issue correctly under the google-appengine category and possibly also the Google API client (if that exists). – Quincyquindecagon

Answer 1

Deadline exceeded while waiting for HTTP response from URL is actually a DeadlineExceededError. When you are making a HTTP request, App Engine maps this request to URLFetch. URLFetch has its own deadline that is configurable. See the URLFetch documentation.

It seems that your client library catches DeadlineExceededError and throws HTTPException. Your client library either passes a deadline to URLFetch (which you would need to change) or the default deadline is insufficient for your request.

Try setting the default URLFetch deadline like this:

from google.appengine.api import urlfetch
urlfetch.set_default_fetch_deadline(45)

Also make sure to check out Dealing with DeadlineExceededErrors in the official Docs.

Keep in mind that any end-user initiated request must complete in 60 seconds or will encounter a DeadlineExceededError. See App Engine Python Runtime Request Timer.

Quincyquindecagon

Hi Bernd, Thanks for your response. Even after we increased the Fetch time to 45 seconds, we are still experiencing deadline issues - we created about 50 sites in a day, out of which 25% of them failed due to deadline exceptions. How do we deal with deadline exceptions? What are some of the work arounds? We create every day about ~30-40 sites and would like to address this issue if we can.Thanks for your help in advance. – Gothicism

The import is actually: from google.appengine.api import urlfetch – Drawplate

If the content you are fetching cannot be fetched within 60 seconds minus the overhead for your request handler, you will need to fetch (call) the URLs from a request initiated by a TaskQueue or a request running on Backend. When using either of these approaches you can increase the deadline to as much as 10 minutes. – Quincyquindecagon

@Bernd Verst Is there any way where we can handle this exception ? I am getting Process terminated because the request deadline was exceeded. (Error code 123) error. – Athematic

@BerndVerst i am getting the same error. In my app it endpoint make db connection ,get data and closes connection and return data. But it is throwing error code 123. As i checked there was 900 database connection and 150+ instances were up. I restarted my db and problem is solved? what is the reason how can i solve that ? – Sacrilegious

Answer 2

The accepted solution did not work for me when working with the very recent versions of httplib2 and googleapiclient. The problem appears to be that httplib2.Http passes it's timeout argument all the way through to urlfetch. Since it has a default value of None, urlfetch sets the limit for that request to 5s irrespective of whatever default you set with urlfetch.set_default_fetch_deadline. It appears that you have a few options.

First option -- You can explicitly pass an instance of httplib2.Http around:

http = httplib2.Http(timeout=30)
...
client.method().execute(http=http)

Second option, you can set the default value using sockets¹:

import socket
socket.setdefaulttimeout(30)

Third option, you can tell appengine to use sockets for requests². To do that, you modify app.yaml:

env_variables:
   GAE_USE_SOCKETS_HTTPLIB : 'anyvalue'

^{1This might only work for paid apps since the socket api might not be present for unpaid apps...
2I'm almost certain this will only work for paid apps since the socket api won't be functional for unpaid apps...}

Prepossessing