Connection Error while using requests to get response from google distance matrix api

Question

I am trying to develop something which involves finding distance between two geo locations.

There are many such combinations of source and destination, which I am passing into google distance API in a loop.

Part of my code:

key_list = [****,****,....]  #google key list
base_url = 'https://maps.googleapis.com/maps/api/distancematrix/json?'

for i in geocodes:
    for j in key_list:

        origin_part = 'origins=' + i[0] + "," + i[1]
        destination_part = '&destinations=' + i[2] + "," + i[3]
        api_url = base_url + origin_part + destination_part + key
        json_response = requests.get(api_url,timeout=10).json()
        if json_response["status"] == "OVER_QUERY_LIMIT":
            j += 1
            i -= 1

This is only a part of my code.

geocodes is a nested list containing sublists

[[source_latitude,source_longitude,destination_latitude,destination_longitude],....]

The code is running fine for sometime, but after some time, it is giving me an error saying that:

HTTPSConnectionPool(host='maps.googleapis.com', port=443): Max retries exceeded with url: /maps/api/distancematrix/json?origins=xx,xx&destinations=xx,xx&key=xxxxx 
(Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f53350c50f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))

I tried using sessions too instead of requests, but the same error is coming.

I want to know what is causing this issue?

And how can I rectify it, better without using try-catch.

Nubile

How many destinations/sources? Please provide a minimal reproducible example that demonstrates the issue – Stirps

@Stirps around 1 lakh distances are to be computed, but the error comes randomly after 1000-2000 rows, it's not specific after how many distance computation the error may come. – Nubile

do you have full access to google maps api? or are you just using the free version? – Somatist

Check if you are in OVER_QUERY_LIMIT in the google's console. – Starrstarred

Answer 1

I think the major problem stems from the fact that the code is formatted incorrectly. The way it is setup now, I don't believe you will be able to prevent that error.

1. Nested for loops

   for i in geocodes:
       for j in key_list:
   

   geocodes is your list of lists containing origin and destination coordinates and key_lists contains all your api keys. The problem here is that geocodes is the first for loop. In essence, every key in your key_list is being used to find the distance matrix for every coordinate pair. Which is wasting api resources.

2. Usage of different api keys

   api_url = base_url + origin_part + destination_part + key
   

   This line of code falls within your nested for loops, and generates your api url. Problem here is it never changes. The last element is key. Since you said this was only a snippet of the code, we do not know what key means however, I assume that it is a static api key that was set earlier in the code. When it should actually be j instead, which would switch the keys.

3. Key switching on error

   if json_response["status"] == "OVER_QUERY_LIMIT":
       j+=1
       i-=1
   

   This line of code isn't exactly working how you think it would. It won't shift to the next j in the for loop, and go back an i. This would just add or subtract 1 from their values, and since there not ints or floats that will throw an error eventually.

Instead I suggest you change you code to this.

base_url = 'https://maps.googleapis.com/maps/api/distancematrix/json?'
key_list = [****,****,....]  #google key list

index,atEnd = 0,False
for coords in geocodes:
    origin_part = 'origins=' + coords[0] + "," + coords[1]
    destination_part = '&destinations=' + coords[2] + "," + coords[3]
    api_url = base_url + origin_part + destination_part + key_list[index]
    json_response = requests.get(api_url,timeout=10).json()

    #Check if your over your daily limit, if so try the next key
    while json_response['status'] == 'OVER_QUERY_LIMIT':
        index += 1
         #if all keys used to max quotas, exit
        if index == len(key_list):
            atEnd = True
            break
        api_url = base_url + origin_part + destination_part + key_list[index]
        json_response = requests.get(api_url, timeout=10).json()
    print json_response

    #Exit main loop if all quotas reached
    if atEnd == True:
        print'All of your keys quotas have been used, try again tomorrow'
        break

Just a note, I'm not well versed in the legal uses of the maps API. However, I am pretty sure that what you are doing violates the terms and conditions of Google, by using more than one account to exceed your daily limit usage.

I would suggest doing the free trail on the google developers console. They supply you with 300 in credit to use their services to go beyond the quotes.

Somatist

Answer 2

As already noted in other responses, your excerpt is not valid code and the use of different keys is probably abusing Google's T's & C's. However, that is not the issue.

What's the problem?

Your problem is quite clearly stated in the requests error. As you can see, the error that you are hitting in urllib3 is here. You are not even getting to the point where you have sent any data to Google.

Looking at the specific message ([Errno -3] Temporary failure in name resolution), the system is telling you that DNS cannot resolve your hostname ("maps.googleapis.com") to an IP address.

It's a well known feature that requests (through urllib3) always calls getaddrinfo to resolve hostnames, so either your OS is not caching results or you have a flaky DNS server. See this SO thread for more details.

What to do about it?

First option is to try a more reliable DNS server - many people use Google's public DNS when they start having such problems.

If that doesn't work, you could try doing the DNS lookup yourself (i.e. call getaddrinfo) and caching the IP address in your application (and then use that instead of the hostname in the URL), or you could install a DNS cache on your machine. These won't prevent the DNS failure, but they should reduce the impact.

Kassel