TooManyRequests Overpass Error

Question

I'm using overpy to query the Overpass API, and the nature of the data is such that I have a lot of queries to execute. I've run into the 429 OverpassTooManyRequests exception and I'm trying to play by the rules. I've tried introducing time.sleep methods to space out the requests, but I have no basis for how long the program should wait before continuing.

I found this link which mentions a "Retry-after" header:
How to avoid HTTP error 429 (Too Many Requests) python

Is there a way to access that header in an overpy response? I've been through the docs and the source code, but nothing stood out that would allow me to access that header so I can pause querying until it's acceptable to do so again.

I'm using Python 3.6 and overpy 0.4.

Bestraddle

Instead of spacing out the requests, why not just catch the OverpassTooManyRequests exception, wait 10 seconds, then try again? If it's too early, then it'll just wait another 10 seconds. – Girand

You need to evaluate the results of the /api/status call. BTW: how much is "a lot of queries"? Are you aware of the usage limits? That's 10'000 queries for all of your users per day. Forget about the "Retry-after" header at this time, it's not yet implemented, see github.com/drolbr/Overpass-API/issues/351 – Aegaeon

@Aegaeon about 450 queries, however, this isn't a script that will be making all these queries repeatedly. The queries are made once, the results are added to an HTML map file and the map file is what gets shared. Ideally, you'd make the queries one time for a given data set and never have to again. So the only user making the queries is me. And I am aware of the 10,000/day limit. – Bestraddle

450 queries doesn't sound too much, unless they're very expensive. Other libraries already implement a logic to evaluate /api/status, here's an example for R, if you don't mind: github.com/hrbrmstr/overpass/blob/master/R/overpass_query.r – Aegaeon

@Aegaeon I'm not familiar with R (I know what it is, but have never used it). Is there a way to evaluate /api/status in Python or OverPy? – Bestraddle

Why not raise an issue with OverPy then? – Aegaeon

You'll have to forgive me, I'm not used to suggesting...improvements? to a module. Do I just suggest that they add a way to evaluate /api/status on their Github repository? – Bestraddle

Sure, why not? :) – Aegaeon

Answer 1

Maybe this isn't quite the answer you're seeking, but I ran into the same issue and fixed it by simply hosting my own OSM database server using docker. Just clone the repo and follow instructions:

https://github.com/mediasuitenz/docker-overpass-api

Ethiop

Answer 2

from http://overpass-api.de/command_line.html do check that you do not have a single 'runaway' request that is taking up all the resources.

Biconvex

When I click the link to overpass-api.de/api/kill_my_queries, I get a blank page. Is that normal? – Bestraddle

Yes, it means that no queries were killed. – Aegaeon

Answer 3

After verifying that I don't have runaway queries, I have taken Peter's advice and added a catch for the TooManyRequests exception that waits 30s and tries again. This seems to be working as an immediate solution.

I will also raise an issue with the originators of OverPy to suggest an enhancement to allow evaluating the /api/status, as per mmd's advice.

Bestraddle

Once you raise an issue with OverPy, can you please also post a link to it here. Thanks. – Aegaeon