What's an effective method to make 50 API requests at a time when there is a limit of 10 calls per second (Rotten Tomatoes API)?

Question

I have an application which has to make 50 to 100 API calls in a loop. Rotten Tomatoes have a limit of 10 calls per second. As a result of this my requests fail in between and i get different results everytime. What is an effective way to make these 50 requests without exceeding the 10 reqs/per second limit ? Here's my code:

$.each(elem, function (index, item) {

    var $placeholder = $('<div>').appendTo("div.content");

    $.ajax({
        type: 'post' ,
        url: moviesSearchUrl + '&q=' + encodeURI(item) + '&page_limit=1',
        dataType: "jsonp",
        async: false, 
        success: searchCallback
    });

    function searchCallback(data) {
        var movies = data.movies;

        var markup = index + ': '+   movies[0].title + '<img class=" bord" src="' + movies[0].posters.thumbnail + '" /><br/>';

        $placeholder.replaceWith(markup);
    }
});

Anthropophagite

What are you wanting to "do" for your users? By that I'm just asking what functionality you want to provide, to see if we can avoid hammering the web service. – Dennadennard

I want to display the Name of the movie along with a poster thumbnail and the cast information. Basically i have a text file containing the names of 100 movies. I would read the file and make a request for that movie name and get back the poster etc.. The functionality is that each user can select the movies they have already seen from the list which gets displayed. I will store it into a database and fetch it when the users return. – Anthropophagite

Usually you save reference calls for dynamic content, if you're just storing a big list then why not store the poster links there too? Make calls for things that change when a user wants to look at the "details" of a particular movie. The query call you're using is really meant to return the results of a user generated search (eg: they use you to search for "air bud" and get to see all 400 air bud movies on your site with one ajax call) – Dennadennard

The text file that contains the movies will keep changing. Its not a constant set of movies. How can i handle that ? – Anthropophagite

Then, again, you'll need to step back further. Why is the text file changing? What are you really trying to do. The Rotten Tomatoes API looks like it is built to do 3 things and 3 things only A) return info on ONE movie B) return info on the movies that match a search C) return info on the movies that match one of their pre-made lists (eg: in theaters, upcoming, etc...). Outside of those pre-made lists if you want lots of information you will either have cache it locally or throttle your requests to the server (beware that will scale very very poorly to multiple concurrent users). – Dennadennard

I am trying to implement the imdb top 250 movies list into my application. Theres no way of getting the list of 250 movies from the IMDB site using an API. So i am storing it in a text file. For now i just update the text file manually. I plan to do it with a script in the future. Is it possible to keep a button after 10 requests and make the next 10 requests when the button is clicked ? – Anthropophagite

Answer 1

It depends what you're trying to do. If the result is going straight (in line) into a user web page you're rendering (and there are no bulk calls which you can do instead of the individual ones) then there's little you can do (you'll take minimum 5 seconds to render that page).

If you're reusing the same content often, then if the service provider's terms allow it may be worth caching the results of the calls for a short period to avoid having to hit all the calls again and again.

Agree with the point above - if you're rendering directly to a web page with multiple users you'll suffer badly - best think of a short caching strategy.

Dituri

Okay. I am new to jQuery and working with APIs. Can you elaborate on how to cache the data i get from the site ? It would be of great help to me. – Anthropophagite

To cache you'll need to actually call the API from your server side application and hold it in something like memcached or your data base. Then you can pull the items from your local storage into your server side rendered page. This is alternative to using the javascript to call the API you're accessing directly (without going through your server) - hope that helps. – Dituri