Curl: sleep/delay between requests

Question

I am trying to download flurry exception logs using the following command.

curl --cookie ./flurry.jar -k -L "https://dev.flurry.com/exceptionLogsCsv.do?projectID=49999&versionCut=versionsAll&intervalCut=allTime&direction=1&offset=[0-100:10]" --output "exception#1.csv"

It works fine and it downloads the csv files based on the offset(10,20,30 etc). I would like to insert a delay between each request. Is it possible to do that in CURL?

Pilsudski

Answer 1

Using bash shell (Linux) :

while :
do
    curl --cookie ./flurry.jar -k -L "https://dev.flurry.com/exceptionLogsCsv.do?projectID=49999&versionCut=versionsAll&intervalCut=allTime&direction=1&offset=[0-100:10]" --output "exception#1.csv"
    sleep 5m
done

It is an infinite loop, and the delay is given by the sleep command.

Edit. On Windows machine, you can do this trick instead :

for /L %i in (0,0,0) do (
    curl --cookie ./flurry.jar -k -L "https://dev.flurry.com/exceptionLogsCsv.do?projectID=49999&versionCut=versionsAll&intervalCut=allTime&direction=1&offset=[0-100:10]" --output "exception#1.csv"
    ping -n XX 127.0.0.1>NUL
)

The sleep command is not available on Windows. But you can use ping to "emulate" it. Just replace the XX above with the number of seconds you want to delay.

Systematology

I am using windows how can i do it in windows? . also looking at the above code, it seems it will run the same command over and over which is not what i want. The command itself will iterate over as I have offset=[0-100:10] . I would like to mention a delay or sleep i the command,. is it possible? – Pilsudski

To iterate over the offset (10, 20, 30, ..., 100), replace (0,0,0) above with (0,100,10). It means start from 0 to 100, increment by 10. But it is not an infinite loop anymore. – Systematology

And use the variable %i to the web address. So it will be ...&offset=%i – Systematology

Put the code in a .bat file, and just run the file directly (double-clicking it). – Systematology

I don't think the asker wants to re-run curl forever; he wants to put delays between successive requests. – Cabbagehead

Warning: Don't include that -k there! That tells cURL NOT to check the certificate! Please see curl.haxx.se/docs/sslcerts.html – Redfin

Answer 2

wget has delay options

wget --wait=seconds

and also random delay option

wget --random-wait

Lubra

Answer 3

in bash, this will pause a random number of seconds in the range 0-60:

for d in {0..100..10}
do
    i=`printf "%03d" $d`
    curl --cookie ./flurry.jar -k -L 'https://dev.flurry.com/exceptionLogsCsv.do?projectID=49999&versionCut=versionsAll&intervalCut=allTime&direction=1&offset='$d --output 'exception'$i'.csv'
    sleep $(($RANDOM*60/32767))
done

Cabbagehead

Answer 4

With curl 7.84.0 and later, you can use request rate limiting with the --rate option:

The request rate is provided as N/U where N is an integer number and U is a time unit. Supported units are s (second), m (minute), h (hour) and d (day, as in a 24 hour unit). The default time unit, if no /U is provided, is number of transfers per hour.

So to wait 10 seconds between each request, use curl --rate 6/m.

Fanchie