How to call cURL without using server-side cache?

Question

Is there a way to tell cURL command not to use server's side cache? e.g; I have this curl command:

curl -v www.example.com

How can I ask curl to send a fresh request to not use the cache?

Note: I am looking for an executable command in the terminal.

Stripy

What makes you think curl is caching the request? – Decennial

I am talking about server side caching; suppose the site is using memcache – Stripy

If you have access to the server you could implement a custom parameter that bypassed the cache, otherwise, no. – Johnsiejohnson

Code your application on the server-side to accept an argument to bypass the cache. – Wohlert

@AlexK. can I pass the parameter using -d option? can you provide an example please? – Stripy

I think what Alex is talking about is sending a parameter in the URL, say http://www.example.com/?caching=off You could also send a custom header: #357205 But of course your server side application has to understand those commands and act on them by turning the caching off. If anyone could disable a remote site's caching mechanisms, it would be a gaping security hole – Decennial

Answer 1

I know this is an older question, but I wanted to post an answer for users with the same question:

curl -H 'Cache-Control: no-cache' http://www.example.com

This curl command servers in its header request to return non-cached data from the web server.

Blink

Why was this downvoted? This worked great for me and it answers the question. – Dipole

I agree with Monkpit. This works great and should be accepted as the answer. – Romance

This helps sometimes, but is not guaranteed to work. The server or proxies in the middle can ignore the no-cache request. – Settler

@Settler Then we put the blame on whoever manage that server. – Vanadium

Answer 2

The -H 'Cache-Control: no-cache' argument is not guaranteed to work because the remote server or any proxy layers in between can ignore it. If it doesn't work, you can do it the old-fashioned way, by adding a unique querystring parameter. Usually, the servers/proxies will think it's a unique URL and not use the cache.

curl "http://www.example.com?foo123"

You have to use a different querystring value every time, though. Otherwise, the server/proxies will match the cache again. To automatically generate a different querystring parameter every time, you can use date +%s, which will return the seconds since epoch.

curl "http://www.example.com?$(date +%s)"

Settler

Trying to use this with GitHub's raw files but no luck :( – Hugely

Yeah it does not work with GitHub's raw files. See https://mcmap.net/q/182316/-github-not-update-raw-after-commit – Stanfield

Answer 3

Neither -H 'Pragma: no-cache' nor -H 'Cache-Control: no-cache' helped me. In browser with "cmd+shift+r" (full reload) I was seeing a new version than the output of curl in terminal.

How to debug for yourself

To get the exact same result, I went to browser > F12 (Dev Tools) > Network/Requests > Right-click on the request > "Copy as cURL" and got the equivalent cURL command to the browser's call.

Then, I pasted that in the terminal and started removing the params one by one, until I found that surprisingly --compressed was making a difference in my case. (Calling CloudFront AWS)

Padding

This is most likely because of the vary header. Maybe there was no cached version for the compressed request, else it would have behaved the same. – Baal

Answer 4

You could try following ways to force not to keep Cache when curl.

Note: The server may or may not be configured to respect the Cache-Control header. Therefore, whether this method will work is dependent on the server or website we’re sending the HTTP request to.

curl command with the Cache-Control header

$ curl -H 'Cache-Control: no-cache, no-store' http://www.example.com

Adding the Pragma HTTP Header

$ curl -H 'Pragma: no-cache' http://www.example.com

Finally, the most common way: bypass the cache by changing the URL

curl -H 'Cache-Control: no-cache, no-store' http://www.example.com?$(date +%s)

Betthel

Answer 5

My problem is I should use single quotes in a query string.

My code

if (event.queryStringParameters && event.queryStringParameters['Name']) {
    responseMessage = 'Hello, ' + event.queryStringParameters['Name'] + '!';
  }

My requests

curl https://cssrq1srud.execute-api.us-east-1.amazonaws.com/serverless_lambda_stage/hello?Name=Terraform

returns zsh: no matches found

curl 'https://cssrq1srud.execute-api.us-east-1.amazonaws.com/serverless_lambda_stage/hello?Name=Terraform'   

returns {"message":"Hello, Terraform!"}

Sharleensharlene

Answer 6

This didn't work for me:

curl -H 'Cache-Control: no-cache' http://www.example.com

but this ended up doing the trick:

curl -H 'Cache-Control: no-cache' http://www.example.com&someFakeParam=$RANDOM

the &someFakeParam=$RANDOM makes the URL unique each time and bypasses caching.

Kadi