How can I get control of Google App Engine caching behavior in WebKit (etags gone crazy)?

Question

Situation: running a Google App Engine site with my static content's default_expiration set to "14d"

Problem: in Chrome and Safari, visiting a URL (not reloading, just putting the cursor in the address bar and hitting Enter), causes a ton of requests to be fired with If-None-Match headers. The responses are always 304 Not Modified, as expected. I can watch these requests get fired in a debugging proxy like Charles or Fiddler.

Want: to avoid these requests and 304 responses entirely for static content -- simply trust the browser's cached content when it's available.

We use the standard "cache static content for a really long time, we'll take care of appending ?version={version} modifications to our query strings when we need to bust the cache" system, so we'd really like to avoid the 304's.

Belief: I think this is caused by the etag header that app engine sends down with every static content response. The app engine SDK does not send this header down, and I don't see this 304 behavior when messing around with the SDK.

Any advice? Can you turn off etags for app engine's static content?

Updated with an example piece of static content: http://www.khanacademy.org/stylesheets/shared-package/compressed.css

Ellett

Can you post a URL to said static content? Then someone can review what headers you're returning with it (expires etc). – Factotum

Nice. I'm no expert on the headers, but I think the answer is there. I see Expires Fri 24 Dec, that looks good. I also see an Etag header, a cache-control, and an age, those may affect whether or not the browser will cache the content. – Factotum

Does this not happen in Firefox, Opera or IE? – Natter

I can't test in IE until I get home to my other computer, but it doesn't happen in Firefox no my mac, no. – Ellett

@Alex Yes, I believe it's the etag header which tells the browser to ignore the Expires header and send an If-None-Match request...that's what I want to avoid. – Ellett

Answer 1

While I don't believe there is any way to control the etags header behavior for GAE, this is caused by a bug in WebKit that causes all static content to be re-downloaded when receiving a 302 redirect after a POST request.

Once WebKit fixes this bug, the issue should go away.

If you must, you can temporarily work around this specific redirect-after-POST bug by redirecting via a Refresh header instead of using a 302 redirect.

https://bugs.webkit.org/show_bug.cgi?id=38690

WebKit image reload on Post/Redirect/Get

http://www.google.com/support/forum/p/Chrome/thread?tid=72bf3773f7e66d68&hl=en

Ellett

Answer 2

You need to remove both the Last-Modified and ETag headers.

By removing the ETag header, you disable caches and browsers from being able to validate files, so they are forced to rely on your Cache-Control and Expires header. Entity tags (ETags) are a mechanism to check for a newer version of a cached file.

Removing both the Last-Modified and ETag header, you will totally eliminate If-Modified-Since and If-None-Match requests and their 304 Not Modified responses, so a file will stay cached without checking for updates until the Expires header indicates new content is available.

More info here: http://www.samaxes.com/2008/04/htaccess-gzip-and-cache-your-site-for-faster-loading-and-bandwidth-saving/.

Unfortunately I don't know how you can turn them off for GAE's static content.

Chipboard

Yeah, agreed...I just need more control in GAE land. This seems like a relatively sizable bug, however. – Ellett

Answer 3

Chrome 9.0, Windows. When loading your home page, default.css, as well as all other .css files are served from cache, without making a request. I think this is a browser-specific behavior, you need to check other browsers as well.

Also, check this Google instructions, they helped me a lot when tuning caching params: http://code.google.com/speed/page-speed/docs/caching.html

Rubino

This could be particular to the mac, but I definitely see this happen in both Chrome and Safari. Since most students use macs at school, this is still a problem worth solving. – Ellett

Answer 4

Since this is an issue with Chrome and Safari, you could use HTML5 App Cache to prevent server calls on static resources completely. Check out an example here.

Natter

True. Feels like a little more work than I should have to do to get rid of those headers, but this may be a last-minute workaround if we decide we have to fix this problem. – Ellett

Answer 5

Your ETag value is fine. ETag does not force revalidation. It just enables it to be more reliable then last modification date. I just browsed to your static content example using Chrome 9, and your content is cached and not unnecessarily revalidated. The problem you saw could be related to "always re-validate" browser settings, which are not the default for most browsers. It could also be a Mac webkit related bug.

Instrumental

Yes, it is only happening on Mac webkit, but the etag is forcing revalidation in some browsers and is the trigger that I need to remove. – Ellett

Answer 6

Try to see whether the same thing happens when you don't hit "enter" or refresh, but simply follow a link. Your browser does something different in those case. Safari particularly only does the requests the way they're supposed to be done if you don't use refresh or explicitly request the page again.

You can try this out very simply on a Mac. Run a simple server with netcat (nc) on some port, let's say 9090:

nc -l 9090

Create a simple page with a link to http://localhost:9090 in it, click it, and watch the headers your nc command displays.

Manually return a response by typing it into nc, e.g. something like

HTTP/1.0 200 OK
ETag: "xyz"
Content-type: text/plain

Some text.

Click the link again and see the If-None-Match header in the request. Do a return after the address in the address bar, and you'll see Safari doesn't send the header.

Carliecarlile

Good advice. My problem is that hitting Enter in the address bar is actually not the real problem (sorry for not stating so), but it's the easiest way I found to reproduce the issue. The real problem is that we see the same behavior when submitting a form on, say, "page A" and redirecting back to the same "page A" after the POST. We do a lot of these form POST submissions, and the extra requests for all static content on "page A" after every redirect are unwanted. – Ellett

I see. Can't help you then, I don't know about GAE configuration options. – Carliecarlile

Answer 7

I posted a new issue to request this feature by appengine-web.xml configuration.

Please check and vote for it!

https://code.google.com/p/googleappengine/issues/detail?id=10339&sort=-id&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log

Sanchez