ReST low latency - how should I reply to a GET while an upload is pending?

Question

I am designing a ReST API which follows the basic CRUD pattern.

My API can receive a request to update a resource which may take a short time to process. Ideally I would like to inform clients that a new version is about to be available and that there is some uncertainty over when the version I have cached actually expires.

So the process I intend to use something like this (improvements welcome):

client: GET /some/item
myapi:  200 OK
        last-modified: time-stamp-of-v1
        etag: some-hash-relating-to-v1-of-my-item-in-this-format
        content: json or whatever
         data/for/some/item/v1... 

client: PUT /some/item
        if-match: some-hash-relating-to-v1-of-my-item-in-this-format
        content: json or whatever
         data/for/some/item/v2... 

myapi:  202 ACCEPTED,   
        content: json or whatever
         time-accepted: time-stamp-after-v1-but-before-v2
         your item will be at /some/item
         here is a URI /some/taskid to track progress

while upload is pending:

client: GET /some/item
myapi:  200 OK
        some/item ... 
        last-modified: time-stamp-of-v1
        etag: some-hash-relating-to-v1-of-my-item-in-this-format
>>>>    expires: time-stamp-after-v1-but-before-v2 <<<
>>>>    warning: 110 Response is stale    <<<<
        content: json or whatever
         data/for/some/item/v1... 

client: GET /some/task/id
myapi:  200 OK
        content: json or whatever
         time-accepted: time-stamp-after-v1-but-before-v2
         your item will be at /some/item
         status/of/upload/v2... 

after task completed:

client: GET /some/item
myapi:  200 OKAY
        some/item/v2 ... 
        last-modified: time-stamp-of-v2
        etag: some-hash-relating-to-v2-of-my-item-in-this-format
        content: json or whatever
         data/for/some/item/v2... 

client: GET /some/task/id
myapi:  303 SEE OTHER
         look-here: /some/item

If you are a proxy and know know your content is stale you can put "warning: 110 - response is stale" in the header. However, in this case the data is not actually invalid yet. I would like to say that I can guarantee it is valid up until the time I received and passed on the upload request (time-stamp-after-v1-but-before-v2 or later as if I am in contact with the upload server). It hasn't really expired at the time I receive the upload request. I just expect its going to. (In fact if the request fails it might not be updated at all).

Now the default choice is just to serve the old content and let the client catch up on its own. This has high latency. If possible, I would like to do better.

For example, if the client knows the document is about to expire it could poll more often or it could try to upgrade the connection to a web-socket and get sent an update the moment I get it (would that still count as ReST?)

There is another case where using expired data must be avoided at all costs. For that scenario I think I want to tell the client that the resource is temporarily unavailable. Using the warning and expires fields as I have above seems correct there. Though it might be better to send a 503 with a suitable retry-after header.

So the question is: how should I reply to a GET while the upload of a new version is pending?

In anticipation of answers along the lines of use a messaging framework like AMQP or zeroMQ instead for low latency, I should point out this API is acting as a AMQP gateway/proxy for clients unwilling to use AMQP directly. Information on using webhooks or websockets would be still be interesting.

Some related useful content is:

• How to proper design a restful API to invalidate a cache?
• https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
• HTTP status code for temporarily unavailable pages
• http://www.albertoleal.me/posts/how-to-prevent-race-conditions-in-restful-apis.html (the etag prevents races from simultaneously uploads)

Basrhin

Answer 1

Tl;Dr;

While upload is pending send:

client: GET /some/item
myapi:  200 OK
        some/item ... 
        last-modified: time-stamp-of-v1
        etag: some-hash-relating-to-v1-of-my-item-in-this-format
        expires: time-stamp-after-v1-but-before-v2
        stale-while-revalidate: 100
        warning: 110 Response is stale
        content: json or whatever
         data/for/some/item/v1... 

At first sight it looks like using Warning is not correct. See https://www.rfc-editor.org/rfc/rfc7234#section-5.5.0

In this case the server is acting as a proxy (though not an HTTP proxy). It is not disconnected from AMQP and "A proxy MUST NOT send stale responses" unless it is disconnected. This is annoying as it looked like the right thing to do here.

4.2.4. Serving Stale Responses

A "stale" response is one that either has explicit expiry information or is allowed to have heuristic expiry calculated, but is not fresh according to the calculations in Section 4.2.

A cache MUST NOT generate a stale response if it is prohibited by an explicit in-protocol directive (e.g., by a "no-store" or "no-cache" cache directive, a "must-revalidate" cache-response-directive, or an applicable "s-maxage" or "proxy-revalidate" cache-response-directive; see Section 5.2.2).

**> A cache MUST NOT send stale responses unless it is disconnected

(i.e., it cannot contact the origin server or otherwise find a
forward path) or doing so is explicitly allowed (e.g., by the
max-stale request directive; see Section 5.2.1).**

A cache SHOULD generate a Warning header field with the 110 warn-code (see Section 5.5.1) in stale responses. Likewise, a cache SHOULD generate a 112 warn-code (see Section 5.5.3) in stale responses if the cache is disconnected.

A cache SHOULD NOT generate a new Warning header field when
forwarding a response that does not have an Age header field, even if the response is already stale. A cache need not validate a response
that merely became stale in transit.

Also

4.4. Invalidation

Because unsafe request methods (Section 4.2.1 of [RFC7231]) such as PUT, POST or DELETE have the potential for changing state on the
origin server, intervening caches can use them to keep their contents up to date.

**> A cache MUST invalidate the effective Request URI (Section 5.5 of

[RFC7230]) as well as the URI(s) in the Location and Content-Location response header fields (if present) when a non-error status code is
received in response to an unsafe request method.**

However a warning is required if stale-while-revalidate is used (see https://www.rfc-editor.org/rfc/rfc5861)

3. The stale-while-revalidate Cache-Control Extension

When present in an HTTP response, the stale-while-revalidate Cache- Control extension indicates that caches MAY serve the response in
which it appears after it becomes stale, up to the indicated number
of seconds.

 stale-while-revalidate = "stale-while-revalidate" "=" delta-seconds

If a cached response is served stale due to the presence of this
extension, the cache SHOULD attempt to revalidate it while still
serving stale responses (i.e., without blocking).

I thought this was unclear so I submitted an errata. This was rejected (though at the time of writing its still showing as reported) on the grounds that the cache control extensions in rfc5861 override the MUST NOT in rfc7234 ("doing so is explicitly allowed" see above).

It is okay to use expires but its not very helpful as it doesn't imply anything.

5.3. Expires

The "Expires" header field gives the date/time after which the
response is considered stale. See Section 4.2 for further discussion of the freshness model.

**> The presence of an Expires field does not imply that the original

resource will change or cease to exist at, before, or after that
time.**

Basrhin