HTTP Range header

Question

I was reading http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 and trying to figure out how to continue a file download.

For example, suppose a file is of length 100 bytes and I have all the 100 bytes. However, I don't know what the expected file size should be, so I ask for the file and specify a Range header that looks like this:

Range: bytes=100-

Is this a valid Range request?

Snowflake

Erm, the example under it cites 'bytes=9500-' as valid, so.... – Cryptocrystalline

The most current ref is RFC7233 -- httpwg.github.io/specs/rfc7233.html – Fabria

You can make a HEAD request first and check the file length. – Surbeck

Answer 1

It's a syntactically valid request, but not a satisfiable request. If you look further in that section you see:

If a syntactically valid byte-range-set includes at least one byte- range-spec whose first-byte-pos is less than the current length of the entity-body, or at least one suffix-byte-range-spec with a non- zero suffix-length, then the byte-range-set is satisfiable. Otherwise, the byte-range-set is unsatisfiable. If the byte-range-set is unsatisfiable, the server SHOULD return a response with a status of 416 (Requested range not satisfiable). Otherwise, the server SHOULD return a response with a status of 206 (Partial Content) containing the satisfiable ranges of the entity-body.

So I think in your example, the server should return a 416 since it's not a valid byte range for that file.

Heritor

So is there any way a client can resume a download without making a HEAD call to first figure out the content length and then do the math and fetch the actual content? I mean some sort of open addressing like "give me all bytes after such and such byte..." – Snowflake

The client will already know if it has all of the data from the original request - it should have either received a Content-Length header in the original response, or if it was chunked encoding it will have received a zero-length chunk to indicate the response was complete. If you haven't saved off this state and just have a chunk of bytes on disk, then yes you'll have to either do a HEAD request or use the Range header to ask for a byte range, and if you get back a 416 response you know you have all the bytes. – Heritor

I think Expect-Continue lets you do streaming chunks more or less like desired? – Rodas

@MarcNovakowski Actually, consider the case of wget and using the -c flag. Since wget doesn't maintain any metadata about the file being complete, suppose the size of the file on disk is 99 bytes. wget will request the byte range "100-", and I feel that the server should respond with a 0 length response since the request is just 1 past the end of the file. – Snowflake

Answer 2

As Wrikken suggested, it's a valid request. It's also quite common when the client is requesting media or resuming a download.

A client will often test to see if the server handles ranged requests other than just looking for an Accept-Ranges response. Chrome always sends a Range: bytes=0- with its first GET request for a video, so it's something you can't dismiss.

Whenever a client includes Range: in its request, even if it's malformed, it's expecting a partial content (206) response. When you seek forward during HTML5 video playback, the browser only requests the starting point. For example:

Range: bytes=3744-

So, in order for the client to play video properly, your server must be able to handle these incomplete range requests.

You can handle the type of 'range' you specified in your question in two ways:

First, You could reply with the requested starting point given in the response, then the total length of the file minus one (the requested byte range is zero-indexed). For example:

Request:

GET /BigBuckBunny_320x180.mp4 
Range: bytes=100-

Response:

206 Partial Content
Content-Type: video/mp4
Content-Length: 64656927
Accept-Ranges: bytes
Content-Range: bytes 100-64656926/64656927

Second, you could reply with the starting point given in the request and an open-ended file length (size). This is for webcasts or other media where the total length is unknown. For example:

Request:

GET /BigBuckBunny_320x180.mp4
Range: bytes=100-

Response:

206 Partial Content
Content-Type: video/mp4
Content-Length: 64656927
Accept-Ranges: bytes
Content-Range: bytes 100-64656926/*

Tips:

You must always respond with the content length included with the range. If the range is complete, with start to end, then the content length is simply the difference:

Request: Range: bytes=500-1000

Response: Content-Range: bytes 500-1000/123456

Remember that the range is zero-indexed, so Range: bytes=0-999 is actually requesting 1000 bytes, not 999, so respond with something like:

Content-Length: 1000
Content-Range: bytes 0-999/123456

Or:

Content-Length: 1000
Content-Range: bytes 0-999/*

But, avoid the latter method if possible because some media players try to figure out the duration from the file size. If your request is for media content, which is my hunch, then you should include its duration in the response. This is done with the following format:

X-Content-Duration: 63.23 

This must be a floating point. Unlike Content-Length, this value doesn't have to be accurate. It's used to help the player seek around the video. If you are streaming a webcast and only have a general idea of how long it will be, it's better to include your estimated duration rather than ignore it altogether. So, for a two-hour webcast, you could include something like:

X-Content-Duration: 7200.00 

With some media types, such as webm, you must also include the content-type, such as:

Content-Type: video/webm 

All of these are necessary for the media to play properly, especially in HTML5. If you don't give a duration, the player may try to figure out the duration (to allow for seeking) from its file size, but this won't be accurate. This is fine, and necessary for webcasts or live streaming, but not ideal for playback of video files. You can extract the duration using software like FFMPEG and save it in a database or even the filename.

X-Content-Duration is being phased out in favor of Content-Duration, so I'd include that too. A basic, response to a "0-" request would include at least the following:

HTTP/1.1 206 Partial Content
Date: Sun, 08 May 2013 06:37:54 GMT
Server: Apache/2.0.52 (Red Hat)
Accept-Ranges: bytes
Content-Length: 3980
Content-Range: bytes 0-3979/3980
Content-Type: video/webm
X-Content-Duration: 2054.53
Content-Duration: 2054.53

One more point: Chrome always starts its first video request with the following:

Range: bytes=0-

Some servers will send a regular 200 response as a reply, which it accepts (but with limited playback options), but try to send a 206 instead to show than your server handles ranges. RFC 2616 says it's acceptable to ignore range headers.

Anglim

What do you do if the content is a live video stream that does not have a fixed duration? – Heehaw

@Joel, you need to reply with a duration even if you don't know it. In that case, just try 0.0. For the client, the duration doesn't matter anyway since you can't usually scan a live stream. If the 0.0 doesn't work, just try something really high like 1000000.00. – Anglim

@VictorStoddard can chunked streaming be applied to regular file download where no Range header is present in client request? How should server respond in that case? – Iqbal

@Iqbal There isn't much difference other than using the Transfer-Encoding header instead of Content-Length in Chunked Transfer Encoding. Chunks can come from a single file and the server can set the chunk size. The client should buffer and piece together the chunks as they're received. Alternatively, HTTP Streaming uses prerecorded segments of a media file, where they are saved on the server as individual parts (ts files). These segments are served using regular HTTP file GET requests obtained from an index file. I've found segmenting to be tricky but that was years ago. – Anglim

Content-Length: 64656927 Accept-Ranges: bytes Content-Range: bytes 100-64656926 Why is Content-Length not '64656827'? – Macrobiotic

@Macrobiotic the content range is 0-indexed but the content length is 1-indexed. – Anglim

Answer 3

Contrary to Mark Novakowski answer, which for some reason has been upvoted by many, yes, it is a valid and satisfiable request.

In fact the standard, as Wrikken pointed out, makes just such an example. In practice, Apache responds to such requests as expected (with a 206 code), and this is exactly what I use to implement progressive download, that is, only get the tail of a long log file which grows in real time with polling.

Crigger

Please read Marc Novakowki's answer again. "satisfiable" has a particular meaning in the RFC, which he quoted. This request is not satisfiable because the requested bytes are beyond the length of the file. – Ansley

Firefox is not the software element that responds to the request, it is an http server – Wowser

The request is perfectly valid, but the last byte of the the 100byte file of the example is at position 99. Therefore the request '100-' is not satisfyable in this particular case. – Leandroleaning

Answer 4

For folks who are stumbling across Victor Stoddard's answer above in 2019, and become hopeful and doe eyed, note that:

a) Support for X-Content-Duration was removed in Firefox 41: https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Releases/41#HTTP

b) I think it was only supported in Firefox for .ogg audio and .ogv video, not for any other types.

c) I can't see that it was ever supported at all in Chrome, but that may just be a lack of research on my part. But its presence or absence seems to have no effect one way or another for webm or ogv videos as of today in Chrome 71.

d) I can't find anywhere where 'Content-Duration' replaced 'X-Content-Duration' for anything, I don't think 'X-Content-Duration' lived long enough for there to be a successor header name.

I think this means that, as of today if you want to serve webm or ogv containers that contain streams that don't know their duration (e.g. the output of an ffpeg pipe) to Chrome or FF, and you want them to be scrubbable in an HTML 5 video element, you are probably out of luck. Firefox 64.0 makes a half hearted attempt to make these scrubbable whether or not you serve via range requests, but it gets confused and throws up a spinning wheel until the stream is completely downloaded if you seek a few times more than it thinks is appropriate. Chrome doesn't even try, it just nopes out and won't let you scrub at all until the entire stream is finished playing.

Thomasthomasa

Here is a long thread from the FF developers talking about supporting these kinds of files. bugzilla.mozilla.org/show_bug.cgi?id=657791 – Thomasthomasa

Answer 5

If you're trying to make a request for content with an unknown length, and you want it to return continuous (or aggregating) response then you may want to consider use of the approach suggested in RFC8673 - namely set the last-byte-pos to 2^^53-1 so your request would look something like this:

GET /resource HTTP/1.1
Host: example.com
Range: bytes=0-9007199254740991

Rutkowski