Video streaming to ipad does not work with Tapestry5

Question

I want to stream a video to my IPad via the HTML5 video tag with tapestry5 (5.3.5) on the backend. Usually the serverside framework shouldn't even play a role in this but somehow it does.

Anyway, hopefully someone here can help me out. Please keep in mind that my project is very much a prototype and that what I describe is simplified / reduced to the relevant parts. I would very much appreciate it if people didn't respond with the obligatory "you want to do the wrong thing" or security/performance nitpicks that aren't relevant to the problem.

So here it goes:

Setup

I have a video taken from the Apple HTML5 showcase so I know that format isn't an issue. I have a simple tml page "Play" that just contains a "video" tag.

Problem

I started by implementing a RequestFilter that handles the request from the video control by opening the referenced video file and streaming it to client. That's basic "if path starts with 'file' then copy file inputstream to response outputstream". This works very well with Chrome but not with the Ipad. Fine, I though, must be some headers I'm missing so I looked at the Apple Showcase again and included the same headers and content type but no joy.

Next, I though, well, let's see what happens if I let t5 serve the file. I copied the video to the webapp context, disabled my request filter and put the simple filename in the video's src attribute. This works in Chrome AND IPad. That surprised me and prompted me to look at how T5 handles static files / context request. Thus far I've only gotten so far as to feel like there are two different paths which I've confirmed by switching out the hardwired "video src" to an Asset with a @Path("context:"). This, again, works on Chrome but not on IPad.

So I'm really lost here. What's this secret juice in the "simple context" requests that allow it to work on the IPad? There is nothing special going on and yet it's the only way this works. Problem is, I can't really serve those vids from my webapp context ...

Solution

So, it turns out that there is this http header called "Range" and that the IPad, unlike Chrome uses it with video. The "secret sauce" then is that the servlet handler for static resource request know how to deal with range requests while T5's doesn't. Here is my custom implementation:

        OutputStream os = response.getOutputStream("video/mp4");
        InputStream is = new BufferedInputStream( new FileInputStream(f));
        try {
            String range = request.getHeader("Range");
            if( range != null && !range.equals("bytes=0-")) {
                logger.info("Range response _______________________");
                String[] ranges = range.split("=")[1].split("-");
                int from = Integer.parseInt(ranges[0]);
                int to = Integer.parseInt(ranges[1]);
                int len = to - from + 1 ;

                response.setStatus(206);
                response.setHeader("Accept-Ranges", "bytes");
                String responseRange = String.format("bytes %d-%d/%d", from, to, f.length());
                logger.info("Content-Range:" + responseRange);
                response.setHeader("Connection", "close");
                response.setHeader("Content-Range", responseRange);
                response.setDateHeader("Last-Modified", new Date().getTime());
                response.setContentLength(len);
                logger.info("length:" + len);

                byte[] buf = new byte[4096];
                is.skip(from);
                while( len != 0) {

                    int read = is.read(buf, 0, len >= buf.length ? buf.length : len);
                    if( read != -1) {
                        os.write(buf, 0, read);
                        len -= read;
                    }
                }


            } else {
                    response.setStatus(200);
                    IOUtils.copy(is, os);
            }

        } finally {
            os.close();
            is.close();
        }

Bodwell

Answer 1

I want to post my refined solution from above. Hopefully this will be useful to someone.

So basically the problem seemed to be that I was disregarding the "Range" http request header which the IPad didn't like. In a nutshell this header means that the client only wants a certain part (in this case a byte range) of the response.

This is what an iPad html video request looks like::

[INFO] RequestLogger Accept:*/*
[INFO] RequestLogger Accept-Encoding:identity
[INFO] RequestLogger Connection:keep-alive
[INFO] RequestLogger Host:mars:8080
[INFO] RequestLogger If-Modified-Since:Wed, 10 Oct 2012 22:27:38 GMT
[INFO] RequestLogger Range:bytes=0-1
[INFO] RequestLogger User-Agent:AppleCoreMedia/1.0.0.9B176 (iPad; U; CPU OS 5_1 like Mac OS X; en_us)
[INFO] RequestLogger X-Playback-Session-Id:BC3B397D-D57D-411F-B596-931F5AD9879F

It means that the iPad only wants the first byte. If you disregard this header and simply send a 200 response with the full body then the video won't play. So, you need send a 206 response (partial response) and set the following response headers:

[INFO] RequestLogger Content-Range:bytes 0-1/357772702
[INFO] RequestLogger Content-Length:2

This means "I'm sending you byte 0 through 1 of 357772702 total bytes available".

When you actually start playing the video, the next request will look like this (everything except the range header ommited):

[INFO] RequestLogger Range:bytes=0-357772701

So my refined solution looks like this:

OutputStream os = response.getOutputStream("video/mp4");

        try {
                String range = request.getHeader("Range");
                /** if there is no range requested we will just send everything **/
                if( range == null) {
                    InputStream is = new BufferedInputStream( new FileInputStream(f));
                    try {
                        IOUtils.copy(is, os);
                        response.setStatus(200);
                    } finally {
                        is.close();
                    }
                    return true; 
                }
                requestLogger.info("Range response _______________________");


                String[] ranges = range.split("=")[1].split("-");
                int from = Integer.parseInt(ranges[0]);
                /**  
                 * some clients, like chrome will send a range header but won't actually specify the upper bound.
                 * For them we want to send out our large video in chunks.
                 */
                int to = HTTP_DEFAULT_CHUNK_SIZE + from;
                if( to >= f.length()) {
                    to = (int) (f.length() - 1);
                }
                if( ranges.length == 2) {
                    to = Integer.parseInt(ranges[1]);
                }
                int len = to - from + 1 ;

                response.setStatus(206);
                response.setHeader("Accept-Ranges", "bytes");
                String responseRange = String.format("bytes %d-%d/%d", from, to, f.length());

                response.setHeader("Content-Range", responseRange);
                response.setDateHeader("Last-Modified", new Date().getTime());
                response.setContentLength(len);

                requestLogger.info("Content-Range:" + responseRange);
                requestLogger.info("length:" + len);
                long start = System.currentTimeMillis();
                RandomAccessFile raf = new RandomAccessFile(f, "r");
                raf.seek(from);
                byte[] buf = new byte[IO_BUFFER_SIZE];
                try {
                    while( len != 0) {
                        int read = raf.read(buf, 0, buf.length > len ? len : buf.length);
                        os.write(buf, 0, read);
                        len -= read;
                    }
                } finally {
                    raf.close();
                }
                logger.info("r/w took:" + (System.currentTimeMillis() - start));




        } finally {
            os.close();

        }

This solution is better then my first one because it handles all cases for "Range" requests which seems to be a prereq for clients like Chrome to be able to support skipping within the video ( at which point they'll issue a range request for that point in the video).

It's still not perfect though. Further improvments would be setting the "Last-Modified" header correctly and doing proper handling of clients requests an invalid range or a range of something else then bytes.

Bodwell

This is useful information; there's no reason why Tapestry can't handle this automatically inside the standard asset handling code; we're just not aware that it needs doing. Adding this level of information to our JIRA is the first step. – Stump

Answer 2

I suspect this is more about iPad than about Tapestry.

I might invoke Response.disableCompression() before writing the stream to the response; Tapestry may be trying to GZIP your stream, and the iPad may not be prepared for that, as video and image formats are usually already compressed.

Also, I don't see a content type header being set; again the iPad may simply be more sensitive to that than Chrome.

Stump

Hi Howard. I think it's great that you take the time to answer T5 (a great framework) here on Stackoverflow. Anyway, I found out what the problem was and added the solution to my question. The TL;DR version is that the iPad doesn't like it if you disregard the "Range" http request header. This might be a problem for T5 also because from what I say, when the framework is serving an asset it will disregard the Range header also. I'll post an answer with more detail. – Bodwell