How to efficiently implement Youtube's Thumbnail preview feature?

Question

I am trying to implement the Youtube's Thumbnail Preview feature for my Simple Video Player. Here's the Snap for it :

Good thing is : It is working smoothly once the Player fetch all the thumbnails previews from a HTTP server.

Bad thing is : Fetching of all Thumbnail previews is taking huge time (20-30 seconds). (For a video (.mp4 file) of 14 minutes (~110 MB), roughly 550 thumbnails previews (160x120) are there)

What I am doing is : When the user will start playing the video, I will make "total_thumbnails" HTTP request to Server to get all of them.

Also-Note :

1. I will do the multiple HTTP request thingy in an Async Task.
2. I will not do it in the fashion, make a request, wait until download complete and then make another request.
3. I will make "total_thumbnails" HTTP requests blindly, so all the request get queued in the pipeline and later receiving responses in parallel.

Extra Details : A HTTP (lighttpd) Server will be running, from where my player will fetch all the thumbnails as soon the user select a video.mp4 to play from the list. Also, the same server will be used by the player to fetch video.mp4 using HTTP streaming.

Problem is : When I will start playing the video, and then I quickly do seeking, I end up seeing this (the white thumbnail is the default one, when the thumbnails mapped to that time is not yet fetch from Server) :

Question is : How efficiently can I fetch all (or some) thumbnail previews so that the User (most of the time) will get the experience of right timely mapped thumbnail?

I have seen youtube's video as soon as the video start (which is quick), the player is able to show all the timely right thumbnails (No matter you drag the thumb to the last minute or hover over the bar's last minutes, almost every time you will see the rightly mapped thumbnail).

Are they downloading all the thumbnails at the same time or downloading the compressed thumbnail previews series or some other intelligent stuff is happening out there?

Has anyone has worked out on this?

Karakalpak

So 14 minutes á 60 seconds and 550 thumbnails make one thumbnail about every (14 * 60) / 550 ≈ 1.53 seconds. Is increasing that interval to let's say 5s an option? – Tripping

Here 1.53 sec (you calculated) is time since http request is made from player to Server till we received the requested thumbnail image.We can't control it, here time taken will depend on http handshaking + image download time from server. – Karakalpak

Oh, okay. But the question still persists: Do you really need that many thumbnails? If you created a thumbnail only every 120 (PAL) or 150 (NTSC) frames, you would greatly cut down on data volume, wouldn't you? – Tripping

Oh, I got you now. Actually, we tried this, the problem was user experience was very bad. How much bad? well while user finished with seeking, the player will seek to nearest previous I-frame, which may not correspond to the thumbnail preview (since previews are 5-sec apart while I-frames are 1-1.5 sec apart) shown to the user. This experience becomes important when video are short (up to 30 minutes). I feel 5-sec or more interval will work in case of big documentaries or movies – Karakalpak

Hm, I see. So what about a rate that isn't fix? After all there are only so many spots users can seek to. So if you take the number of frames in the video and divide it by the number of spots a user can seek to, you should end up with a constant number of thumbnails regardless of the actual video length. Having said that, I really believe cutting down data volume is key to the solution here. – Tripping

Do I understand correctly that you make 550 separate requests to the server? That’s obviously asking for optimization. – Restoration

Indeed, and i need all those 550, plus-minus 10 is acceptable. I need to download them as quickly as possible, so that while the user is doing seek I can have valid thumbnail previews to show. – Karakalpak

Answer 1

1. Group multiple thumbnails into single container image. Use canvas (I'm not a JS developer but believe that this is the right word) to extract each of thumbnails separately on client side. For example, here is example of such container image used by youtube. This works well with all sorts of protocols (e.g. with or without keep-alive enabled).
2. Preprocess thumbnails to reduce their size. Reduce jpeg quality as much as possible (~q=70). Also you may try to blur thumbnails a bit or to reduce number of colors.
3. Optimize order of downloads. For example, if you have video with length 2:55:
   1. First, download container image with 8 thumbs covering full range of video time: 0:00, 0:25, 0:50, 1:15, 1:40, 2:05, 2:30, 2:55. This makes your service to appear as "working instantly" for slow clients.
   2. Next, download 4 container images with 32 thumbs total covering full range of video, but more densely: 0:00, 0:06, 0:11, 0:17, ...
   3. Now, download gradually all other thumbnails without any particular order.

Darg

I will try this approach next, I will post the result of how it worked out for me, Still youtube Quickness and accuracy is overwhelming, within 1-2 secs you can see and feel the right time thumbnails loaded all over scrub bar (hover or drag the thumb, it doesn't matter) – Karakalpak

Answer 2

For a video (.mp4 file) of 14 minutes (~110 MB), roughly 550 thumbnails previews (160x120) are there

The main factor here is probably that you make 550 separate requests to the HTTP server. I assume that you do it like this: request thumbnail k, wait for it to download, then request thumbnail k+1. This is very inefficient, because the HTTP server is sitting idle while thumbnail k is being downloaded and the next request is being uploaded.

Solution 1

Combine all 550 thumbnails into one big file, and request that instead of 550 individual files.

Maybe there is a good existing file format that you could reuse for this purpose. Tar comes to mind, but it’s probably a bad choice, because (1) it doesn’t seem to support random access (i.e. get the kth thumbnail directly), and (2) it adds 512 bytes of overhead per thumbnail.

Anyway, it should be easy to come up with your own file format. Something like this:

• the first 4 bytes give n, which is the number of thumbnails in the file;
• the next 4×n bytes give the offset of each thumbnail relative to the beginning of the file;
• after that, just the thumbnails themselves, stitched end-to-end.

Solution 2

Use HTTP pipelining—a feature of HTTP/1.1 where you send many requests (perhaps all 550) at once, then read many responses at once. You don’t have to wait for each thumbnail to download before you request the next one.

You will need two things for this to work.

First, your HTTP client must support HTTP pipelining. I don’t know what’s the state of the art on your platform, but in Python land this is a rare feature. One client that seems to support it is libcurl (via CURLMOPT_PIPELINING and CURLMOPT_MAX_PIPELINE_LENGTH). libcurl is available for most platforms and has bindings for most languages.

Second, you may need to change your Lighttpd configuration. By default, the server.max-keep-alive-requests variable is set to 16, which means that the server will close a connection after handling 17 requests on it, and you will have to establish a new one. You probably want this number to be larger.

Pipelining a large number of requests may (or may not) have bad side effects on both the client and the server, such as unexpected memory usage. Please test for yourself.

Also, if there are any HTTP intermediaries (like proxies) between the client and the server, they can break HTTP pipelining.

Measurements

I ran some quick and dirty tests. Lighttpd 1.4.31 serves 550 thumbnails (4.2M total), with server.max-keep-alive-requests = 600, from Berlin. Client is in Moscow. Average over 10 runs.

• Simple approach (request-download-request, using Python’s http.client): 27.3 s.
• Combined file (using Python’s http.client): 2.95 s.
• Pipelining 550 requests (no real HTTP client, just raw sockets): 3.04 s.

Restoration

As for your Para before Solution 1 Tag, please check my edits - the Also--Note part. And as per Solution-1 you proposed we tried that and voila.!!, it took 3 seconds for fetching all thumbnails, but the problem is the tar file or any other user-defined file will be around 6-7 MB (for bigger videos, size can be even more), and the platform in which I want this, we are restricted to make request of response size of not more than 64K, so for 7 MB I have to make more than ~110 HTTP request. – Karakalpak

And I am using HttpURLConnection class, to make all requests, it does support HTTP pipelining, it stills takes 20 secs around, I think the server.max-keep-alive-requests for my Lighttpd is less, plus Server config changes aren't allowed. – Karakalpak

@Karakalpak Try using a sniffer to check that pipelining is happening as expected: that you aren’t waiting for each response, and aren’t pipelining too many requests on each connection (every pipelined request after max-keep-alive-requests + 1 is wasted). – Restoration

Solution 1 is pretty solid. But why come up with a new fileformat? Just create a giant image and use time * thumbnail_width as an offset. Oh and tar does support random seeking. After all it's been designed for tape archives. Compressed tars OTOH ... tough luck. – Tripping

@Tripping I too also agree with solution 1, but the problem is, in my state of the art this solution may not work because the server will only send maximum 64K bytes for one request. So, to retrieve the data with more than 64K in size, you need to make two or more request. So maybe I feel solution 1 can safely be eschewed. – Karakalpak

@Karakalpak I've been using lighttpd for years and never encountered such a limitation. What is the cause of that? – Tripping

Oops, Sorry for confusing you, the Server which I am finally going to use has a proprietary implementation (and hence, some restrictions are incorporated), This Lighttpd Server I am using only as a test server to realize the preview feature efficiently since uploading this feature in actual field environment required myriad changes.(you can refer the very first comment I've already mentioned this) – Karakalpak

@Karakalpak Ouch, I somehow missed that. That's a hefty constraint. I'd suggest using a ranged request to compensate for that, but it looks like it weren't worth the overhead in this case. – Tripping

Answer 3

There is another solution which will cause the image files to be larger however much easier to display on an html page. As others suggested you can create a sprite sheet however not the old fashioned way. You can combine your thumbnails as a comma/colon separated list of base64 images.

E.g. {base64_encoded_image},{time},..., extract this into tuples of image/time and then display it. Html pages are capable of displaying base64 encoded images by default so no further processing will be required on the client.

NOTE I must warn you that base64 will be approximately 37% larger. Gziped it will be comparable however Gziped binary will still be smaller. As base64 uses 6 bits per each 8 bits. binary uses all 8 bits.

Unlock