How to store 40.000 images using HTML5 or Phonegap?

Question

I have searched a long time for a way to store a large number of images in HTML5, or access them stored locally and display them.

The application is a product ordering and catalog, like an e-commerce web site but for bulk orders, not final clients, each image is a product, that is why there is a need to have 40k images. The need comes from the requirement that the salesperson can use the tablet in offline mode for days, weeks, and only sync with the ERP once he has connection or wants to sync.

Each salesperson has a dedicated Android tablet Tegra3 Quad-Core, 32Gb for the application. But the application also supports iPad and Chrome on the desktop.

However I have faced the following limitations:

1. Use html5 offline (app cache) - This is limited in iOS devices to 50Mb, and hard to manage the manifest file.
2. Use webSQL database with base64 encoded images. Same size limitations (50Mb).
3. Use localCache with base64, again size limitations (5 to 10Mb).
4. Convert my HTML5 app to use PhoneGap File API, but then I lose desktop Support for Windows PCs, and have some issues with web services.

So perhaps I am missing an alternative way to do this?

Is there an HTML5 way to store a large number of images for offline viewing? FileSystem API is not supported in mobile devices and being able to run on mobile devices is a strong requirement.

Sheepdog

Why do you need to save all 40k images? The limitations you described are there for a reason - so that apps won't hog a device's storage. Try to pare down the list dynamically, based on what the user actually needs to view offline. – Factional

It's a product catalog each product has an image there are 43.000 products. That is why I need to have the 40k images, updated question to reflect the requirement. – Sheepdog

This is a horrible idea. You need to go back to your client and find out what there top sellers are and cache those images in the app. Otherwise load the images remotely as required. – Cosmogony

@SimonMacDonald All products need to be available in offline mode, that is the challenge, this is an HTML5 application that allows you to perform orders offline and send them once you have connection. The application needs to work in offline mode. The application is already implemented, only the image catalog is missing. – Sheepdog

Answer 1

The solution is going to pass to a port of the HTML5 application to a phoneGap application using the File API.

HTML5 different implementations is the reason for this decision. Since AppCache is very limited in most mobile browsers, as well as the lack of support of the FileSystem API. The fact that you can only store from 16-52Mb on mobile browsers is a limiting factor to a HTML5 web app that requires large amounts of locally stored data (available offline).

Sheepdog

I'm curious if you achieved your object storing thousands of images on a device, with an HTML5 Cordova app, using the File API. And if so, if you have any resources which helped out. I'm basically in the same situation. – Ramsgate

Hi Brad, in the end i created a zip and unzip with multiple images. hundreads is ok, but thousands is not very easy to manage. Its possible, its just not very nice. – Sheepdog

Ah yes, our native iOS app uses zipped assets, makes sense. – Ramsgate

Hey @Sheepdog Did you try the github.com/chrisben/imgcache.js plugin? any conclusion on supporting 40k images with cordova webapp? – Concoct

Answer 2

Shouldn't Phonegap's own Implementation of the Filesystem API work on mobile? I mean that's why Phonegap was created in the first place.

Straight from the Cordova API Docs for FileReader:

Supported Platforms

• Android
• BlackBerry WebWorks (OS 5.0 and higher)
• iOS
• Windows Phone 7 and 8
• Windows 8

Psychosis

I am testing that solution right now. The issue is that it then does not work on Desktop that also has to be supported – Sheepdog

From the looks of it the Phonegap FS API should be available with the stock implementation of the FS Api, implemented by Desktop Browsers. Otherwise I would create a very thin abstraction layer. – Psychosis

Thanks for your feedback, unfortunatly it looks like I will have no other alternative. Ill have to port the application to PhoneGap, the webSQL and webServices, so not trivial... sucks that I can't find a pure HTML5 way of doing this. But I had to try and call out on the community. – Sheepdog

I don't quite understand why you have to port it. All you need is a simple abstraction layer for Phonegap and the HTML5 Filesystem API. I would design that layer in rather high level way and NOT try to abstract the whole FS API. – Psychosis

I have to port my current HTML5 application that is already in production for over 6 months to phonegap and then start the development of this phase of the project. – Sheepdog

Ah now I get it. I misread the title of the question. I was under the assumption you already have Phonegap project. – Psychosis

At the time HTML5 was enough, now that there is a need to meet some very specific requirements there is a need to access more resources than those that are available in HTML5 – Sheepdog

Answer 3

Random though, sprite sheet? would make just one file and would minimize requests

Efflorescence

Welcome to Stack Overflow! Can you elaborate your answer please and give more information on how to do it and what advantages it brings? – Aftergrowth

I actually thought about that. However there is a need to reference the images by name, and there isn't an easy way to group them. – Sheepdog

Answer 4

How about running a local webserver that can only accept local connections? In Apache, somthing like "Allow from 127.0.0.1" in the configuration file should do the trick. Using a local webserver wold let you sidestep any HTML limitations.

Alwin

Hum that is something I had not thought about... But can you run a local web server on a iPad or Android Tablet? Ill try to dig this a bit depper, also Ill need to see how to handle this on the client side – Sheepdog

And also there is the issue of updating the local web server with new images... still and interesting option to investigate. – Sheepdog

Answer 5

You can simply store it in your sqlite database. You can add one more field (column) to your product detail table.

You can store image either in BLOB format OR in base64 format. (in your database).

using phoneGap you can easily access your database and different tables from your DB.

For more performance and to save bandwidth : What you can do is you can apply lazy loading of images. When ever you download new image just store it into database.

This way you can avoid the I/O Operations and also if you are using MVC then in your product detail object (Modal class) you will have image object also.

hope this suggestion works for you :)

Metrical