Querying the MediaStore: Joining thumbnails and images (on ID)

Question

I'm working on a "photo gallery"-type app for Android. It started as a Final Project for the Developing Android Apps at Udacity, so it's overall structure (activities, contentproviders etc) should be very sound, and it was accepted for certification by Udacity/Google.

However, it's still not 100% finished, and I'm still trying to improve on it.

What I want to do should really be quite straight-forward; load all images on device (as thumbnails) into a GridView in the MainActivity, with a DetailActivity which shows the full size image + some meta data (title, size, date etc).

The course required us to write a ContentProvider, so I've got a query() function which essentially fetches data from the MediaStore, and returns a cursor to MainActivity's GridView. On my device, at least, (Sony Xperia Z1, Android 5.1.1) this works almost perfectly. There are some bugs and quirks, but by and large I can consistently find all images on my phone in my app, and click on them to view details.

However, when I tried installing the app on my friend's Sony Xperia Z3, everything failed. No images showed up, although I obviously checked there were in fact ~100 photos on his phone. Same on another friend's phone (brand new Samsung S6) :-(

This is The Main Problem. On my phone, where stuff works, the "secondary" bugs involve when a new photo is taken by the camera, it's not automatically loaded into my app (as a thumbnail). It seems I need to figure out how to trigger a scan, or whatever is needed to load/generate new thumbs. That's also quite high on my wish list.

As I said, I'm confident all this really ought to be quite simple, so maybe all my difficulties indicate I'm approaching the problem in the entirely wrong way? Here's what my query() function is doing:

1. get a cursor of all thumbnails, from MediaStore.Media.Thumbnails.EXTERNAL_CONTENT_URI

2. get a cursor of all images, from MediaStore.Media.Images.EXTERNAL_CONTENT_URI

3. join these, on MediaStore.Media.Thumbnails.IMAGE_ID = MediaStore.Media.Images._ID using a CursorJoiner

4. return the resulting retCursor (as produced in the join)

-- please find full code in this previous post.

Although this looks correct (to me), maybe it's really not the way to go about this? I'm joining thumbs and images, by the way, such that I can show some meta data (e.g. date taken) along with the thumbnail, in the GridView. I've identified the problem to the joining, in particular, because if I simplify this to only loading thumbs into the GridView, then this all works fine -- also on my friend's phone. (With the exception of loading new photos.)

Somehow, my assumption that IMAGE_ID and _ID are always consistent is not correct? I've seen a post on AirPair, describing a similar gallery app, and there the tutorial actually goes about this slightly differently. Rather than attempting to join cursors, he gets the thumbnails cursor and iterates over it, adding data from Images using individual queries to the MediaStore... But is that the most efficient way to do this? - Nevertheless, his solution does join the thumbnail to the corresponding image on ID:

Cursor imagesCursor = context.getContentResolver().query(
            MediaStore.Images.Media.EXTERNAL_CONTENT_URI, 
            filePathColumn, 
            MediaStore.Images.Media._ID + "=?", new String[]{imageId},  // NB!
            null);

In summary, I need help with the following:

• am I querying the MediaStore correctly?
• is it safe to join thumbs and images, on ID -- will that be stable/in sync at all times?
• how can my app automatically generate/fetch thumbnails of new images?

Dorcas

Relevant: #3680857 – Lucey

So, joining cursors is maybe not so extremely superior to just getting images, and iterating over the cursor to fetch each individual thumbnail? – Dorcas

So I take it the answer to my questions is to query images table directly, and only get thumbnails individually, in secondary queries. As I understand it, it's not always the case that thumbnails even exist, that the table is updated for a given image. – Dorcas

Answer 1

Here is my test case, which demonstrates the lack of support in CursorJoiner for descending ordered cursors. This, however, is documented specifically in the CursorJoiner source code, so I'm not trying to critize but merely show how this can be circumvented (or hacked).

The test case shows how the assumption of ascending ordering makes the need for "flipping", or reversing, all of the choices made by the CursorJoiner (comparator result, incrementation of cursors, etc). What I'd really like to try next is to modify the CursorJoiner class directly, to try to add support for DESC ordering.

Please note that it seems the part about ordering by ID*(-1) maybe is not strictly necessary for this to work. In the following example, I did not negate the ID columns (plain DESC ordering, not "pseudo-ASC" with negative sequences), and it still works.

Test case

String[] colA = new String[] { "_id", "data", "B_id" };
String[] colB = new String[] { "_id", "data" };

MatrixCursor cursorA = new MatrixCursor(colA);
MatrixCursor cursorB = new MatrixCursor(colB);

// add 4 items to cursor A, linked to cursor B
// the data is ordered DESCENDING
// all cases, LEFT/RIGHT/BOTH, are included
cursorA.addRow(new Object[] { 5, "Item A", 1004 });  // BOTH
cursorA.addRow(new Object[] { 4, "Item B", 1003 });  // LEFT
cursorA.addRow(new Object[] { 3, "Item C", 1002 });  // BOTH
cursorA.addRow(new Object[] { 2, "Item D", 1001 });  // LEFT
cursorA.addRow(new Object[] { 1, "Item E", 1000 });  // BOTH
cursorA.addRow(new Object[] { 0, "Item F", 500 });  // LEFT

// similarily for cursorB (DESC)
cursorB.addRow(new Object[] { 1004, "X" });   // BOTH
cursorB.addRow(new Object[] { 1002, "Y" });   // BOTH
cursorB.addRow(new Object[] { 999,  "Z" });    // RIGHT
cursorB.addRow(new Object[] { 998,  "S" });    // RIGHT
cursorB.addRow(new Object[] { 900,  "A" });    // RIGHT
cursorB.addRow(new Object[] { 1000, "G" });   // BOTH

// join these on ID
CursorJoiner cjoiner = new CursorJoiner(
        cursorA, new String[] { "B_id" },   // left = A
        cursorB, new String[] { "_id" }     // right = B
);

// enable workaround
boolean desc = true;

int count = 0;
for (CursorJoiner.Result joinerResult : cjoiner) {
    Log.v("TEST", "Processing (left)=" + (cursorA.isAfterLast() ? "<empty>" : cursorA.getLong(2))
                + " / (right)=" + (cursorB.isAfterLast() ? "<empty>" : cursorB.getLong(0)));

     // flip the CursorJoiner.Result (unless Result.BOTH, or either cursor is exhausted)
    if (desc && joinerResult != CursorJoiner.Result.BOTH
             && !cursorB.isAfterLast() && !cursorA.isAfterLast())
        joinerResult = (joinerResult == CursorJoiner.Result.LEFT ? CursorJoiner.Result.RIGHT : CursorJoiner.Result.LEFT);

    switch (joinerResult) {
        case LEFT:
            // handle case where a row in cursorA is unique
            Log.v("TEST", count + ") join LEFT. cursorA is unique");

            if (desc) {
                // compensate cursor increments
                if (!cursorB.isAfterLast()) cursorB.moveToPrevious();
                if (!cursorA.isLast()) cursorA.moveToNext();
            }
            break;

        case RIGHT:
            Log.v("TEST", count + ") join RIGHT. cursorB is unique");
            // handle case where a row in cursorB is unique

            if (desc) {
                if (!cursorB.isLast()) cursorB.moveToNext();
                if (!cursorA.isAfterLast()) cursorA.moveToPrevious();
            }
            break;

        case BOTH:
            Log.v("TEST", count + ") join BOTH: " + cursorA.getInt(0) + "," + cursorA.getString(1) + "," + cursorA.getInt(2) + "/" + cursorB.getInt(0) + "," + cursorB.getString(1));
            // handle case where a row with the same key is in both cursors
            break;

    }

    count++;
}
Log.v("TEST", "Join done!");

and the output:

V/TEST: Processing (left)=5 / (right)=1004
V/TEST: 0) join BOTH: 4,Item A,1004/1004,X
V/TEST: Processing (left)=4 / (right)=1002
V/TEST: 1) join LEFT. cursorA is unique
V/TEST: Processing (left)=3 / (right)=1002
V/TEST: 2) join BOTH: 2,Item C,1002/1002,Y
V/TEST: Processing (left)=2 / (right)=999
V/TEST: 3) join RIGHT. cursorB is unique
V/TEST: Processing (left)=2 / (right)=998
V/TEST: 4) join RIGHT. cursorB is unique
V/TEST: Processing (left)=2 / (right)=900
V/TEST: 5) join RIGHT. cursorB is unique
V/TEST: Processing (left)=2 / (right)=1000
V/TEST: 6) join LEFT. cursorA is unique
V/TEST: Processing (left)=1 / (right)=1000
V/TEST: 7) join BOTH: 0,Item D,1000/1000,F
V/TEST: Processing (left)=0 / (right)=---
V/TEST: 8) join LEFT. cursorA is unique
V/TEST: Join done!

Dorcas

Answer 2

OK, so it seems I finally figured all this out. Thought I'd share this here, for anyone else who might be interested.

What am I trying to achieve?

• Query thumbnails and images on device (via MediaStore)
• Join these into one cursor, ordered descending (newest images on top)
• Handle the case of missing thumbnails

After lots of trial and error, and playing around with the MediaStore, I've learned that the thumbnails table (MediaStore.Images.Thumbnails) can not be expected to be up-to-date, at any given time. There will be images missing thumbnails, and vice versa (orphaned thumbnails). Especially when the camera app takes a new photo, apparently it doesn't immediately create a thumbnail. Not until the Gallery app (or equivalent) is opened, is the thumbnail table updated.

I got various helpful suggestions on how to work myself around this problem, mainly centered on just querying the images table (MediaStore.Images.Media) and then, somehow, extend the cursor with thumbnails one row at a time. While that did work, it caused the app to be extremely slow and consumed a lot of memory for ~2000 images on my device.

It really should be possible to simply JOIN (left outer join) the thumbnails table with the images table, such that we get all images and the thumbnails when these exist. Otherwise, we leave the thumbnail DATA column to null, and just generate those particular missing thumbnails ourselves. What would be really cool is to actually insert those thumbnails into the MediaStore, but that I have not looked into yet.

The main problem with all this was using the CursorJoiner. For some reason, it requires both cursors to be ordered in ascending order, let's say on ID. However, that means oldest images first, which really makes for a crappy gallery app. I found that the CursorJoiner can be "fooled", however, into permitting descending order by simply ordering by ID*(-1):

Cursor c_thumbs = getContext().getContentResolver().query(
                    MediaStore.Images.Thumnails.EXTERNAL_CONTENT_URI,
                    null, null, null, 
                    "(" + MediaStore.Images.Thumnails.IMAGE_ID + "*(-1))");

Cursor c_images= getContext().getContentResolver().query(
                    MediaStore.Images.Media.EXTERNAL_CONTENT_URI,
                    null, null, null, 
                    "(" + MediaStore.Images.Media._ID + "*(-1))");

As long as the rows match up, though, this works fine (the BOTH case). But when you run into rows where either cursor is unique (the LEFT or RIGHT cases) the reversed ordering messes up the inner workings of the CursorJoiner class. However, a simple compensation on the left and right cursors is sufficient to "re-align" the join, getting it back on track. Note the moveToNext() and moveToPrevious() calls.

// join these and return
// the join is on images._ID = thumbnails.IMAGE_ID
CursorJoiner joiner = new CursorJoiner(
        c_thumbs, new String[] { MediaStore.Images.Thumnails.IMAGE_ID },  // left = thumbnails
        c_images, new String[] { MediaStore.Images.Media._ID }   // right = images
);

String[] projection = new String{"thumb_path", "ID", "title", "desc", "datetaken", "filename", "image_path"};

MatrixCursor retCursor = new MatrixCursor(projection);

try {
    for (CursorJoiner.Result joinerResult : joiner) {

        switch (joinerResult) {
            case LEFT:
                // handle case where a row in cursorA is unique
                // images is unique (missing thumbnail)

                // we want to show ALL images, even (new) ones without thumbnail!
                // data = null will cause a temporary thumbnail to be generated in PhotoAdapter.bindView()

                retCursor.addRow(new Object[]{
                        null, // data
                        c_images.getLong(1), // image id
                        c_images.getString(2), // title
                        c_images.getString(3),  // desc
                        c_images.getLong(4),  // date
                        c_images.getString(5),  // filename
                        c_images.getString(6)
                });

                // compensate for CursorJoiner expecting cursors ordered ascending...
                c_images.moveToNext();
                c_thumbs.moveToPrevious();
                break;

            case RIGHT:
                // handle case where a row in cursorB is unique
                // thumbs is unique (missing image)

                // compensate for CursorJoiner expecting cursors ordered ascending...
                c_thumbs.moveToNext();
                c_images.moveToPrevious();
                break;

            case BOTH:

                // handle case where a row with the same key is in both cursors
                retCursor.addRow(new Object[]{
                        c_thumbs.getString(1), // data
                        c_images.getLong(1), // image id
                        c_images.getString(2), // title
                        c_images.getString(3),  // desc
                        c_images.getLong(4),  // date
                        c_images.getString(5),  // filename
                        c_images.getString(6)
                });

                break;
        }
    }
} catch (Exception e) {
    Log.e("myapp", "JOIN FAILED: " + e);
}

c_thumbs.close();
c_images.close();

return retCursor;

Then, in the "PhotoAdapter" class, which creates elements for my GridView and binds data into these from the cursor returned from the ContentProvider (retCursor above), I create a thumbnail in the following manner (when the thumb_path field is null):

String thumbData = cursor.getString(0);  // thumb_path
if (thumbData != null) {
    Bitmap thumbBitmap;
    try {
        thumbBitmap = BitmapFactory.decodeFile(thumbData);
        viewHolder.iconView.setImageBitmap(thumbBitmap);
    } catch (Exception e) {
        Log.e("myapp", "PhotoAdapter.bindView() can't find thumbnail (file) on disk (thumbdata = " + thumbData + ")");
        return;
    }

} else {

    String imgPath = cursor.getString(6);   // image_path
    String imgId = cursor.getString(1);  // ID 
    Log.v("myapp", "PhotoAdapter.bindView() thumb path for image ID " + imgId + " is null. Trying to generate, with path = " + imgPath);

    try {
        Bitmap thumbBitmap = ThumbnailUtils.extractThumbnail(BitmapFactory.decodeFile(imgPath), 512, 384);
        viewHolder.iconView.setImageBitmap(thumbBitmap);
    }  catch (Exception e) {
        Log.e("myapp", "PhotoAdapter.bindView() can't generate thumbnail for image path: " + imgPath);
        return;
    }
}

Dorcas

Answer 3

The accepted answer got me started on this question, but it contain a couple of small errors.

case LEFT:
            // handle case where a row in cursorA is unique
            // images is unique (missing thumbnail)
case RIGHT:
            // handle case where a row in cursorB is unique
            // thumbs is unique (missing image)

These are backwards. The documentation contradicts itself, and is likely where the mistake got made. From the source code of CursorJoiner:

case LEFT:
        // handle case where a row in cursorA is unique

Then in the enum for Result from the source code:

public enum Result {
    /** The row currently pointed to by the left cursor is unique */
    RIGHT,
    /** The row currently pointed to by the right cursor is unique */
    LEFT,
    /** The rows pointed to by both cursors are the same */
    BOTH
}

So I am guessing this is why you were force incrementing the cursors.

 //compensate for CursorJoiner expecting cursors ordered ascending...
                c_images.moveToNext();
                c_thumbs.moveToPrevious();

The iterator in CursorJoiner automatically increments the cursors for you.

This should be the working code (This code will also merge internal storage and external storage into a single cursor):

        Cursor[] thumbs = new Cursor[2];
        thumbs[0] = mActivity.getContentResolver().query(
                MediaStore.Images.Thumbnails.EXTERNAL_CONTENT_URI,
                new String[]{
                        MediaStore.Images.Thumbnails._ID ,
                        MediaStore.Images.Thumbnails.IMAGE_ID,
                        MediaStore.Images.Thumbnails.DATA
                },
                null,
                null,
                MediaStore.Images.Thumbnails.IMAGE_ID + "*(-1)"
        );
        thumbs[1] = mActivity.getContentResolver().query(
                MediaStore.Images.Thumbnails.INTERNAL_CONTENT_URI,
                new String[]{
                        MediaStore.Images.Thumbnails._ID ,
                        MediaStore.Images.Thumbnails.IMAGE_ID,
                        MediaStore.Images.Thumbnails.DATA
                },
                null,
                null,
                MediaStore.Images.Thumbnails.IMAGE_ID + "*(-1)"
        );
        Cursor thumbCursor = new MergeCursor(thumbs);
        Cursor[] cursors = new Cursor[2];
        cursors[0] = mActivity.getContentResolver().query(
                MediaStore.Images.Media.EXTERNAL_CONTENT_URI,
                new String[]{
                        MediaStore.Images.Media._ID,
                        MediaStore.Images.Media.DATA,
                        MediaStore.Images.Media.ORIENTATION,
                        MediaStore.Images.Media.BUCKET_DISPLAY_NAME,
                        MediaStore.Images.Media.BUCKET_ID,
                        MediaStore.Images.Media.MIME_TYPE
                },
                null,
                null,
                MediaStore.Images.Media._ID + "*(-1)"
        );
        cursors[1] = mActivity.getContentResolver().query(
                MediaStore.Images.Media.INTERNAL_CONTENT_URI,
                new String[]{
                        MediaStore.Images.Media._ID,
                        MediaStore.Images.Media.DATA,
                        MediaStore.Images.Media.ORIENTATION,
                        MediaStore.Images.Media.BUCKET_DISPLAY_NAME,
                        MediaStore.Images.Media.BUCKET_ID,
                        MediaStore.Images.Media.MIME_TYPE
                },
                null,
                null,
                MediaStore.Images.Media._ID + "*(-1)"
        );
        Cursor photoCursor = new MergeCursor(cursors);
        CursorJoiner cursorJoiner = new CursorJoiner(
                thumbCursor,
                new String[]{
                        MediaStore.Images.Thumbnails.IMAGE_ID
                },
                photoCursor,
                new String[]{
                        MediaStore.Images.Media._ID,
                }
        );
        Cursor finalCursor= new MatrixCursor(new String[]{
                MediaStore.Images.Media._ID,
                MediaStore.Images.Media.DATA,
                MediaStore.Images.Media.ORIENTATION,
                MediaStore.Images.Media.BUCKET_DISPLAY_NAME,
                MediaStore.Images.Media.BUCKET_ID,
                MediaStore.Images.Media.MIME_TYPE,
                "thumb_data"
        });
        for (CursorJoiner.Result joinerResult : cursorJoiner) {
            switch (joinerResult) {
                case RIGHT:
                    finalCursor.addRow(new Object[]{
                            photoCursor.getLong(photoCursor.getColumnIndex(MediaStore.Images.Media._ID)),
                            photoCursor.getString(photoCursor.getColumnIndex(MediaStore.Images.Media.DATA)),
                            photoCursor.getLong(photoCursor.getColumnIndex(MediaStore.Images.Media.ORIENTATION)),
                            photoCursor.getString(photoCursor.getColumnIndex(MediaStore.Images.Media.BUCKET_DISPLAY_NAME)),
                            photoCursor.getLong(photoCursor.getColumnIndex(MediaStore.Images.Media.BUCKET_ID)),
                            photoCursor.getString(photoCursor.getColumnIndex(MediaStore.Images.Media.MIME_TYPE)),
                            null
                    });
                    break;
                case BOTH:
                    finalCursor.addRow(new Object[]{
                            photoCursor.getLong(photoCursor.getColumnIndex(MediaStore.Images.Media._ID)),
                            photoCursor.getString(photoCursor.getColumnIndex(MediaStore.Images.Media.DATA)),
                            photoCursor.getLong(photoCursor.getColumnIndex(MediaStore.Images.Media.ORIENTATION)),
                            photoCursor.getString(photoCursor.getColumnIndex(MediaStore.Images.Media.BUCKET_DISPLAY_NAME)),
                            photoCursor.getLong(photoCursor.getColumnIndex(MediaStore.Images.Media.BUCKET_ID)),
                            photoCursor.getString(photoCursor.getColumnIndex(MediaStore.Images.Media.MIME_TYPE)),
                            thumbCursor.getString(thumbCursor.getColumnIndex(MediaStore.Images.Thumbnails.DATA)),
                    });
                    break;
            }

        }
        photoCursor.close();
        thumbCursor.close();

Spinner

I notice the typo in the documentation (mixed up left and right), but that's not what's causing the problems here. If the cursors are sorted in ascending order, this all works as described. However, when "fooling" the joiner to handle descending order, everything gets turned around. I'll post a new answer below, with some more details (since these comment fields are so tiny). – Dorcas