Sharing via Seekable Pipe or Stream with Another Android App?

Question

Lots of Intent actions, like ACTION_VIEW, take a Uri pointing to the content the action should be performed upon. If the content is backed by a file -- whether the Uri points directly to the file, or to a ContentProvider serving the file (see FileProvider) -- this generally works.

There are scenarios in which developers do not want to have the content reside in a file for sharing with other apps. One common scenario is for encryption: the decrypted data should reside in RAM, not on disk, to minimize the risk of somebody getting at that decrypted data.

My classic solution to sharing from RAM is to use ParcelFileDescriptor and createPipe(). However, when the activity responding to ACTION_VIEW (or whatever) gets an InputStream on that pipe, the resulting stream is limited compared to the streams you get when the ContentProvider is serving up content from a file. For example, this sample app works fine with Adobe Reader and crashes QuickOffice.

Based on past related questions, my assumption is that createPipe() is truly creating a pipe, and that pipes are non-seekable. Clients that attempt to "rewind" or "fast forward" run into problems as a result.

I am seeking a reliable solution for sharing in-memory content with a third-party app that gets around this limitation. Specifically:

• It has to use a Uri syntax that is likely to be honored by client apps (i.e., ACTION_VIEW implementers); solutions that involve something obtuse that client apps are unlikely to recognize (e.g., pass such-and-so via an Intent extra) do not qualify

• The data to be shared cannot be written to a file as part of the sharing (of course, the client app could wind up saving the received bytes to disk, but let's ignore that risk for the moment)

• Ideally it does not involve the app looking to share the data opening up a ServerSocket or otherwise exacerbating security risks

Possible suggested ideas include:

• Some way to reconfigure createPipe() that results in a seekable pipe

• Some way to use a socket-based FileDescriptor that results in a seekable pipe

• Some kind of RAM disk or something else that feels like a file to the rest of Android but is not persistent

A key critierion, if you will, of a working solution is if I can get a PDF served from RAM that QuickOffice can read.

Any suggestions?

Thanks!

Kayser

Use DocumentsProvider which deals in ParcelFileDescriptors? Could probably be backported. The Google Vault example does something like what you are asking: android.googlesource.com/platform/development/+/master/samples/… – Burkholder

BTW client caching the data should be perfectly OK, you are already giving them access anyways. You cannot create RAM disks without root access (or intermediary service). If you use native code you could try memory streams (created with fmemopen) which have a file descriptor. You should be able to pass it around with Parcels. – Burkholder

Apparently fmemopen is not included in bionic but can be implemented with some work. It does have open_memstream() though so it could be done the other way around (QuickOffice passing FD to app, app writes data to it, returns, QuickOffice closes FD, reads memory), but of course you'll need to modify QuickOffice. – Burkholder

If you create a local web server which can serve the unencrypted PDF to applications like so:- code.google.com/p/free-android-apps/wiki/Project_LocalHTTPD but I'm not sure how you can control which applications get access this way. – Dawson

@grahaminn: One problem with the Web server approach is that a lot of apps don't support the http scheme themselves, but instead expect a browser (or whatever) to have downloaded the file for them. Also, there's the issue of securing that server against arbitrary access (see my warning against ServerSocket). – Kayser

If you have no control over the applications which access it (as I expect is the case with these 3rd party apps), then I can't see a way for you to reliably require authentication from them. – Dawson

@NikolayElenkov: Sorry for the delay in getting back to you. I haven't played with the new document provider framework yet, so I don't know if they somehow solve this problem under the covers. It feels like they are doing what I do, which results in a non-seekable stream, but maybe they have some wrapper that addresses the problem. I will move that up my priority list to experiment with. Thanks! – Kayser

Indeed, without the cooperation of the 3rd party apps in some custom verification scheme going beyond the Android APIs, it's not clear how you would hope to tell if you are handing the "protected" content off to a well know viewer with no save functionality, or an imposter with the same package name created and installed specifically for the purpose of exporting everything it is handed. But if you do have their cooperation, you can come up with a custom data sharing scheme which avoid these limitations. – Burr

MemoryFile could have been a potential solution along the lines of the RAM disk possibility. However, it has no public API for acquiring file descriptors, and it didn't seem to fix the issue when I tried it anyway (with reflection). The logcat only reported a remote exception without any description. It's method documentation warns against treating it's file descriptor as normal up to Gingerbread, but the ContactsProvider used this in order to return an AssetFileDescriptor for photo thumbnails queried from the database. – Childbirth

@ChrisStratton: Installing and choosing secure and known applications to view sensitive data should be the responsibility of the user and not the applications providing the data. – Childbirth

If security is the users concern, there is little issue here. The reality is that people who want schemes like this generally want them to restrict what the user can do with their device. – Burr

@ChrisStratton: In this case, that's not my objective. Persisting decrypted data opens a window in time when that data can be accessed sans encryption, until such time as the data can be securely erased. Secure erasure is problematic with flash memory, so avoiding persistence is important. It's entirely possible that this concern covers DRM-style scenarios -- I haven't given that much thought. I am concerned about data that the user wants to keep encrypted and trying to cover some bases to minimize what all has access to the decrypted material. – Kayser

That's what full disk encryption is for. Of course one should always remember that all these measures are basically nuisance level, as there are too many attack vectors, even for the original form of the content. – Burr

@ChrisStratton: Due to Google's approach to using the same PIN/password for full-disk encryption (FDE) as the lockscreen, FDE tends to be used with weak passwords. There's a Linux boot CD that can brute-force Android FDE fairly well, though I think Google improved matters in Android 4.4. Moreover, FDE is only relevant if the device is powered off. – Kayser

Of course - all these approaches are nuisance level as I already said. If your content is actually interesting an attacker will extract the key for the original encryption from your apk, or root the device. – Burr

@ChrisStratton: Only fools put an encryption key in an APK. Rooting a device is necessary but not sufficient to decrypt encrypted content on internal storage. – Kayser

Answer 1

You've posed a really difficult combination of requirements.

Lets look at your ideas for solutions:

Possible suggested ideas include:

• Some way to reconfigure createPipe() that results in a seekable pipe

• Some way to use a socket-based FileDescriptor that results in a seekable pipe

• Some kind of RAM disk or something else that feels like a file to the rest of Android but is not persistent

The first one won't work. This issue is that the pipe primitive implemented by the OS is fundamentally non-seekable. The reason is supporting seek that would require the OS to buffer the entire pipe "contents" ... until the reading end closes. That is unimplementable ... unless you place a limit on the amount of data that can be sent through the pipe.

The second one won't work either, for pretty much the same reason. OS-level sockets are not seekable.

At one level, the final idea (a RAM file system) works, modulo that such a capability is supported by the Android OS. (A Ramfs file is seekable, after all.) However, a file stream is not a pipe. In particular the behaviour with respect to the end-of-file is different for a file stream and a pipe. And getting a file stream to look like a pipe stream from the perspective of the reader would entail some special code on that side. (The problem is similar to the problem of running tail -f on a log file ...)

---

Unfortunately, I don't think there's any other way to get a file descriptor that behaves like a pipe with respect to end-of-file and is also seekable ... short of radically modifying the operating system.

If you could change the application that is reading from the stream, you could work around this. This is precluded by the fact that the fd needs to be read and seeked by QuickOffice which (I assume) you can't modify. (But if you could change the application, there are ways to make this work ...)

---

By the way, I think you'd have the some problems with these requirements on Linux or Windows. And they are not Java specific.

---

UPDATE

There have been various interesting comments on this, and I want to address some here:

1. The OP has explained the use-case that is motivating his question. Basically, he wants a scheme where the data passing through the "channel" between the applications is not going to be vulnerable in the event that the users device is stolen (or confiscated) while the applications are actually running.

   Is that achievable?

   • In theory, no. If one postulates a high degree of technical sophistication (and techniques that the public may not know about ...) then the "bad guys" could break into the OS and read the data from shared memory while the "channel" remained active.

   • I doubt that such attacks are (currently) possible in practice.

   • However, even if we assume that the "channel" writes nothing to "disc" there could still be traces of the channel in memory: e.g.

     • a still mounted RAMfs or still active shared memory segments, or

     • remnants of previous RAMfs / shared memory.

     In theory, this data could in theory be retrieved, provided that the "bad guy" doesn't turn of or reboot the device.

2. It has been suggested that ashmem could be used in this context:

   • The issue of there being no public Java APIs could be addressed (by writing 3rd-party APIs, for example)

   • The real stumbling block is the need for a stream API. According the "ashmem" docs, they have a file-like API. But I think that just means that they conform to the "file descriptor" model. These FDs can be passed from one application to another (across fork / exec), and you use "ioctl" to operate on them. But there is no indication that they implement "read" and "write" ... let alone "seek".

   • Now, you could probably implement a read/write/seekable stream on top of ashmem, using native and Java libraries on both ends of the channel. But both applications would need to be "aware" of this process, probably to the level of providing command line options to set up the channel.

   These issues also apply to old-style shmem ... except that the channel setup is probably more difficult.

3. The other potential option is to use a RAM fs.

   • This is easier to implement. The files in the RAMfs will behave like "normal" files; when opened by an application you get a file descriptor that can be read, written and seeked ... depending on how it was opened. And (I think) you should be able to pass a seekable FD for a RAMfs file across a fork/exec.

   • The problem is that the RAMfs needs to be "mounted" by the operating system in order to use it. While it is mounted, another (privileged) application can also open and read files. And the OS won't let you unmount the RAMfs while some application has open fds for RAMfs files.

   • There is a (hypothetical) scheme that partly mitigates the above.

     1. The source application creates and mounts a "private" RAMfs.
     2. The source application creates/opens the file for read/write and then unlinks it.
     3. The source application writes the file using the fd from the open.
     4. The source application forks / execs the sink application, passing the fd.
     5. The sink application reads from the (I think) still seekable fd, seeking as required.
     6. When the source application notices that the (child) sink application process has exited, it unmounts and destroys the RAMfs.

     This would not require modifying the reading (sink) application.

     However, a third (privileged) application could still potentially get into the RAMfs, locate the unlinked file in memory, and read it.

However, having re-reviewed all of the above, the most practical solution is still to modify the reading (sink) application to read the entire input stream into a byte[], then open a ByteArrayInputStream on the buffered data. The core application can seek and reset it at will.

Motherly

"However, a file stream is not a pipe" -- the RAM disk is a means to the end of "don't store the decrypted bits on disk". I don't need a pipe; I need a way of sharing in-RAM content with other apps. I mention "pipe" in the title because titles are meant to be short. :-) "If you could change the application that is reading from the stream, you could work around this" -- a probable follow-on from this will be to attempt to document what is and is not possible with a non-seekable pipe, to try to guide developers in a path to improve compatibility here. Thanks! – Kayser

"The RAM disk is a means to the end of "don't store the decrypted bits on disk". Not so. If the RAM pages holding the RAM file system get paged out, they will end up on the persistent storage device. Typically you don't have any control over that ... – Motherly

"A probable follow-on from this will be to attempt to document what is and is not possible with a non-seekable pipe ..." - What is not possible is seeking / rewinding, or doing anything that requires these. In fact the (application-side) workaround to seeking a non-seekable stream / pipe / socket is to read and buffer the entire stream in memory, then "seek" against the buffered copy. Of course, that means that you have enough memory ... which is likely to be a scaling issue. – Motherly

" ... to try to guide developers in a path to improve compatibility here." - I'm not sure if anything is possible to improve compatibility in the general case. I think that a requirement for a seekable input stream is reasonable if the application characteristics warrant it. My take is that your requirements are highly unusual, and requiring an application to cope with that kind of thing is a bit much. – Motherly

"If the RAM pages holding the RAM file system get paged out" -- except that Android does not page. "What is not possible is seeking / rewinding, or doing anything that requires these" -- I was referring to things at a higher level (e.g., which parsers require seeking). – Kayser

"your requirements are highly unusual" -- if you think that videos, music files, PDFs, and the like are unusual document formats, you are welcome to do so. Most multimedia players, and a decent percentage of PDF readers, appear to require seekable streams. Wanting such content to remain secure at rest (i.e., encrypted when stored) is becoming increasingly important, and I have had a lot of people ask me over the past couple of years how to accomplish this. Having failed in all my attempts, silly me thought to ask on SO. – Kayser

1) The unusualness is in your usecase is in the insistence that the data should not be stored on persistent storage. 2) The other problem with a RAM file system is that other applications can read it while it is existence ... especially privileged ones. 3) At any rate the reasons that it won't (ultimately) work in a secure fashion are due to the OS designers perception that this use-case is too unusual (and too difficult) to support. Convince them. – Motherly

The Android Linux kernel has the ashmem driver, which allows two processes to share memory maps securely. Unfortunately, there doesn't seem to be a method for using it in the public Java API. Also, it didn't seem to work for sharing with QuickOffice either, as per my comment on the question. – Childbirth

@Childbirth - You are correct. The key stumbling block with a shared memory-based solution is that you can't create file descriptors. That means you have to do a significant amount of the low level work in the applications themselves. (And that's particularly awkward for Java ...) – Motherly

I should also point out that this question seems to be motivated by use-cases where you are trying to keep transiently decrypted data secret from the owner of "the machine". It is well understood that this is not possible in practice. Even if you stop the decrypted data ever being stored on disc, a kernel hacker could mod his kernel to capture the data read / written using read() or write() syscalls. Or a Java hacker could pull out the information using a Java debugger. – Motherly

@StephenC: Actually, ashmem works by sharing file descriptors, and will work with standard file I/O operations. And to reply to your second comment, the question actually seems to be about safeguarding the decrypted data from people other than the owner of the device. Storing it in the filesystem provides a permanent and easy way for e.g. a thief to access it; storing it in RAM avoids this pitfall. – Childbirth

@StephenC: "I should also point out that this question seems to be motivated by use-cases where you are trying to keep transiently decrypted data secret from the owner of "the machine"." -- no, it is motivated by keeping the transiently-decrypted data secret from those who presently hold the device. My personal focus is where those who presently hold the device are not the device's owner, but rather have taken the device, perhaps by force. – Kayser

Answer 2

It's not a general solution to your problem, but opening a PDF in QuickOffice works for me with the following code (based on your sample):

@Override
public AssetFileDescriptor openAssetFile(Uri uri, String mode) throws FileNotFoundException {
    try {
        byte[] data = getData(uri);
        long size = data.length;
        ParcelFileDescriptor[] pipe = ParcelFileDescriptor.createPipe();
        new TransferThread(new ByteArrayInputStream(data), new AutoCloseOutputStream(pipe[1])).start();
        return new AssetFileDescriptor(pipe[0], 0, size);
    } catch (IOException e) {
        e.printStackTrace();
    }
    return null;
};

private byte[] getData(Uri uri) throws IOException {
    AssetManager assets = getContext().getResources().getAssets();
    InputStream is = assets.open(uri.getLastPathSegment());
    ByteArrayOutputStream os = new ByteArrayOutputStream();
    copy(is, os);
    return os.toByteArray();
}

private void copy(InputStream in, OutputStream out) throws IOException {
    byte[] buf = new byte[1024];
    int len;
    while ((len = in.read(buf)) > 0) {
        out.write(buf, 0, len);
    }
    in.close();
    out.flush();
    out.close();
}

@Override
public Cursor query(Uri url, String[] projection, String selection, String[] selectionArgs, String sort) {
    if (projection == null) {
        projection = new String[] { OpenableColumns.DISPLAY_NAME, OpenableColumns.SIZE };
    }

    String[] cols = new String[projection.length];
    Object[] values = new Object[projection.length];
    int i = 0;
    for (String col : projection) {
        if (OpenableColumns.DISPLAY_NAME.equals(col)) {
            cols[i] = OpenableColumns.DISPLAY_NAME;
            values[i++] = url.getLastPathSegment();
        }
        else if (OpenableColumns.SIZE.equals(col)) {
            cols[i] = OpenableColumns.SIZE;
            values[i++] = AssetFileDescriptor.UNKNOWN_LENGTH;
        }
    }

    cols = copyOf(cols, i);
    values = copyOf(values, i);

    final MatrixCursor cursor = new MatrixCursor(cols, 1);
    cursor.addRow(values);
    return cursor;
}

private String[] copyOf(String[] original, int newLength) {
    final String[] result = new String[newLength];
    System.arraycopy(original, 0, result, 0, newLength);
    return result;
}

private Object[] copyOf(Object[] original, int newLength) {
    final Object[] result = new Object[newLength];
    System.arraycopy(original, 0, result, 0, newLength);
    return result;
}

Cutup

Hmmmm... interesting. I'll have to experiment with this more soonish. Many thanks! – Kayser

I tried copying your code into the relevant sample, and QuickOffice 6.3.1.041 still fails on launch on a Nexus 4 running Android 4.4 (whereas Adobe Reader, PDF To Go, and Repligo Reader all work). I can definitely see how what you are proposing would improve compatibility, and I may cut over my examples to use this approach, even though I cannot reproduce your success with QuickOffice. Thanks again! – Kayser

Thank you for your feedback. I modified my answer and added a query() method as in link, now it works for me also on a Nexus 7 with Android 4.4 and QuickOffice 6.3.1.041 – Cutup

That definitely helps. I will make sure that I ripple this into my samples (and book) in the coming weeks. Thanks again! – Kayser

That's ugly example, copying entire file to memory, then serving it. You're asking for out-of-memory error. But you could simply change it to stream input directly through pipe, without loading to memory. – Nous

Answer 3

I believe you're looking for StorageManager.openProxyFileDescriptor, function added in API 26. This will give you ParcelFileDescriptor, needed for your ContentProvider.openAssetFile to work. But you can also grab its file descriptor and use it in file I/O: new FileInputStream(fd.getFileDescriptor())

In function description is :

This can be useful when you want to provide quick access to a large file that isn't backed by a real file on disk, such as a file on a network share, cloud storage service, etc. As an example, you could respond to a ContentResolver#openFileDescriptor(android.net.Uri, String) request by returning a ParcelFileDescriptor created with this method, and then stream the content on-demand as requested. Another useful example might be where you have an encrypted file that you're willing to decrypt on-demand, but where you want to avoid persisting the cleartext version.

It works with ProxyFileDescriptorCallback, which is your function to provide I/O, mainly read pieces of your file from various offsets (or decrypt it, read from network, generate, etc).

As I tested, it's well suited also for video playback over content:// scheme, because seeking is efficient, no seek-by-read as is the option for pipe-based approach, but Android really asks relevant fragments of your file.

Internally Android uses some fuse driver to transfer the data between processes.

Nous

Thanks! I tested it back in the O Developer Preview timeframe, filed a bunch of issues, and the feedback that I got was... unhelpful. That was a few years ago, so perhaps things have improved since then. And I haven't gone looking to see if anyone published updated samples. If you got it working, though, that's great to hear! – Kayser

Answer 4

I've been experimenting with @josias code. I found some of the query(...) calls came with a projection of _data. Including the data for that column and setting the actual length means more file types can be opened in more apps. Always including _data even when not in the passed in projection allows opening even more file types.

Here is what I ended up with:

private static final String[] PROJECTION = {OpenableColumns.DISPLAY_NAME, OpenableColumns.SIZE, "_data"};

@Override
public Cursor query(Uri url, String[] projection, String selection, String[] selectionArgs, String sort) {

    byte[] data = getData(mSourcePath, url);

    final MatrixCursor cursor = new MatrixCursor(PROJECTION, 1);

    cursor.newRow()
        .add(url.getLastPathSegment())
        .add(data.length)
        .add(data);

    return cursor;
}

Bovine