Writing many files on Android 11
Asked Answered
O

3

11

Scenario

I am trying to create many files, as a feature for the user. For example, I might write an app which creates one file for each song they listened to for the past few weeks (e.g. a text file with lyrics). I can't make the user pick the directory and filename for each file I generate, it would take them hours. The user should have access to these documents from other apps.

Solutions (not really)

In Android 11, it seems like Storage Access Framework is not going to be useful. I noticed there were 2 initially interesting options:

  1. Create a new file (which creates launches activity the user interacts with to save one file only), described here
  2. Grant access to a directory's contents (which allows read access but no way to write any documents), described here.
  3. Note: A solution already exists if you target Android 10 and request WRITE_EXTERNAL_STORAGE. Unfortunately this permission is denied on Android 11. Therefore, I cannot use the solution posted here.
  4. Get access to the directory using Storage Access Framework (specifically ACTION_OPEN_DOCUMENT_TREE) to get access to a directory, but I couldn't write files into this directory. (I get FileNotFoundException, as this is not a normal path, but a tree path.)

Verdict

I guess in my case, I need to go down the route of "manage all files on a storage device", as described here which involves launching an extra activity with ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION. I'd rather not ask permission to read the entire drive or make the user go into the settings app to give permission. Writing files shouldn't require any permission...

How would you handle saving many (non-media) files for the user?

Oakes answered 6/11, 2020 at 19:13 Comment(9)
use database to store your data and then allow user to "export" its content to separate file when needed?Orsini
does the user really need the files? Does he interact with those files? What is the reason for creating those files?Calculation
@MarcinOrlowski: what if I wanted to export the database into 10 files with different information? The user has to use SAF 10 times? I think this is unreasonable.Oakes
@LenaBru: There are many use cases for saving "many" files, for example, a NLP app might want to create a text file for each picture, describing its contents. A music app might want to write separate lyrics. It is also useful for debugging certain types of apps, where we want to see that a file was saved in a good format. On Android 11, you can't see into app specific directories (internal or external), even using adb shell or Android Studio File Explorer. These files/ documents can be used by the user as files, shared, organised in folder, opened with other apps. Don't forget about files.Oakes
it seems like Storage Access Framework is not going to be useful. Well it is. You only have to let the user choose one directory once and you are done for the life time of your app.Schreib
Granted, my challenge is not a challenge faced by the apps that are most popular today, which hide files from the user. Having said that, Google have introduced the Files app, so I would say files are still pretty important.Oakes
Get access to the directory using Storage Access Framework (specifically ACTION_OPEN_DOCUMENT_TREE) to get access to a directory, but I couldn't write files into this directory. Then you did something wrong as after that you can create folders and files in it for the lifetime of your app.Schreib
You can use ACTION_OPEN_DOCUMENT_TREE for writing multiple files! See this answerInvolutional
Read this: https://mcmap.net/q/89292/-android-11-scoped-storage-permissionsUnbar
O
17

There are 2 methods, but first, I'll define the file and directory name, which I will later save inside the external (sdcard) Downloads folder

val outputFilename = "my_file"
val outputDirectory = "my_sub_directory" // The folder within the Downloads folder, because we use `DIRECTORY_DOWNLOADS`

The working, but deprecated method:

Although the Environment.getExternalStoragePublicDirectory was deprecated, it still works on apps targeting and running on Android 11.

file = File(Environment.getExternalStoragePublicDirectory(
    Environment.DIRECTORY_DOCUMENTS),
    "$outputDirectory/$outputFilename"
)
val outputStream = FileOutputStream(file)

The MediaStore way:

Alternatively, you could use MediaStore's Files collection, suggested Gaurav Mall here. I didn't know about the MediaStore files collection...

I rewrote it in kotlin and modified it for writing files here:

val resolver = context.contentResolver
val values = ContentValues()
// save to a folder
values.put(MediaStore.MediaColumns.DISPLAY_NAME, outputFilename)
values.put(MediaStore.MediaColumns.MIME_TYPE, "application/my-custom-type")
values.put(MediaStore.MediaColumns.RELATIVE_PATH, Environment.DIRECTORY_DOWNLOADS + "/" + outputDirectory)
val uri = resolver.insert(MediaStore.Files.getContentUri("external"), values)
// You can use this outputStream to write whatever file you want:
val outputStream = resolver.openOutputStream(uri!!)
// Or alternatively call other methods of ContentResolver, e.g. .openFile

In both cases below, you don't need WRITE_EXTERNAL_STORAGE for Android 29 and above.

Oakes answered 7/11, 2020 at 23:38 Comment(2)
After digesting this problem, I think another good example (but for images) is here.Oakes
The MediaStore solution is great, because it's easy to use, not confusing and actually works. Confirmed on Android 10+. Thank you.Hodges
S
1

In answer to your question, a properly implemented StorageAccessFramework will work on all Android versions >= 5.0.

I created a class that fully implements my needs and allows both writing files and reading them. You can specify a standard directory (SAF will open it, but the user can choose any other directory). You can specify useOnlyProvidedDirectoryPath as true if you only need to retrieve data from the specified directory (this can also be useful when working with multiple directories).

SAF allows you to get information about the provided path. You can retrieve and write information to multiple directories at once if needed. On modern OS versions, you will need to grant access for each required directory, but this is easy enough to do.

So, add the following class to the project:

import android.app.Activity
import android.content.Intent
import android.content.UriPermission
import android.net.Uri
import android.os.Build
import android.os.storage.StorageManager
import android.util.Log
import androidx.activity.ComponentActivity
import androidx.activity.result.contract.ActivityResultContracts
import androidx.documentfile.provider.DocumentFile
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.io.FileOutputStream
import java.io.IOException

/**
 * Storage Access Framework directory worker.
 *
 * @param activity Used to create registerForActivityResult and get Context.
 *
 * @param directoryPath Sets the start directory (Environment.DIRECTORY_* can be used). For
 *   example, you can use "Music/MyApp". Make sure you have this directory on device to test it,
 *   otherwise start directory will be undefined.
 *
 * @param useOnlyProvidedDirectoryPath Use if you want to use directoryPath only. Don't use if you
 *   have provided directoryPath purely for user experience and there is no need to write data to
 *   this directory. If it's false and directoryPath isn't found, the last user accepted directory
 *   will be used. It's **`false`** by default.
 *
 * @param shouldBeReadable Set to true if you need read access. It's **`true`** by default.
 *
 * @param shouldBeWriteable Set to true if you need write access. It's **`true`** by default.
 */
class StorageAccessFrameworkDirectory(
    private val activity: ComponentActivity,
    private val directoryPath: String,
    private val useOnlyProvidedDirectoryPath: Boolean = false,
    private val shouldBeReadable: Boolean = true,
    private val shouldBeWriteable: Boolean = true,
) {
    private val showDirectoryPickerActivity = with(activity) {
        registerForActivityResult(ActivityResultContracts.OpenDocumentTree()) { uri ->
            // if user accepted request, and we don't need to useOnlyProvidedDirectoryPath
            // or need to use it and user chosen it - take permission for future use
            if (uri != null && (
                    !useOnlyProvidedDirectoryPath ||
                        isPersistedUriEqualsWithDirectoryPath(uri, directoryPath)
                    )
            ) {
                var flags = 0
                if (shouldBeReadable) {
                    flags = flags or Intent.FLAG_GRANT_READ_URI_PERMISSION
                }
                if (shouldBeWriteable) {
                    flags = flags or Intent.FLAG_GRANT_WRITE_URI_PERMISSION
                }

                contentResolver.takePersistableUriPermission(uri, flags)
            }
        }
    }

    /**
     * Shows is usable directory available or not.
     */
    fun isUsable(): Boolean {
        return activity.getUsableDirectory() != null
    }

    /**
     * Shows system directory picker. Use it if isUsable() is false.
     */
    fun showDirectoryPicker() {
        val initialUri = activity.getFilePickerInitialUri(directoryPath)
        showDirectoryPickerActivity.launch(initialUri)
    }

    /**
     * Writes data from ByteArray to file.
     */
    suspend fun writeFile(
        fileName: String,
        mimeType: String,
        data: ByteArray
    ) = withContext(Dispatchers.IO) {
        val uri = activity.getUsableDirectory()
        if (!shouldBeWriteable || uri == null) {
            return@withContext
        }

        try {
            DocumentFile.fromTreeUri(activity.applicationContext, uri)
                ?.createFile(mimeType, fileName)
                ?.let { activity.applicationContext.contentResolver.openOutputStream(it.uri) }
                ?.buffered()
                ?.use { it.write(data) }
        } catch (e: IOException) {
            Log.e("SAF writeFile", e.toString())
        }
    }

    /**
     * Allows to continuously write data to file from ByteArrays, provided in continuousWork lambda.
     */
    suspend fun writeFileContinuously(
        fileName: String,
        mimeType: String,
        continuousWork: suspend ((ByteArray) -> Unit) -> Unit
    ) = withContext(Dispatchers.IO) {
        val uri = activity.getUsableDirectory()
        if (!shouldBeWriteable || uri == null) {
            return@withContext
        }

        try {
            DocumentFile.fromTreeUri(activity.applicationContext, uri)
                ?.createFile(mimeType, fileName)
                ?.let { activity.applicationContext.contentResolver.openOutputStream(it.uri) }
                ?.buffered()
                ?.use { continuousWork { newBytes -> it.write(newBytes) } }
        } catch (e: IOException) {
            Log.e("SAF writeFileContinuous", e.toString())
        }
    }

    /**
     * Returns data from a file in the selected folder, or null if it can't be read.
     */
    suspend fun readFile(fileName: String): ByteArray? = withContext(Dispatchers.IO) {
        val uri = activity.getUsableDirectory()
        if (!shouldBeReadable || uri == null) {
            return@withContext null
        }

        try {
            // you can also find in directories with directoryDocumentFile.listFiles()
//          println("Usable directory contains:\n" +
//              DocumentFile.fromTreeUri(activity.applicationContext, uri)?.listFiles()
//                  ?.joinToString { "${it.name} (isFile: ${it.isFile}), isDirectory: ${it.isDirectory})" })
            with(DocumentFile.fromTreeUri(activity.applicationContext, uri)?.findFile(fileName)) {
                if (this == null || !this.canRead()) {
                    return@withContext null
                }

                println("found ${this.name}: ${this.uri}")

                return@withContext activity.applicationContext.contentResolver
                    .openInputStream(this.uri)
                    ?.buffered()
                    ?.use { it.readBytes() }
            }
        } catch (e: IOException) {
            Log.e("SAF readFile", e.toString())
            return@withContext null
        }
    }

    private fun Context.getUsableDirectory(): Uri? {
        val usablePersistedUris = getListOfPersistedUris().filter {
            val haveProblemsWithUri = (shouldBeReadable && !it.isReadPermission) ||
                (shouldBeWriteable && !it.isWritePermission)

            !haveProblemsWithUri
        }

        val predefinedDirectory = usablePersistedUris.find {
            isPersistedUriEqualsWithDirectoryPath(it.uri, directoryPath)
        }

        if (predefinedDirectory != null || useOnlyProvidedDirectoryPath) {
            return predefinedDirectory?.uri
        }

        return usablePersistedUris.lastOrNull()?.uri
    }

    companion object {
        private fun Context.getFilePickerInitialUri(directoryPath: String): Uri? {
            if (Build.VERSION.SDK_INT < Build.VERSION_CODES.O) {
                // you can't set directory programmatically on Android 7.1 or lower, so just return null
                return null
            }

            val rootPath = if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) {
                // you can't get the root path from the system on old Android, use a hardcoded string
                "content://com.android.externalstorage.documents/document/primary%3A"
            } else {
                // you can get the root path from the system on Android 10+, get it with intent
                val intent = (getSystemService(Activity.STORAGE_SERVICE) as StorageManager)
                    .primaryStorageVolume
                    .createOpenDocumentTreeIntent()

                val systemDefaultPath = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) {
                    intent.getParcelableExtra("android.provider.extra.INITIAL_URI", Uri::class.java)
                } else {
                    @Suppress("Deprecation")
                    intent.getParcelableExtra("android.provider.extra.INITIAL_URI")
                }

                systemDefaultPath.toString().replace("/root/", "/document/") + "%3A"
            }

            return Uri.parse(rootPath + directoryPath)
        }

        /**
         * Checks if directoryPath is in system list of persisted URIs and has read/write access.
         */
        fun Context.isDirectoryUsable(
            directoryPath: String,
            shouldBeReadable: Boolean,
            shouldBeWriteable: Boolean,
        ): Boolean = with(
            this.getListOfPersistedUris().find {
                isPersistedUriEqualsWithDirectoryPath(it.uri, directoryPath)
            },
        ) {
            val haveProblemsWithDirectory = this == null ||
                (shouldBeReadable && !this.isReadPermission) ||
                (shouldBeWriteable && !this.isWritePermission)

            !haveProblemsWithDirectory
        }

        private fun Context.getListOfPersistedUris(): List<UriPermission> =
            contentResolver.persistedUriPermissions

        private fun isPersistedUriEqualsWithDirectoryPath(
            uri: Uri,
            directoryPath: String
        ): Boolean {
            val formattedDirectoryPath = directoryPath.formattedPath()
            val uriPath = uri.toString()
                .replace("content://com.android.externalstorage.documents/tree/primary%3A", "")
                .replace("content://com.android.externalstorage.documents/tree/home%3A", "Documents/")
                .replace("%2F", "/")
                .formattedPath()

//          println("isPersistedUriContainsDirectoryPath:\n" +
//              "  uri: $uri\n" +
//              "  directoryPath: $directoryPath" +
//              "  uriPath: $uriPath\n" +
//              "  formattedDirectoryPath: $formattedDirectoryPath")
            return formattedDirectoryPath == uriPath
        }

        /**
         * Adds / to the beginning if missing, removes it from the end, and replaces with %2F. Examples:
         *
         * "" -> "%2F"
         *
         * "/" -> "%2F"
         *
         * "Documents" -> "%2FDocuments"
         *
         * "/Documents/" -> "%2FDocuments"
         */
        private fun String.formattedPath() = this.removeSuffix("/")
            .let { if (it.firstOrNull() != '/') "/$it" else it }.replace("/", "%2F")
    }
}

Next, you need to create a class object in your Activity (since the class saves the activity instance, don't save this object in ViewModel, singletons and other activities to avoid memory leaks). Here you can specify a default path and parameters to use the class read-only, write-only, or only on a specific path:

private val safDirectory = StorageAccessFrameworkDirectory(
    activity = this,
    directoryPath = Environment.DIRECTORY_DOCUMENTS,
)

Next, if safDirectory.isUsable() is false, you need to show system picker with safDirectory.showDirectoryPicker(), or if it's true, you can read or write in chosen directory. For example:

if (tableFileFormat == TableFileFormat.XLSX) {
    safDirectory.writeFile(
        "file_name_without_extension", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
        getExcelFile(), // ByteArray with full file data
    )
} else {
    safDirectory.writeFileContinuously(
        "file_name_without_extension", "text/comma-separated-values",
    ) { writer -> writeCsvFile(writer) } // continuous ByteArray writer function
}

If you need an example of a function that writes data sequentially, here it is:

suspend fun writeCsvFile(writer: suspend (ByteArray) -> Unit) {
    writer(
        listOf("№", "name")
            .joinToString(separator = ",", postfix = "\n").toByteArray(Charsets.UTF_8),
    )
    names.forEachIndexed { index, name ->
        writer(
            listOf(index + 1, name)
                .joinToString(separator = ",", postfix = "\n").toByteArray(Charsets.UTF_8),
        )
    }
}

You can extend this class to get a list of files in a folder, save a directory selected by the user (if you want to interact with multiple directories, and the user should be able to select them), and so on.

Soggy answered 12/9, 2023 at 8:7 Comment(0)
S
-2

On Android 11 you can create a subdirectory in public directory Documents using the standard File classes as you could in below 10. And also in 10 if you knew how. Then create your files in that subdirectory. No fuss.

You have write access to a lot of other public directories too.

You do not need all files access for this.

The files you create in this way are accessable by the user using the official Files app on the device.

Schreib answered 6/11, 2020 at 20:21 Comment(18)
Have you done this on an app which targets Android 11? getExternalStoragePublicDirectory was deprecated, they recommend using MediaStore, Intent#ACTION_OPEN_DOCUMENT or Context#getExternalFilesDir(String). None of these options are viable for me, as I explained in the question. Therefore, I don't have write access to a lot of other public directories too. (Sure, I still have write access to MediaStore, but I'm writing documents, and the docs have stated I need to use SAF for it)Oakes
For more explanation as to why your method won't work, read https://mcmap.net/q/202447/-how-to-access-downloads-folder-in-android and its commentsOakes
All what i said works targetting Android 11 api 30. Just try and you will see. That functions are deprecated does not mean they do not work.Schreib
Context#getExternalFilesDir(String). None of these options are viable for me, as I explained in the question. ??? There is nothing in your post about getExternalFilesDir nor that you would have explained anything about using it.Schreib
I am trying to many small files, as a feature for the user. The external files directory is still internal to the app, and only located on external storage. Therefore, it won't be useful as a feature for the user. (Not user accessible)Oakes
Of course it is accessable for the user. Whatever you mean with accessable. The user can use the Files app to see and get all files if that is what you are after.Schreib
have you actually got this working before on an app targeting Android 11 running on 11?Oakes
Also, about getExternalFilesDir, they are not accessible to the user. Please read developer.android.com/reference/android/content/… These files are internal to the applications, and not typically visible to the user as media. With Android 11, it cannot even be seen on through adb or file explorers. It seems to me like you don't know what you're talking about.Oakes
they are not accessible to the user. It is unclear to me which action the user is doing to see those files. And as said before: All files your app creates in its getExternalFilesDIr are visible to the user using the official FIles app on the Android 11 device.Schreib
No they're not. Clearly you don't know what you're talking about. Here is a screenshot, API 29 (10) on left, API 30 on right. External files directory (in the sdcard, located in Android/data), is showing empty on API 30. i.imgur.com/khceZZo.pngOakes
I have no device and all i tell is like found on Pixel 3 Xl emulator api 30.Schreib
Anyhow, if you create a folder in Documents and write files in them they are visible by Files app too. Please check. I just created an R emulator. There the ../Android/datafolder is not visible indeed. But files created in Documents are.Schreib
Yes, but to write these files in Documents (or anywhere outside MediaStore) on Android 11, it seems to me I need to use storage access framework per file, or use All Files Access. Both are not good options. However, i found https://mcmap.net/q/87307/-getexternalstoragepublicdirectory-deprecated-in-android-q which may allow me to contribute documents into the MediaStoreOakes
No. You need nothing of that all.Reread my answer! You just can write using the FIle class and such to the Documents, Pictures, Alarms an more public directories. This is the third or fourth time i tell you so. It is time you start coding.Schreib
And how do you get those paths? The methods to actually get those paths don't work on Android 11. Maybe you can share some codeOakes
Which methods ar you talking about? Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOCUMENTS) gives you a nice File object and full path. Deprecated ? Yes indeed. Does that matter? No.Schreib
It works... Thank you! getExternalStoragePublicDirectory is deprecated but it works. The comments above the method definition stated: the path returned from this method is no longer directly accessible to apps., but it works for writing new files at least. (didn't test for other use cases)Oakes
@Ben Butterworth, Please dont edit my answers. Use comments if you have anything to add.Schreib

© 2022 - 2024 — McMap. All rights reserved.