Why do some Android phones cause our app to throw an java.lang.UnsatisfiedLinkError?

Question

We're experiencing a java.lang.UnsatisfiedLinkError on some of the Android phones that are using our app in the market.

Problem description:

static
{
    System.loadLibrary("stlport_shared"); // C++ STL        
    System.loadLibrary("lib2"); 
    System.loadLibrary("lib3"); 
}

Crashes the app in on of the System.loadLibrary() lines with a java.lang.UnsatisfiedLinkError. java.lang.UnsatisfiedLinkError: Couldn't load stlport_shared from loader dalvik.system.PathClassLoader[dexPath=/data/app/app_id-2.apk,libraryPath=/data/app-lib/app_id-2]: findLibrary returned null

Solution approach

We started running some custom diagnostics on all our installs to check if every lib is unpacked in the /data/data/app_id/lib folder.

PackageManager m = context.getPackageManager();
String s = context.getPackageName();
PackageInfo p;
p = m.getPackageInfo(s, 0);
s = p.applicationInfo.dataDir;

File appDir = new File(s);
long freeSpace = appDir.getFreeSpace();

File[] appDirList = appDir.listFiles();
int numberOfLibFiles = 0;
boolean subFilesLarger0 = true;
for (int i = 0; i < appDirList.length; i++) {

    if(appDirList[i].getName().startsWith("lib")) {
        File[] subFile = appDirList[i].listFiles(FileFilters.FilterDirs);   
        numberOfLibFiles = subFile.length;
        for (int j = 0; j < subFile.length; j++) {
            if(subFile[j].length() <= 0) {
                subFilesLarger0 = false;
                break;
            }
        }
    }
}

On every test phone that we have numberOfLibFiles == 3 and subFilesLarger0 == true. We wanted to test if all libs are unpacked properly and are larger then 0 byte. In addition we're looking at freeSpace to see how much disk space is available. freeSpace matches the amount of memory that you can find in Settings --> Applications at the bottom of the screen. The thinking behind this approach was that when there is not enough space on the disk available that the installer might have problems unpacking the APK.

Real world scenario

Looking at the diagnostics, some of the devices out there do NOT have all 3 libs in the /data/data/app_id/lib folder but have plenty of free space. I'm wondering why the error message is looking for /data/app-lib/app_id-2. All our phones store their libs in /data/data/app_id/lib. Also the System.loadLibrary() should use a consistent path across installation and loading the libs? How can I know where the OS is looking for the libs?

Question

Anyone experiencing problems with installing native libs? What work arounds have been successful? Any experience with just downloading native libs over the internet when they are not existent and storing them manually? What could cause the problem in the first place?

EDIT

I now also have a user who runs into this problem after an application update. The previous version worked fine on his phone, an after an update the native libs seem missing. Copying the libs manually seem to cause trouble as well. He is on android 4.x with a non rooted phone without custom ROM.

EDIT 2 - Solution

After 2 years of spending time on this problem. We came up with a solution that works well for us now. We open sourced it: https://github.com/KeepSafe/ReLinker

Esthonia

Do you happen to know what devices it is not working on? Is it possible you are building arm only, and not for x86? – Phene

yes we know the devices. There are some Samsung Galaxy SII and LG phones. All the phones that are crashing are ARM phones. We don't build for x86. – Esthonia

Do you mean "stlport_shared" from the system image or is this something you are bundling directly with your app? The system-provided one is just "stlport", although you can also compile/link with stlport_static. Also, what OS versions are the failing devices (Gingerbread or something more recent)? – Carchemish

We ship stl_port with the app our self. The problem appears across all android versions. – Esthonia

You might try linking instead to gnustl_shared. I had several device specific issues with stlport. Note the GPL v3 has an exception for this library and allows you to link proprietary software. – Ireland

Related? code.google.com/p/android/issues/detail?id=35962 – Photovoltaic

This might be related as well: groups.google.com/forum/#!topic/android-ndk/X1mRZwBxZLc – Esthonia

Can you not use the sltport that ships with the device? It is possible that there are specific changes that need to be made some devices for stlport to work correctly.. – Phene

We decided to ship stlport our self as it was only added in NDK r9 and we found that not every handset manufacture implements this appropriately. – Esthonia

Just as a update. We haven't been able to solve that problem. We just have those crashes. – Esthonia

@Esthonia - see my answer below, suggesting manual unzipping of the original apk to extract the correct shared library. Works for me in all the tests I could perform, and just today uploaded a build with this solution to Google Play. Will see if I'll still get any UnsatisfiedLinkErrors for my native lib after this update. – Cistaceous

@Cistaceous see my edit. We solved the problem finally. – Esthonia

Answer 1

I have the same trouble, and the UnsatisfiedLinkErrors comes on all versions of Android - over the past 6 months, for an app that currently has over 90000 active installs, I had:

Android 4.2     36  57.1%
Android 4.1     11  17.5%
Android 4.3     8   12.7%
Android 2.3.x   6   9.5%
Android 4.4     1   1.6%
Android 4.0.x   1   1.6%

and the users report that it usually happens just after the app update. This is for an app that gets around 200 - 500 new users per day.

I think I came up with a simpler work-around. I can find out where is the original apk of my app with this simple call:

    String apkFileName = context.getApplicationInfo().sourceDir;

this returns something like "/data/app/com.example.pkgname-3.apk", the exact file name of my app's APK file. This file is a regular ZIP file and it is readable without root. Therefore, if I catch the java.lang.UnsatisfiedLinkError, I can extract and copy my native library, from the inside of .apk (zip) lib/armeabi-v7a folder (or whatever architecture I'm on), to any directory where I can read/write/execute, and load it with System.load(full_path).

Edit: It seems to work

Update July 1, 2014 since releasing a version of my product with the code similar to the listed below, on June 23, 2014, did not have any Unsatisfied Link Errors from my native library.

Here is the code I used:

public static void initNativeLib(Context context) {
    try {
        // Try loading our native lib, see if it works...
        System.loadLibrary("MyNativeLibName");
    } catch (UnsatisfiedLinkError er) {
        ApplicationInfo appInfo = context.getApplicationInfo();
        String libName = "libMyNativeLibName.so";
        String destPath = context.getFilesDir().toString();
        try {
            String soName = destPath + File.separator + libName;
            new File(soName).delete();
            UnzipUtil.extractFile(appInfo.sourceDir, "lib/" + Build.CPU_ABI + "/" + libName, destPath);
            System.load(soName);
        } catch (IOException e) {
            // extractFile to app files dir did not work. Not enough space? Try elsewhere...
            destPath = context.getExternalCacheDir().toString();
            // Note: location on external memory is not secure, everyone can read/write it...
            // However we extract from a "secure" place (our apk) and instantly load it,
            // on each start of the app, this should make it safer.
            String soName = destPath + File.separator + libName;
            new File(soName).delete(); // this copy could be old, or altered by an attack
            try {
                UnzipUtil.extractFile(appInfo.sourceDir, "lib/" + Build.CPU_ABI + "/" + libName, destPath);
                System.load(soName);
            } catch (IOException e2) {
                Log.e(TAG "Exception in InstallInfo.init(): " + e);
                e.printStackTrace();
            }
        }
    }
}

Unfortunately, if a bad app update leaves an old version of the native library, or a copy somehow damaged, which we loaded with System.loadLibrary("MyNativeLibName"), there is no way to unload it. Upon finding out about such remnant defunct library lingering in the standard app native lib folder, e.g. by calling one of our native methods and finding out it's not there (UnsatisfiedLinkError again), we could store a preference to avoid calling the standard System.loadLibrary() altogether and relying on our own extraction and loading code upon next app startups.

For completeness, here is UnzipUtil class, that I copied and modified from this CodeJava UnzipUtility article:

import java.io.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;

public class UnzipUtil {
    /**
     * Size of the buffer to read/write data
     */

    private static final int BUFFER_SIZE = 4096;
    /**
     * Extracts a zip file specified by the zipFilePath to a directory specified by
     * destDirectory (will be created if does not exists)
     * @param zipFilePath
     * @param destDirectory
     * @throws java.io.IOException
     */
    public static void unzip(String zipFilePath, String destDirectory) throws IOException {
        File destDir = new File(destDirectory);
        if (!destDir.exists()) {
            destDir.mkdir();
        }
        ZipInputStream zipIn = new ZipInputStream(new FileInputStream(zipFilePath));
        ZipEntry entry = zipIn.getNextEntry();
        // iterates over entries in the zip file
        while (entry != null) {
            String filePath = destDirectory + File.separator + entry.getName();
            if (!entry.isDirectory()) {
                // if the entry is a file, extracts it
                extractFile(zipIn, filePath);
            } else {
                // if the entry is a directory, make the directory
                File dir = new File(filePath);
                dir.mkdir();
            }
            zipIn.closeEntry();
            entry = zipIn.getNextEntry();
        }
        zipIn.close();
    }

    /**
     * Extracts a file from a zip to specified destination directory.
     * The path of the file inside the zip is discarded, the file is
     * copied directly to the destDirectory.
     * @param zipFilePath - path and file name of a zip file
     * @param inZipFilePath - path and file name inside the zip
     * @param destDirectory - directory to which the file from zip should be extracted, the path part is discarded.
     * @throws java.io.IOException
     */
    public static void extractFile(String zipFilePath, String inZipFilePath, String destDirectory) throws IOException  {
        ZipInputStream zipIn = new ZipInputStream(new FileInputStream(zipFilePath));
        ZipEntry entry = zipIn.getNextEntry();
        // iterates over entries in the zip file
        while (entry != null) {
            if (!entry.isDirectory() && inZipFilePath.equals(entry.getName())) {
                String filePath = entry.getName();
                int separatorIndex = filePath.lastIndexOf(File.separator);
                if (separatorIndex > -1)
                    filePath = filePath.substring(separatorIndex + 1, filePath.length());
                filePath = destDirectory + File.separator + filePath;
                extractFile(zipIn, filePath);
                break;
            }
            zipIn.closeEntry();
            entry = zipIn.getNextEntry();
        }
        zipIn.close();
    }

    /**
     * Extracts a zip entry (file entry)
     * @param zipIn
     * @param filePath
     * @throws java.io.IOException
     */
    private static void extractFile(ZipInputStream zipIn, String filePath) throws IOException {
        BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(filePath));
        byte[] bytesIn = new byte[BUFFER_SIZE];
        int read = 0;
        while ((read = zipIn.read(bytesIn)) != -1) {
            bos.write(bytesIn, 0, read);
        }
        bos.close();
    }
}

Greg

Cistaceous

thanks, I saw that, just haven't had the chance to try it. Looks like a great idea. Are you on twitter or so? I'm sure there are more problems that are worth exchanging. – Esthonia

@Esthonia - Twitter is not my favorite way of communicating, I work best through traditional email, see "Contacts" page on my web site, hyperionics.com. Also, I released my app version using the above method, for 3 days now no unsatisfied link errors, but will see in a longer run. – Cistaceous

where do you make call for initNativeLib in the app lifecylce? – Unilateral

This really does not matter where, provided it is before you actually try to use any of the native calls. Preferably it should not be on UI thread, if the long operation of unzipping and moving the native lib kicks in. I actually call it when my service starts. – Cistaceous

Answer 2

EDIT: Since I got another crash report yesterday for one of my apps I dug a bit deeper into the matter and found a third very likely explanation for that problem:

Google Play Partial APK Update Goes Wrong

To be honest, I did not know about that feature. The APK file name suffix "-2.apk" made me suspicious. It is mentioned in the crash message of this question here and I could also find that suffix in the crash report of that customer of mine.

I believe the "-2.apk" hints at a partial update that probably delivers a smaller delta to Android devices. That delta apparently does not contain native libraries when they did not change since the previous version.

For whatever reason the System.loadLibrary function tries to look up the native library from the partial update (where it doesn't exist). It's both a flaw in Android and in Google Play.

Here is a very relevant bug report with an interesting discussion about similar observations: https://code.google.com/p/android/issues/detail?id=35962

It looks like Jelly Bean may be flawed in terms of native library installation and loading (both crash reports that came in were Jelly Bean devices).

If this is really the case I suspect that some forced change in the NDK library code may fix the problem, like changing some unused dummy variable for each release. However, that should be done in a way that the compiler preserves that variable without optimizing it away.

EDIT (12/19/13): The idea of changing the native library code for each build unfortunately does not work. I tried it with one of my apps and got an "unsatisfied link error" crash report from a customer who updated anyway.

Incomplete APK installation

That's unfortunately just out of my memory and I cannot find a link anymore. Last year I read a blog article about that unsatisfied link issue. The author said that this is a bug in the APK installation routine.

When copying native libraries to their target directory fails for whatever reason (device ran out of storage space, maybe also messed up directory write permissions...) the installer still returns "success", as if native libraries were just "optional extensions" to an app.

In this case the only workaround would be reinstalling the APK while making sure there's enough storage space for the app.

However, I cannot find any Android bug ticket nor the original blog article anymore and I searched for it for quite a bit. So this explanation may be in the realms of myths and legends.

"armeabi-v7a" directory takes precedence over "armeabi" directory

This bug ticket discussion hints at a "concurrency issue" with installed native libraries, see Android bug ticket #9089.

If there is an "armeabi-v7a" directory present with just a single native library, the whole directory for that architecture takes precedence over the "armeabi" directory.

If you try to load a library that is just present in "armeabi" you'll get an UnsatisfiedLinkException. That bug has been flagged as "works as intended" by the way.

Possible workaround

In either case: I found an interesting answer to a similar question here on SO. It all boils down to packaging all native libraries as raw resources to your APK and copy on first app start the correct ones for the current processor architecture to the (app private) file system. Use System.load with the full file paths to load these libraries.

However this workaround has a flaw: since the native libraries will reside as resources in the APK Google Play won't be able to find them and create device filters for the mandatory processor architectures anymore. It could be worked around by putting "dummy" native libraries into the lib folder for all target architectures.

Overall I do believe this issue should be properly communicated to Google. It seems as if both Jelly Bean and Google Play are flawed.

It usually helps to tell the customer with that problem to reinstall the app. This is unfortunately not a good solution if app data loss is a concern.

Emmetropia

Great info in here. We're building against arm and armv7a. We have all libs for both architectures. The unpacking problems seems to make sense for the problem we're having. It's across all android versions. I know some people who ship their libs in the asset folder and unpack it to app/files and load it from there with System.load() We have their stuff embedded and it never failed. I'll give that a try. – Esthonia

@Esthonia I got another such crash report from a user and I think it's very suspicious that the APK file is suffixed with "-2.apk". It happened after an update. Could it be just a temporary glitch? Did your users report that your app works again on second start? Right now I'm trying to figure out another workaround, like clearing the dex cache. Maybe that will also reinstall native libs on boot. – Emmetropia

I think the -2.apk is just the temporary apk while it downloads, and it basically does a swap with the other apk.. I am not certain (would need to dig into the source code to verify). – Phene

@Phene I believe these incremental updates are just a Google Play feature / bug and you won't be able to find anything related in the Android source code. But if you find something anyway please tell me :-) – Emmetropia

@NobuGames - I don't believe that -1.apk, -2.apk etc. suffixes indicate that something went wrong with the install. While testing and installing debug versions of my app, I see that -1 or -2 are added to the apk filename routinely, often change with each install, and there is nothing wrong with the app functioning. There must be something else, most probably the native lib does not install due to out of storage space on the relevant partition. See my reply for a simpler work-around I found. – Cistaceous

Answer 3

Android started packaging their libraries in /data/app-lib// sometime around Ice Cream Sandwich or Jellybean, I can't remember which.

Are you building and distributing both for "arm" and "armv7a"? My best guess is that you only built for one of the architectures and you're testing on the other.

Reckford

Do you have more information around the change of library storage location (hasAllNativeLibs) – Esthonia

Answer 4

In case you just release with app bundles, This issue might be related to https://issuetracker.google.com/issues/127691101

It happens on some devices of LG or old Samsung devices where the user has moved the app to the SD Card.

One way of fixing the issue is to use Relinker library to load your native libraries instead of directly calling System.load method. It did work for my app's use case.

https://github.com/KeepSafe/ReLinker

Another way is to block the movement of the app to the SD card.

You can also keep android.bundle.enableUncompressedNativeLibs=false in your gradle.properties file. But it will increase the app download size on Play Store as well as its disk size.

Ramekin

Answer 5

After having this issue myself and doing a bit of research (aka googling), I was able to fix the problem by targeting a lower SDK.

I had compileSdkVersion set to 20 (which worked on 4.4)

and also

minSdkVersion 20 targetSdkVersion 20

Once I changed them to 18 (I was in a position to do that without any bigger implications) I was able to run the app without problems on Lollipop.

Hope it helps - It drove me nuts for days.

Trioecious

interesting. We had our lowest API lvel 9 for a long time, and now it's 15. The problem still is our biggest problem. We've recently been working on some lower level solution. Will update once confirmed it works – Esthonia

Lowering the targetSdkVersion is not a solution. – Sedimentary