Android 11 - System.loadLibrary for native C++ library takes 60+ seconds, works perfectly fast on Android 10 and below

Question

In our game application for Android which is based on the game engine cocos2d-x, with most of the code being written in C++, we have a very strange and critical issue since Android 11:

When the native library gets loaded in onLoadNativeLibraries it now suddenly takes 60+ seconds. Before Android 11, it all worked fine and it loaded in 0.2-3 seconds. Now when you start the game, you have a 60+ seconds gray screen.

We already figured out that JNI_OnLoad gets called directly after the 60 second stall is over.

Here's the code of the onLoadNativeLibraries function:

protected void onLoadNativeLibraries()
{
    try
    {
        ApplicationInfo ai = getPackageManager().getApplicationInfo(getPackageName(), PackageManager.GET_META_DATA);
        Bundle bundle = ai.metaData;
        String libName = bundle.getString("android.app.lib_name");
        System.loadLibrary(libName); // line of 60 seconds stall
    }
    catch (Exception e)
    {
        e.printStackTrace();
    }
}

We already tried time profiling, but without any success. It just hows that it's spending a lot of time on that function. Also pausing via the debugging doesn't lead to any further clues. The native debugger doesn't show anything on the C++ side of the code.

Does anybody have any idea why this is happening or what we could try to figure it out? Any help would be highly appreciated :)

Disciplinant

Do you explictly set android:extractNativeLibs in your manifest? Which Android Gradle Plugin version do you build with? Is the load time the same in release mode as in debug mode? – Respirable

@Respirable No, we don't do that - should we set it to something? We use gradle-5.6.4. It's also very long in release mode, yes. – Disciplinant

I meant the Android Gradle Plugin. The one you specify as a dependency in your project-level build.gradle file. It ought to have a version starting with 3 or 4. – Respirable

Also, does this always happen? Or only the first time after installing a new version of the app? – Respirable

@Respirable Sorry, sure makes sense. There we have classpath 'com.android.tools.build:gradle:3.6.1' Yes, it happes always. 100% of the time. – Disciplinant

Answer 1

Short Answer:

It's a bug in Android 11 fixed by google but not deployed yet.

Meanwhile, if you don't care about calling static and thread_local variables destructors in your lib when program is exiting/library is unloaded, pass the flag -fno-c++-static-destructors to the compiler. (see long answer for a more granular solution using clang annotations)

I used this flag on my project (not cocos2d) with no issue and lib is loading even faster than before.

Long Answer:

Unfortunately this is a performance regression introduced in android 11 (R) by the Google team. The issue is being tracked by google here.

To summarize, when System.loadLibrary() is called, the system registers a destructor for each of the C++ global variables contained in the loaded library, using __cxa_atexit()

Since Android 11 (R), the implementation of this function in android has changed:

• In Q, __cxa_atexit uses a linked list of chunks, and calls mprotect twice on the single chunk to be modified.
• In R, __cxa_atexit calls mprotect twice on a single contiguous array of handlers. Each array entry is 2 pointers.

This change regressed the performance drastically when they are many C++ global variables which seems to be the case in cocos2d so libraries.

Google has already implemented a fix https://android-review.googlesource.com/c/platform/bionic/+/1464716 but as stated in the issue:

this won't be in Android 11 until the March QPR at the earliest, and since this isn't a security issue it won't be mandatory for OEMs to actually take that patch.

Google Team suggests also some workarounds at the app level by removing or skipping the destructors on global variables:

• For a particular global variable, the [[clang::no_destroy]] attribute skips the destructor call.
• Pass -fno-c++-static-destructors to the compiler to skip the destructors for all static variables. This flag also skips destructors for thread_local variables. If there are thread_local variables with important destructors, those can be annotated with [[clang::always_destroy]] to override the compiler flag.
• Pass -Wexit-time-destructors to the compiler to make it warn on every instance of an exit-time destructor, to highlight where the __cxa_atexit registrations are coming from.

Westerly

-fno-c++-static-destructors seems to work perfectly. Thanks a lot. We already tried to debug this with all kinds of tricks for 4 days straight to no avail. Thank you :) – Disciplinant