This answer brings together hard-to-find but important information from many excellent websites and stackoverflow answers/comments on this under-documented NDK subject:
Avoiding Crashes: NDK Version, minSdKVersion
, targetSdkVersion
, and APP_PLATFORM
(in project.properties)
The short version
To prevent your native Android app from mysteriously crashing on older customer devices, the short answer is to make your NDK APP_PLATFORM
(in project.properties
or Application.mk
) the same as your minSdkVersion
(in AndroidManifest.xml
).
But depending on what NDK features you use, doing so may severely limit the set of customers who can download your app.
To find out why that is and whether you have other options, read on...
Four different concepts
NDK Version (e.g. r10e, r13b): this is the version of the Android NDK release (the tar/zip file) you download from Google. Here are all the NDK versions
minSdkVersion
(e.g. 15,19,21) is a setting you make in your AndroidManifest.xml in a <uses-sdk>
element under the <manifest>
element. This setting affects both NDK (native) and Java development.
targetSdkVersion
(e.g. 15,19,21) is a different setting you make in your AndroidManifest.xml in a <uses-sdk>
element under the <manifest>
element. This setting affects only Java development.
APP_PLATFORM
(e.g. 15,19,21) is a setting you make in your Native NDK project. Typically this setting is found in the project.properties
file at the root of your project as the last number at the end of the target=
line, e.g. target=Google Inc.:Google APIs:21
for level 21 (and usually the way that "21" got there is by being the -t/--target
option to a command-line invocation of the android create project
or android update project
command). You can also make this setting by putting APP_PLATFORM := android-21
into your Application.mk
file.
Three of these concepts use API Level Numbers
minSdkVersion
, targetSdkVersion
and APP_PLATFORM
use Android's consistent numbering scheme for API levels:
Click here to see Google's API Level Chart
As you can see, the levels roughly correspond to Android releases, but do not even remotely match the number of the Android release (that would be too easy). For example, API level 10 corresponds to Android OS 2.3.3 and 2.3.4.
The cute code-names like "Lollipop" and "Nougat" are at a coarser granularity than API versions. For example, several API versions (21 and 22) are "Lollipop"
These same API version numbers are encoded by the Java Build.VERSION_CODES
type:
Click here to see Build.VERSION_CODES
For example, level 22 is LOLLIPOP_MR1
Not every number exists for each concept. For example, you might want to support API level 10, but there is no APP_PLATFORM
10 so you would go back to the last available APP_PLATFORM
, APP_PLATFORM
9.
Click here to see distribution of API versions in the installed base in the wild
What is minSdkVersion
?
This setting is by far the easiest to understand. When you set minSdkVersion
to, say, 21 (corresponding to Android OS 5.0), that means that the Google Play Store will only advertise your app as supporting Android OS 5.0 and beyond, and Android will prevent users from installing your app if their Android device has lower than Android OS 5.0 installed.
So minSdkVersion
is the lowest OS you support. Nice and simple.
This setting obviously has implications both for Java and C code: you want to set it to the lowest OS version that both parts of your code can support.
Both targetSdkVersion
and APP_PLATFORM
must be greater than or equal to minSdkVersion
.
What is targetSdkVersion
?
This setting only has effect in the Android Java world (it has no effect on your C NDK code). Its fulfills a purpose in the Android Java world that is similar to APP_PLATFORM
in the Android C NDK world.
Sometimes you want your app to support older devices, but also use new features only available on newer Java API versions.
For example, Android added a nifty Java VoiceInteractor
API that's only supported in API 23 (Android 6.0) or later. You might want to support VoiceInteractor
if your customers have a new device, but still have your app run on older devices.
By setting targetSdkVersion
to 23, you are making a simple contract with Android:
- Android agrees to give your app code access to API-23-only Java features like
VoiceInteractor
- In return, you agree to check, at runtime in your Java code, whether that feature is available on the customer's device before calling it (otherwise your app will crash).
This contract works because in Java, it is "ok" if your code has references to classes/methods that might not exist on the customer's device, as long as you don't call them.
The contract applies to all Android Java features added after your minSdkVersion
up to and including your targetSdkVersion
.
In addition to giving you access to certain new Java APIs, the targetSdkVersion
setting also enables or disables certain well-documented compatibility behaviors:
Click here to see which behavior changes come with each targetSdkVersion
These well-documented changes also form a kind of contract. For example, around Android 4, Google migrated their Android device designs away from having a dedicated menu button and towards having an on-screen Action bar. If your app has a targetSdkVersion
lower than 14 (Android 4.0.1), Google would put a software menu button up on the screen to make sure your app kept working even if the device didn't have a dedicated menu button. But by choosing a targetSdkVersion
greater than or equal to 14 at build time, you are promising Google that you either don't have a menu or you use an Action bar, so Google no longer puts up the software menu button.
What is APP_PLATFORM
?
APP_PLATFORM
performs a similar function in the C NDK world that targetSdkVersion
performs in the Java world.
But, sadly, due to a combination of limitations in the C language and bad behavior by Google, APP_PLATFORM
is significantly more dangerous and, frankly, nearly unusable.
Let's start from the beginning...
APP_PLATFORM
is an NDK-only setting that tells the build-ndk
tool which subdirectory of your NDK to look in for certain key include files and libraries which collectively are referred to as an NDK "platform." Each NDK distribution (each NDK tar/zip that we developers download from Google) contains multiple platforms.
For example, if you set APP_PLATFORM
to android-21
, build-ndk
will look in:
$(ndk_directory)/platforms/android-21/arch-$(architecture)/usr/include
$(ndk_directory)/platforms/android-21/arch-$(architecture)/usr/lib
for include files and libraries.
If you installed your NDK by simply downloading a zip/tar from Google's NDK Downloads Website, then $(ndk_directory)
is simply the directory where you extracted the file.
If you installed your NDK by first downloading the Android (Java) SDK and then running the Android SDK Manager to install the "NDK" item, then $(ndk_directory) is $(sdk_directory)/ndk-bundle
, where $(sdk_directory)
is wherever your SDK is installed.
$(architecture)
is arm
, arm64
, x86
, etc.
What is in a "platform"?
The $(ndk_directory)/platforms/android-XX
directory contains two super-important things:
- all your C library calls like
fopen()
, atof()
, sprintf()
etc. The C library on Android is called "bionic."
- Android-specific NDK calls/types/defines like
AInputQueue
and EGLContext
What changes at different APP_PLATFORM
levels?
In each android-XX
version, Google adds more calls to the NDK. For example,
APP_PLATFORM
API level 9 added the very useful NativeActivity
APP_PLATFORM
API level 18 added OpenGL ES 3.0
Some APP_PLATFORM
versions also add calls to the C library, and/or "fix" things that are missing (for example, the token PTHREAD_KEYS_MAX
got added in APP_PLATFORM
21).
Click here to read Google's incomplete documentation on what changed in each APP_PLATFORM
level
So far this is similar to the Java world. Nobody expects Google or any other OS vendor to make every new feature available on old devices, especially when those features rely on hardware only found on newer devices (e.g. faster processors, new camera features, new audio features).
But Google's NDK team did a naughty thing that the Java team did not.
In some APP_PLATFORM
versions, Google made gratuitous, breaking API changes that cannot possibly be excused by any legitimate argument such as those above.
These are the types of breaking API changes that Android Java developers would never accept. For example, Google has
- renamed C library functions and
- changed C library functions from being inlined to being not inlined
The most serious case of this was APP_PLATFORM
21, where Google made many breaking changes that generated an extremely high number of stackoverflow issues (many examples here and more below).
But there also have been changes in previous APP_PLATFORM
s (e.g. signal()
in API 19).
And there are even some breaking changes in APP_PLATFORM
s after 21 such as APP_PLATFORM
24 (e.g. std::vector::resize
as Karu mentions in a comment of this question).
So this is clearly a bad Google habit that is here to stay.
Why do these changes make my app crash on old devices?
To see why these naughty changes are a problem, remember that the C library on Android is a shared library, meaning that the implementation of non-inline, non-macro calls like sprintf()
is not compiled into your program but rather present in the C library on your test devices and on each customer device.
So it doesn't just matter what API version you have in your development environment. It also matters what API version of C library is on each device where your app might run.
Suppose your app calls atof()
and you build your app with APP_PLATFORM
21 and test it on your modern test devices that run Android 5 or later (API version 21 or later). Everything looks ok.
Then you release your app and suddenly find tons of customers with Android OS versions 4.4 and earlier (API versions less than 21) report your app crashing on their devices.
What's going on?
In APP_PLATFORM
21 (Android 5), atof()
is a regular (not inline
, not macro) function. So the native part of your app (the myapp.so
file that ndk-build
will create, and that you load from your Java code using System.loadLibrary("myapp")
) will be marked as having a dependency on an external function called atof() in the C library.
When you run your app on a given device, Android will open your myapp.so
, see the dependency on atof()
, and find atof()
in the C library on that device.
But the shock surprise is that in APP_PLATFORM
s earlier than 21, atof()
was an inline
function in the platform header files, meaning that:
- its implementation (its definition, its code, its body) was in the header file and got compiled into your app when you built your app
- there is no
atof()
implementation in the C library on any customer device with API < 21 (any customer device running Android < 5). There never needed to be, since atof()
was inline back in those days.
So when you run your app on devices running API version < 21 (Android OS < 5), the Java call System.loadLibrary("myapp")
fails because the run-time loader cannot find all the symbols needed by your myapp.so
. Android knows your myapp.so
needs atof()
but cannot find atof()
in the C library on the device. Crash.
This atof()
example is only one of many, undocumented or barely-documented breaking changes that Google brazenly refers to as WorkingAsIntended. Besides
atof()
, you can find huge numbers of other stackoverflow items with the same cause (e.g. with mkfifo()
and, unbelievably, even rand()
)
How can I fix it?
In the atof()
example above, you might say to yourself "okay, if there is no atof()
on older devices, I'll provide one in my app and ship a new app version."
And in fact that would work.
But you'll have a sinking feeling in your stomach when you realize there is no answer to a much more important question:
How can I know what changed, and what old devices will be affected?
Here's the real kicker. You can't.
Unlike the Android Java API, where Google carefully maintains backwards compatibility with old APIs, clearly documenting any behavior changes that are keyed to the targetSdkVersion
parameters, there is no such documentation for
Android NDK APP_PLATFORM
levels.
Like the Java API, you can look up an NDK call and find out what is the earliest API version (the earliest customer Android OS) where that call is supported.
But unlike the Java API with targetSdkVersion
, when you change your NDK APP_PLATFORM
level, you will not be able to find any Google documentation that tells you:
- what API changes (possibly even C library API changes) exist that might break your app on older devices. For example, a list of functions like
atof()
, mkfifo()
and rand()
for which you would need to provide your own implementation for older devices
- what effect not providing those re-implemented routines will have on the lowest Android OS that you can now support with your app
Simply put, Google won't tell you the earliest Android version that each APP_PLATFORM
supports.
If you happen to have a lot of old devices lying around and a lot of time, you could try your app on every possible old Android version and see what crashes with missing C library symbols, providing custom implementations for functions that are not found. Of course that's only the first level of testing: in reality Google could have made breaking changes where the symbol is still there (so no crash), but the call behaves differently. This would never be accepted at the Java level but for some reason Google feels entitled to do it with the NDK.
Of course, nobody has time to do this, nor should developers have to.
So effectively what that means is that this is the official Google NDK policy:
Every time you increase the APP_PLATFORM
of your project, you get access to new APIs, but you also get some breaking changes that will cause your app to crash on some older devices. Oh, and we're not going to give you a specific list of those changes. Nor are we going to tell you the earliest Android OS version on which your app is still guaranteed to work.
And effectively what that means is:
Every time you increase the APP_PLATFORM
of your project, you have to set minSdkVersion
equal to APP_PLATFORM
, preventing your app from running on older devices. Otherwise your app may crash on some older devices.
It is hard to overstate how tragic this is.
Google is effectively telling you "in order to use new NDK features, you must abandon all your customers with old devices and abandon future sales to customers with older devices."
To make this tragedy concrete with a real-world example, notice that Google added support for OpenGL ES 3.1 in API Level 21 (Android OS 5.0). Suppose you wanted to support new OpenGL ES 3.1 features on new devices, but still support OpenGL ES 3.0 (API level 18 (Android OS 4.3)) and OpenGL ES 2.0 (API level 5 (Android OS 2.0)) on older devices. This is a very likely scenario since (unlike the transition from OpenGL ES 1 to 2) the changes in OpenGL ES 2 to 3 are quite minor and cumulative.
In order to support ES 3.1 from your app with Google's absurd NDK policy, you would have to drop support for all devices with less than Android 5.
Are there workarounds?
Kind of, but it's unlikely any developer has the time for them.
The first workaround was mentioned above: carefully test your app on every possible old Android version, not only for symbol-not-found crashes but behavior changes too.
The second workaround is that you can, in theory, "ship" different versions of your NDK code to customers with different API versions.
The easiest way is probably do that at the NDK level. For example, you could build multiple myapp.so
s in your NDK build, each with a different APP_PLATFORM
value in Application.mk, and bundle all of them into your app .apk
. Then from your Java code you could System.LoadLibrary()
a different .so
depending on the API version of the customer's device.
This would be similar in structure to how NDK developers currently bundle multiple NDK versions for each architecture (e.g. armeabi
, armeabi-v7a
, mips
, x86
).
However, there is a massive practical difference: unlike the multiple ABIs which ndk-build
more or less provides for free without wasting developer time, the developer would have to spend a lot of time hacking both the NDK and Java build scripts to create and distribute multiple APP_PLATFORM
.so
versions.
Then every time developers change their C code, they must carefully consider how each function they call behaves (if it even exists) in each API version. This kind of work is totally expected and acceptable for calls that relate directly to new hardware features, but it is totally ridiculous that Android's NDK team makes us do this for calls like atof()
and rand()
.
The third workaround is the one I suspect most developers do: fix problems as angry customers report them, and pray that there are not more such crashes to come (or causing customers to give their apps bad reviews and never reporting the issue to the developer).
dlsym() is a non-workaround
You might compare the C NDK APP_PLATFORM
compatibility issue to the much cleaner Java targetSdkVersion
and say
"hey, if I can set targetSdkVersion
and then check for new features at runtime in Java, can't I set APP_PLATFORM
and check for new features in runtime at C?"
Well, no.
The first problem is that in order to do this in C, unlike in Java, you have to refrain from even referencing the routine in your code. Then you'd have to open the C library with dlopen()
and try to extract the routine you want with dlsym()
. Let's not even get into the likelihood of vendor device-dependency even finding the C library. Plus the complexity that due to Android breaking changes, some routines have changed name so even the name you look up would have to depend on the API version of the device.
But the second, worse problem is that sometimes you are not the one making the call. As we will explain just below, the compiler may insert calls to routines that Google has broken, like stpcpy()
and std::vector::resize
, and you are not in a position to replace these invocations with a call to dlopen()
and dlsym()
. The only way to prevent the compiler from calling them is to reduce APP_PLATFORM
, and this defeats the purpose of accessing new features on compatible devices.
Would having a list of changed calls help?
Amazingly, no. The problem is even worse than it sounds.
Let's say hypothetically Google did publish a complete list of all routines like atof()
with breaking changes. You could just scan your code for those routines, and if you don't call them, you're safe, right?
Wrong.
It turns out that sometimes the bleeding compiler will call those breaking-changes routines without them actually appearing in your code:
- if you build to
APP_PLATFORM
21 (Android 5) and run on older devices (Android <5), you may be shocked to see crashes because the older devices cannot find stpcpy()
, a routine you never call. It turns out the compiler will notice certain stpcpy()
-like patterns in your code and replace them with a call to stpcpy()
! This can be seen in many stackoverflow examples: example 1 example 2 example 3 example 4. If you attempt to implement stpcpy()
yourself for backwards compatibility, you will get an infinite loop unless you are clever enough to implement it in a sufficiently non-stpcpy()
way! Insanity.
- if you build to
APP_PLATFORM
24 you will encounter a similar problem with std::vector::resize
as Karu mentions in a comment of this question).
In both cases, the compiler decides it can insert these calls because it inspects the set of include files you are using---your APP_PLATFORM
---and decides the calls are available. You can't reduce APP_PLATFORM
without losing access to the new routines you want to use on new devices. Catch-22.
How could Google get away with this?
Summarizing the above, Google's effective NDK policy is:
Every time you increase the APP_PLATFORM
of your project, you have to set minSdkVersion
equal to APP_PLATFORM
, preventing your app from running on older devices, unless you are willing to do massive exhaustive testing on old devices, or pray.
I could never actually find an official Google policy statement to this effect. Google's official documentation says exactly the opposite. In particular, this passage from Google's official NDK Levels documentation is utter nonsense:
Each new release of NDK headers and libraries for a given Android API level is cumulative; you are nearly always safe if you use the most recently released headers when building your app. For example, you can use the NDK headers for Android API level 21 for an app targeting API level 16. By doing so, however, you increase your APK's footprint.
In fact the exact opposite is true: it is imperative that you use API level 16 headers to target API level 16 devices, otherwise your app will crash if you use one of a large set of APIs like atof()
with undocumented breaking changes.
About the most help we ever got from Google was a cryptic NDK build warning WARNING: APP_PLATFORM android-XX is larger than android:minSdkVersion
without any associated documentation that I could find.
I hope this answer has been eye-opening and helpful for developers, and perhaps as we increase awareness amongst developers it might motivate the Google NDK developers to respect backwards compatibility as much as the Google Java developers do.
References: many, many links to other stackoverflow answers and other webpages are interleaved in the text above.