how to build an APK and separate libraries that the app loads dynamically

Question

The short summary is: How do I build an APK and separate libraries (by which I mean sets of classes (and ideally, resources too) in some form, such as JAR, AAR or DEX files), but not include those libraries in the APK; instead, the app loads them at run time?

Detail

So my main question is how to build such an app (e.g. Gradle configuration). How do I specify which classes go into which JAR or DEX files? Do I create an Android Studio module for each DEX file I want to end up with?

A closely related question is how the Java code should then load the external libraries and access their classes at run time. For the latter, I'm hopeful that the approach shown at accessing to classes of app from dex file by classloader would work.

I've tried the instructions at https://developer.android.com/studio/projects/android-library.html, but that builds an APK that does include the dependency library.

I've also tried Multidex (https://developer.android.com/studio/build/multidex.html), but that doesn't seem to leave the developer any control over which classes go in which DEX file, and furthermore, packages them all into a single APK. AFAICT there is no way to control the loading of these DEX files at run time.

Background

There's a possibility of the "X-Y problem" here, so I'd better explain the background.

I'm building an app for a client. It's not going to be distributed through an app store, so it won't have access to the normal mechanism for updates. Instead, the client wants the app to be able to update itself by downloading new components of itself to replace the old components, without a need to manually sideload a new APK. The primary motive here is that the updates have to be easy for non-technical users. If the app can control the update process, it can make it smooth and guide the user.

Moreover, the app will be used in areas where internet access is scarce and expensive, so the client wants to be able to issue app updates in smaller chunks (e.g. 2MB) rather than forcing the user to re-download the whole app to receive a small update.

One aspect of the requirements I should mention, in case it matters, is that the libraries to be loaded at run time are supposed to live on a microSD card. This can also help with distribution of updates without internet access.

The current status of the app is that it's about 50% written: That is, a couple of earlier versions have been released, but the app now needs to be modified (restructured) to meet the above requirements, as well as others.

Metagnathous

Consider building your own app installer - as apks are just zip files - it will be up to you to deliver them to device #5804499 – Hyperaemia

It's an idea that has crossed my mind. But (a) how would I then break updates into small chunks -- would I have to make each chunk a separate app to be installed? How would this impact the process of developing the overall app (app collection)? (b) the accepted answer at the question you linked to uses reflection to access an undocumented feature. Is it reliable? (c) Will this let me install the apps to microSD card, even if the user hasn't chosen to install apps to SD card? – Metagnathous

I hope you have great insurance, as the lawsuits from the security problems your approach presents would bankrupt many people. "Do I create an Android Studio module for each DEX file I want to end up with?" -- I would assume so, getting the DEX out of the intermediate build results. "how the Java code should then load the external libraries and access their classes at run time" -- as you indicated, DexClassLoader is probably what you want. – Package

"but that builds an APK that does include the dependency library" -- you'll need a tiny library module in common between the APK module and what I'll call the DEX modules. That common module would define the Java interfaces that both sides will use. Then, you can remove the dependencies on the DEX modules from the APK module, using DexClassLoader, reflection, and so on to load your DEXes. At least, this is how I would approach it, if somebody was threatening to nuke Miami if I failed to deliver an implementation meeting your specs. I do not know if this will work, as I haven't tried it. – Package

Depending on what the app is and does, you might be better served using a hybrid app framework, arranging to load the HTML/CSS/JS from a cache that you can update. This avoids all of the DEX headaches, assuming that a reasonable percentage of the updates would be limited to those Web assets. – Package

@CommonsWare: Thanks for your response. Can you elaborate on the security problems? Why would this approach be more of a problem than just handing someone an APK on a microSD card? I.e. if I wanted to compromise their device, couldn't I do it just as well with a simple sideloaded app, as with a sideloaded app that dynamically loads classes? Are you thinking of the risk of a third party planting malicious modules to be loaded? I was planning to have the app core require dynamically loaded modules to be signed with a cert; does that help? – Metagnathous

@LarsH: "couldn't I do it just as well with a simple sideloaded app" -- if you mean a sideloaded update to your existing app, then no, unless the attacker stole your signing key. Android would refuse to install the update. "Are you thinking of the risk of a third party planting malicious modules to be loaded?" -- yes, or modifying existing modules. "does that help?" -- possibly, if done correctly from the outset. Then it becomes a timing attack. Having persistent read/write code is just very difficult to secure well. – Package

@CommonsWare: I didn't mean an update, but a sideloaded original app. Thanks again for your help. – Metagnathous

Both JRebel for Android and Bazel mobile-install manage incremental updates of an app via adb. Maybe this can give you clues on how to do incremental updates. That being said, I think you shouldn't invest time in this complex task, and just rely on the Google Play store. – Guv

An alternative approach would be to split you app in smaller more specialized apps, and let the update append on the full apk for those. Things like ContentProvider and intents can help you decouple the code. – Guv

@rds: Thanks for these ideas. Can you clarify what you mean by "let the update append on the full apk for those"? – Metagnathous

if i understand you these two posts answer you: #24135658 #6858307 – Annotation

silent side-loading paulononaka.wordpress.com/2011/07/02/… there are some workarounds I have used in the past like jni libraries that are really executables, I'll update you. – Chengtu

Tested gradle build ok for the dexloader project. Loads com.example.toastlib.jar from the SDcard (not assets folder). ( you must read the README.md file in the project to build it). – Chengtu

Is this of any use ? #39513548 – Chengtu

Answer 1

This tutorial is a good start for external loading of DEX files. Only three small files of source (MainActivity.java, LibraryInterface.java, LibraryProvider.java) and it copies secondary_dex.jar from the assets folder, into internal application storage [outdex/dex] (the internet is also stated as possible in the tutorial). You have to build it with ant, because it uses custom build steps. I tried it, it works fine. Worth a look.
custom class loading in Dalvik and ART

UPDATE this code has been ported to Android Studio gradle (no need for ant). https://github.com/timrae/custom-class-loader
Tested ok. Copies com.example.toastlib.jar from the SDcard into internal application storage [outdex/dex],(not assets folder). ( you must read the README.md file in the project to build it).

Q: How do I add an Activity, I cannot add it to the manifest ?
A: Use Fragments, they don't need entries in the manifest.

Q: A Jar with resources that is meant to be added to an existing project needs to be able to merge its resources with the project's own resources (R.).
A: Hacks are available, Data file...
Packaging Android resource files within a distributable Jar file

Q: The external file has wrong permissions.
A: Import it.

Q: I need to add uses-permission.
A: Use API23 you can programmatically add uses-permissions (but they still need to be declared in the Manifest, so the new permissions model is probably not much use to us).

This section is for more general users (@LarsH has more specific requirements about updates), The example above is 17kb apk and 1 kb jar. You could put the bulk of you code in the one-off jar, and updates would involve just loading an new Apk (and then importing the bulk code jar, to minimise the data transfer). When the Apk gets too big, start again with a small Apk and everything migrated to another jar (import 2 jar's). You need to balance coding effort, user experience, maintainability, supportability, bandwidth, android rules, play store rules (if these words even exist ;O)).

NOTE Dalvik is discontinued

The successor of Dalvik is Android Runtime (ART), which uses the same bytecode and .dex files (but not .odex files), with the succession aiming at performance improvements transparent to the end users. The new runtime environment was included for the first time in Android 4.4 "KitKat" as a technology preview, and replaced Dalvik entirely in later versions; Android 5.0 "Lollipop" is the first version in which ART is the only included runtime.

Chengtu

Thanks, I had seen this tutorial as a possible approach. Regressing to ant is a big drawback, as currently gradle takes care of a bunch of details that I don't need to worry about -- if we move to ant, we'll have to learn about them and how to get ant to do those, and maintain the build configuration moving forward after Google has moved away from it. Or, there might be a way to port the custom build steps into gradle? – Metagnathous

good news this code has already been ported to Android studio gradle github.com/timrae/custom-class-loader redundant now but quote "Gradle provides excellent integration with Ant." docs.gradle.org/current/userguide/ant.htmlALSO in gradle ant.importBuild 'build.xml' – Chengtu

This port looks very helpful. Thanks ... I will check it out. – Metagnathous

I haven't been able to get this answer implemented in order to verify what it can do, but it's the most promising answer the question received during the time I had to award the bounty, and it does substantially answer the question. So, have a bounty! – Metagnathous

Thanks LarsH I tried my best, interesting question. I will verify it works on gradle without ant, and update. – Chengtu

I bow to ur superior skilz. :-) Seriously, I've been pursuing several different options on this problem, as well as other assignments, and haven't yet gotten back to trying this one out. – Metagnathous

"Use API23 you can programmatically add uses-permissions" -- not as far as I am aware. – Package

Tested gradle build ok for the dexloader project. Loads com.example.toastlib.jar from the SDcard (not assets folder). ( you must read the README.md file in the project to build it). – Chengtu

@JonGoodwin - Should I really just compile and build apk file and rename it to jar and then use dx tool to extract dex file as it states at https://mcmap.net/q/245591/-is-it-possible-to-dynamically-load-a-library-at-runtime-from-an-android-application ? Will such dex file work when loaded? – Cayman

@ΩmegaI have since discovered (bytebuddy.net) is probably the best way to do it.[#47390522} – Chengtu

Answer 2

You could try to build multiple apk's with the same sharedUserId and the same process.

This is the plugin mechanism used by Threema

Edit: More about Theema

Threema has one main app and two plugins:

• main app: https://play.google.com/store/apps/details?id=ch.threema.app
• QR-Code Plugin: https://play.google.com/store/apps/details?id=ch.threema.qrscannerplugin
• Voicemessage Plugin: https://play.google.com/store/apps/details?id=ch.threema.voicemessageplugin

Doing so the main app does not need the permissions for accessing the camera or microphone

Gamophyllous

Interesting idea. Can you tell me more about what Threema does with this? What components are in the initially-launched APK/app, vs. the ones that are launched later? – Metagnathous

So Threema applies updates via the Play store. Unfortunately, as described in the original post, the Play store isn't an option, and sideloading new APK's for each update is seen as difficult for users. If there were a way for the app to install the updated APK's with minimal manual intervention required on the part of the user, this would be a good way to approach the problem. – Metagnathous

Your app could download the new apk and start the installation process. This way the user just have to click one button – Gamophyllous

It's theoretically possible for an app to start the installation process on an APK, using special privileges ... I'm not sure what the restrictions are. Best case, unless the user has "Security > Unknown sources" turned on (which we have to recommend against), they would have to do that manually; then turn it off manually afterwards. I think there are also at least two or three button taps for the installation itself - Install; Use Package Installer (Once only / Always) (if you haven't already made a persistent choice); and approve access permissions. Which defeats the purpose of the question. – Metagnathous