Using Gradle to split external libraries in separated dex files to solve Android Dalvik 64k methods limit
Asked Answered
S

4

23

Is there a proper/easy way to solve the 64k methods limit using Gradle?

I mean some custom Gradle task to use pre-dexed jars to create separated dex files, instead of a single classes.dex.

Thank you

Ivan

Current status

Currently, I'm struggling with GMS: it brings in 20k methods to use Analytics. I use Proguard to strip down what's not need, but still... 72k methods and counting...

I can split classes.dex in two file using dx parameter --multi-dex. I achieved it manually editing

sdk/build-tools/android-4.4W/dx

and editing last line like this:

exec java $javaOpts -jar "$jarpath" --multi-dex "$@"

My APK file now contains __classes.dex__ and __classes2.dex__.

I'm trying to dynamically load the second file with a few methods:

Unfortunately still no luck. I really hope some Google/Facebook/Square guru can provide a proper solution.

Stingo answered 12/5, 2014 at 16:21 Comment(6)
Note that if your problem is solely in debug builds, because release builds' use of ProGuard strips out enough stuff, there's a recipe floating around for enabling ProGuard for your debug builds as well.Impudent
We tried proguard on debug build too. It's insanely slow and don't really solve the problem because we have lots of third-party libs that don't support progard very well: guava, retrofit, rx, dagger... We really need to split the dex.Stingo
This isn't an answer but might be helpful. Use the same package name in every library (modify AndroidManifest.xml). If you don't, then R.java might create a lot of extra unneeded variables. This helped me out tons in a project with the 64k method count issue.Quadrillion
@hamen It seems that sschuberth gave the acceptable answer, would you mind make it acceptable? thanks!Singh
When running into this for GMS / GPS in particular, also see this Google Play services and DEX method limits article.Woermer
They read that post by Jake Wharton :DStingo
W
25

Update for Android Gradle plugin 2.2.0: It is not possible to access the dex task anymore, but in exchange additionalParameters was introduced as part of dexOptions. Use it like

android {
  dexOptions {
    additionalParameters += '--minimal-main-dex'
    // additionalParameters += '--main-dex-list=$projectDir/<filename>'.toString()
    // additionalParameters += '--set-max-idx-number=55000'
  }
}

Update for Android Gradle plugin 0.14.0: There is now direct multi-dex support via the new multiDexEnabled true directive (requires build-tools 21.1.0, support repository revision 8, and Android Studio 0.9).

Original answer: Ever since Gradle Android plugin 0.9.0 you actually can pass the --multi-dex to dx by adding this to you app's build.gradle file:

afterEvaluate {
    tasks.matching {
        it.name.startsWith('dex')
    }.each { dx ->
        if (dx.additionalParameters == null) {
            dx.additionalParameters = ['--multi-dex']
        } else {
            dx.additionalParameters += '--multi-dex'
        }

        // Add more additional parameters like this:
        dx.additionalParameters += '--main-dex-list=class-list.txt'
        dx.additionalParameters += '--minimal-main-dex'
    }
}

So far for the creating he multiple dex files. To actually use the multiple dex files, take a look at https://github.com/casidiablo/multidex (which is a fork of Google's upcoming MultiDex support library).

Woermer answered 22/9, 2014 at 13:46 Comment(5)
I'll definitely give it a try! ThanksStingo
@hamen If it worked for you, would you mind accepting the answer?Woermer
As soon as I can test it. I'm not working on a project that big anymore. Sorry. Be patient, please.Stingo
Heads up official solution from Google out. MultiDex support library. This blog post explains it pretty well https://medium.com/@mustafa01ali/dexs-64k-limit-is-not-a-problem-anymore-well-almost-2b1faac3508Philpott
It would have been nice if the author of that article actually were giving some credit to this StackOverflow thread from where he was obviously copying some answers.Woermer
P
8

In case gms was your issue and you are using gradle

Starting from gms version 6.5 you can choose individual API libraries

for example to include only the Maps API :

compile 'com.google.android.gms:play-services-maps:6.5.87'

and here is the complete list :

      com.google.android.gms:play-services-base:6.5.87
      com.google.android.gms:play-services-ads:6.5.87
      com.google.android.gms:play-services-appindexing:6.5.87
      com.google.android.gms:play-services-maps:6.5.87
      com.google.android.gms:play-services-location:6.5.87
      com.google.android.gms:play-services-fitness:6.5.87
      com.google.android.gms:play-services-panorama:6.5.87
      com.google.android.gms:play-services-drive:6.5.87
      com.google.android.gms:play-services-games:6.5.87
      com.google.android.gms:play-services-wallet:6.5.87
      com.google.android.gms:play-services-identity:6.5.87
      com.google.android.gms:play-services-cast:6.5.87
      com.google.android.gms:play-services-plus:6.5.87
      com.google.android.gms:play-services-appstate:6.5.87
      com.google.android.gms:play-services-wearable:6.5.87
      com.google.android.gms:play-services-all-wear:6.5.87
Portray answered 9/12, 2014 at 19:6 Comment(0)
U
2

An example project partitioning and loading different dex files can be found here:

https://code.google.com/p/android-custom-class-loading-sample/

EDIT: For Gradle you already have an answer

Custom Class Loading in Dalvik with Gradle (Android New Build System)

Ulland answered 12/5, 2014 at 16:32 Comment(4)
It uses Ant. I'd like to use Gradle.Stingo
About the edit: That solution seems to be for Java, not Android.Stingo
@hamen: The "Custom Class Loading in Dalvik with Gradle" link points to an SO question and answer that is about custom class loading in Dalvik with Gradle.Impudent
Can you point me to some doc to split the class.dex in something like: rx.dex, retrofit.dex, mockito.dex... and then load them at runtime?Stingo
S
2

I am the maintainer of https://github.com/creativepsyco/secondary-dex-gradle/ and I am a gradle n00b, therefore I chose the path of BASH scripts although I think it can be done directly in the build file. OR can be refactored to run as a plugin, i might do that when I am upto to terms with Gradle. Here is the reason for my logic.

In order to understand how to split the DEX you must know the build system task order. If you are using gradle then you must know that there are a series of tasks injected inside the build cycle.

For example:

:sdk:processReleaseJavaRes UP-TO-DATE
:sdk:packageReleaseJar
:sdk:compileReleaseNdk UP-TO-DATE
:sdk:packageReleaseJniLibs UP-TO-DATE
:sdk:packageReleaseLocalJar UP-TO-DATE
:sdk:packageReleaseRenderscript UP-TO-DATE
:sdk:packageReleaseResources UP-TO-DATE
:sdk:bundleRelease
:app:prepareComAndroidSupportAppcompatV71910Library UP-TO-DATE
:app:prepareComFacebookAndroidFacebook3141Library UP-TO-DATE
:app:prepareDebugDependencies
:app:compileDebugAidl UP-TO-DATE
:app:compileDebugRenderscript UP-TO-DATE
:app:generateDebugBuildConfig UP-TO-DATE
:app:generateDebugAssets UP-TO-DATE
:app:mergeDebugAssets UP-TO-DATE
:app:generateDebugResValues UP-TO-DATE
:app:generateDebugResources UP-TO-DATE
:app:mergeDebugResources UP-TO-DATE
:app:processDebugManifest UP-TO-DATE
:app:processDebugResources UP-TO-DATE
:app:generateDebugSources UP-TO-DATE
:app:compileDebugJava
:app:preDexDebug
:app:dexDebug
:app:processDebugJavaRes UP-TO-DATE
:app:validateReleaseConfigSigning
:app:packageDebug
:app:zipalignDebug
:app:assembleDebug

In order to do Dexing you should be able to be able to inject your custom task between the dex* and the process* tasks. If you can do this, then Multiple DEXing becomes easy.

The Bash script here essentially does this, if you examine the debug task, it will basically:

  • Get the Library Jar file to dex, usually it is build specific & exists in the exploded-aar folder for Android Libraries & run the DEX tool on it
  • Copy this over to the assets folder, which exists in the final libs folder to be packaged inside the app
  • All the library resources etc are already merged, which means there is a need to unzip & zip the file again.

In the gradle build script

 // For Debug simply remove the library from getting dex and create it
                //----------------------- Extra Debug Step ----------------//
                def libraryFiles = new ArrayList<?>()
                def secondaryFile = new ArrayList<?>()

                variant.dex.libraries.each {
                    File file ->
                        if (!file.absolutePath.contains("lib/unspecified/classes.jar")) {
                            libraryFiles.add(file)
                        } else {
                            secondaryFile.add(file)
                        }
                }
                variant.dex.libraries = libraryFiles
                //----------------------- Extra Debug Step ----------------//

                packagingTask.dependsOn variant.javaCompile
            }

This manually removes the library from getting dexed, so that it can be generated via the bash script.

I think you can figure out the dexing during the release process in the same way. The other important thing to note is that the Proguard Task is controlled by android gradle plugin and you can't change much about it. Issue with Proguard Rules:

  • Each pass of proguard is different, we don't want to end up in a situation where our 2 DEXes have different proguard mappings
  • This leaves us in a situation where we cannot proguard our libraries, but this is not really desirable.
  • Must generate the dex file after proguard to make sure the mappings are the same. Gradle has no support to merge assets after Proguard (We want to put the dex files in the assets folder)

The other important chunk of code resides in SecondaryDex.java which essentially loads up the second dex file & injects the path of DEX file into the runtime class path. You can optimize this and just inject the path instead of reading the DEX file everytime the app is resumed.

I did the secondary Dex experiment on Google Play Services (which adds 20K methods) and was able to separate into a separate DEX file. This way my main dex file is unaffected by the bloat in Google Play services.

To understand how the Gradle task cycle works, you can refer to the BasePlugin.groovy source, you can see that it is difficult to control some aspects until there is a proper API for accessing variant objects and build tasks.

Scrivens answered 11/7, 2014 at 7:21 Comment(1)
I believe you can just add an action to the begining of the existing dex tasks that sets things up for multidex. Inserting a task is trickier.Quint

© 2022 - 2024 — McMap. All rights reserved.