Android: my application is too large and gives "Unable to execute dex: method ID not in [0, 0xffff]: 65536"?

Question

I am trying to integrate my application with Box, Dropbox, and Google Drive. All 3 of these services require a number of 3rd party jars. Additionally, my application already requires a few 3rd party jars. Now when I try to run my application from eclipse I get the following error:

Unable to execute dex: method ID not in [0, 0xffff]: 65536 Conversion to Dalvik format failed: Unable to execute dex: method ID not in [0, 0xffff]: 65536

It seems that this error occurs because my application has too many methods. I'm fairly certain the bulk of these methods are from the 3rd party jars, so it is unrealistic to try to solve this by simplifying my code. I found these two suggestions online.

1. add dex.force.jumbo=trueto project.properties (and use adt version 21). I did this but still get the error.

2. Use multiple dex files as explained here: http://android-developers.blogspot.co.il/2011/07/custom-class-loading-in-dalvik.html. This seems likely to be the only option, but I don't understand how it applies in my case. The issue is that services like Drive have too many dependencies. Wouldn't this solution require that I modify the Drive source to use inflection when referring to its dependencies? (this is clearly not an option).

3. Use proguard to shrink remove unused code/methods. Exporting my application with proguard does work, and the document service integration works as expected on a >4.0 device. However, classnotfound errors are thrown when testing on a 2.3 device.

So, I am hoping for some advice on this issue. Is option 2 a solution for my case? Is there another solution I should consider?

Elisavetgrad

"All 3 of these services require a number of 3rd party jars" -- AFAIK, none of those require any third party JARs. You have elected to use JARs, and hence their dependent JARs, to access those services. However, since all of those services are accessible by means other than Java-based clients, they all have some underlying Web service API. For example, in the case of Dropbox, it is a REST-style API: dropbox.com/developers/core/api – Abby

It would be more accurate to say that the "android libraries" for these services require a number of 3rd party jars? – Elisavetgrad

My point is that you do not need to use those libraries. Python developers do not use those JARs. Ruby developers do not use those JARs. JavaScript developers do not use those JARs. Those JARs are there for you as a convenience, but if they are giving you grief, drop down a layer and access their Web service API more directly. – Abby

i understand your point and agree it is a valid, though far from ideal, solution. – Elisavetgrad

I would say proguard is your best bet. I would spend some time investigating why you're getting ClassNotFound exceptions. You probably need to tweak the proguard configuration. – Grimonia

@Grimonia One concern with the proguard solution: If proguard reduces the method count sufficiently, how can I check by how much? If its just barely small enough to avoid this issue, then I will be likely to have the same trouble in the future? – Elisavetgrad

@Elisavetgrad You can run dexdump on it with the -f option, and look at the method_ids_size field. – Grimonia

@Grimonia Awesome, thanks. I've looking for a way to do this. So, I install dexdump and run DexDump -f /data/app/com.myapp.myapp.apk. It spits out a bunch of information for each class in my app. Any suggestion for a way to use this to find the total method declarations? – Elisavetgrad

It's right near the top -- should look like method_ids_size : 36700. dexdump is part of the SDK. – Sporadic

Answer 1

You can also develop one or more of these as a plugin to your main app, in the form of a separate APK available for download. That APK would expose some component that the main app would use -- since I do not know the nature of your integration with these services, I cannot make a more specific recommendation about that. You would use your own signature-level custom <permission> to secure communications between the two apps. And, as a bonus, if using the third-party library adds requirements for additional permissions, you would only need those permissions in the plugin APK, keeping your main APK smaller.

Abby

this is the most realistic solution I've seen so far. I will leave the question unanswered for a bit, in hopes of more feedback, but I likely will end up using this. – Elisavetgrad

I am unable to find an example of this plugin architecture. I would like to find examples showing how to check if the plugin is installed, and how to use custom permissions for secure communications. Could you point me in the right direction? – Elisavetgrad

@ab11: Well, since the nature of a plugin architecture is heavily dependent upon what the plugins are actually doing, you will find it difficult to find general-purpose examples. Here is an example of a pair of projects implementing a plug-in UI by means of passing RemoteViews from the plugin to the host: github.com/commonsguy/cw-omnibus/tree/master/RemoteViews Roman Nurik's DashClock shows an app widget/lockscreen widget that supports plugins: code.google.com/p/dashclock – Abby

Answer 2

***NEW**** All of the other answers are now outdated. Here's the new fix

Android 5.0 and higher

Multi-dex support is included automatically. From the docs:

Android 5.0 and higher uses a runtime called ART which natively supports loading multiple dex files from application APK files. ART performs pre-compilation at application install time which scans for classes(..N).dex files and compiles them into a single .oat file for execution by the Android device. For more information on the Android 5.0 runtime, see Introducing ART.

Below Android 5.0

Simply add the Android mult-dex support tool to your gradle build:

android {
    compileSdkVersion 21
    buildToolsVersion "21.1.0"

    defaultConfig {
        ...
        minSdkVersion 14
        targetSdkVersion 21
        ...

        // Enabling multidex support.
        multiDexEnabled true
    }
    ...
}

dependencies {
  compile 'com.android.support:multidex:1.0.0'
}

Jacal

Answer 3

The Dalvik VM can have a maximum of 65536 methods per dex file, due to the bytecode instruction set not having a way to refer to method numbers requiring more than 16 bits (as pointed out by @danfuzz in the comments).

While it is possible to fix this using multiple dex files, Facebook found another fix that they could deploy within their app to get around the problem.

Orola

after many reads, I'm still not clear how I can implement their solution. I add something to my app to change an internal dalvik buffer? – Elisavetgrad

@Elisavetgrad You need to make a native extension using the android-ndk, which increases the size of the native buffer. – Orola

any possibility there is an example online of how to do this? also, in general, do you think this is a realistic option for a developer (me) that is very unclear about the solution? – Elisavetgrad

@Elisavetgrad AFAIK, facebook hasn't released the code for this (yet). Though you could probably be fine with splitting into multiple dex files, if your app isn't too complicated. – Orola

my concern with splitting into multiple dex files is that it would not be of use for the external services. my understanding is that once spilt, I can dynamically load and reflectively reference the methods in the second dex file. which i could use in my internal source, but would be of no use for the drive/box/dropbox code which contains the bulk of the methods. although, i'm not very clear on this and could definitely use feedback – Elisavetgrad

@Elisavetgrad I'm afraid I can't be of much practical help here. I know the theory, but I've never run into this problem myself, so can't say. – Orola

Note: The restriction isn't a "memory allocation issue." It is that the bytecode instruction set doesn't have a way to refer to method numbers requiring more than 16 bits. I had a fix for that problem in progress when I left the Android team, and it seems that nobody picked it up in the mean time (alas). – Lucky

The Facebook issue was a memory allocation issue, and isn't the same as the 16-bit method reference issue. Note in particular their fix was only needed for older versions of Android (froyo / gingerbread) and does nothing on honeycomb and later. The 16-bit limit applies to all past and present versions equally. – Sporadic

Having this same issue, but I'm a bit confused as to why I'm getting it. I used to compile via ANT in Android Studio, but today I migrated it over to gradle manually (created a new project and copied the old code, modules, and support libs in). It's the same project as before, but obviously something changed be it the way the libraries are referenced or how gradle works itself which has caused it to start throwing this error. Any ideas? – Weiman

Answer 4

See vm/LinearAlloc.c and you can find this code: (5MiB under Android 2.3.3, 8MiB after Android 4.0 as my investigation)

#define DEFAULT_MAX_LENGTH (5*1024*1024)

...

LinearAllocHdr* pHdr;

...

pHdr->mapLength = DEFAULT_MAX_LENGTH;

I suppose that the 'Facebook fix' is editing this memory by using native C pointer. IMHO LinearAlloc problem and this method ID problem is different thing.

Drucie

Answer 5

I faced this issue recently. After scouring the web for some more detailed implementation, I realized there wasn't much out there other than:

• Good but a little outdated now that Gradle is around: http://android-developers.blogspot.co.il/2011/07/custom-class-loading-in-dalvik.html
• Not much details but a vague idea of how it could be done: https://www.facebook.com/notes/facebook-engineering/under-the-hood-dalvik-patch-for-facebook-for-android/10151345597798920

I realized that the problem was not necessarily that there were too many methods in my code, but that the full dex of my code and other libraries was the issue. So, if I could compile my code against the libraries but not include them in the classes.dex, and then dex the libraries separately, and then put it all together at runtime it should work. The only issue left to address is the class loader, which Facebook mentioned in passing.

So with a little reflection and some Groovy code, I think I came up with a relatively stable way to package the libraries and the application code into separate dex files.

https://gist.github.com/nickcaballero/7045993

Heterogeneous

I'll look at that, but just wanted to inform you of my terrible solution. I have some external libraries that i only use for certain things; in deveolpment i don't need to include them in my application unless i'm working on those specific functionalities. so, i remove them from the libs folder so they aren't built into the app and put them on the class path so my code compiles; obviously my app will crash if those features are used, but i'll put the needed one back in to develop on it. When I build for release, I include the libs because progruard removes unused classes from the libs. – Elisavetgrad

Answer 6

Most of the problems with hitting the 65k method limit are related with the use of the mastodontic Google Play Services in your apps. Recently, you can get more granularity when using it.

Following this guide, you can use only parts that you want. Probably this will fix the problem, avoiding some black-magic tricks, or using multiDex. For example, if you only want Google Maps in your app (and you aren't using ads, wallet, google wear, analytics, etc...), using the entire dependency is a waste of time/space. You can use it that way:

compile com.google.android.gms:play-services-base:6.5.87
compile com.google.android.gms:play-services-maps:6.5.87

You can read the entire list of "parts" in this link

Ethnarch

Answer 7

Here is a script I wrote for counting the number of methods in each jar (and in total) for a specific folder.

Once you count the methods you can concentrate on refactoring and removing heavy libraries.

Alphabetical

Answer 8

You need to enable the Dex support for that. So you need to do these steps:

1. Gradle plugin v0.14.0 for Android adds support for multi-dex. To enable, you just have to declare it in build.gradle:

android {
   defaultConfig {
      ...
      multiDexEnabled = true
   }
}

2. if app support > 5.0 (that is, if your minSdkVersion is 20 or below) you also have to dynamically patch the application ClassLoader, so it will be able to load classes from secondary dexes. for that you can add this lib.

 dependencies {
      ...
      compile 'com.android.support:multidex:1.0.0'
    }

3. enable in code for that you have these option. choose one which suits you best

A. Add MultiDexApplication in manifest manifest

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.android.multidex.App"> <application
android:name="android.support.multidex.MultiDexApplication">
</application>
</manifest>

B. Extend the application by MultiDexApplication

public class App extends MultiDexApplication { .. }

C. install it in application in attaching base context.

public class App {
    protected void attachBaseContext(Context base) {
        super.attachBaseContext(base);
        MultiDex.install(this);
        ..
    }

}

For more go through this link MultiDex.

Politick