android - [DEX] Sky’s the limit? No, 65K methods..-PHP Chinese Network Q&A

No, 65K methods is Dealing with the DEX methods limit on Android and Google Play Services.

It happens in the blink of an eye. Before, you are an happy Android developer, head down on your (or your company’s) application, adding the coolest libraries to provide more functionalities and to write simpler code. Afterwards, you stare at the dreaded output that states:

Unable to execute dex: method ID not in [0, 0xffff]: 65536
Conversion to Dalvik format failed: Unable to execute dex: method ID not in [0, 0xffff]: 65536
And you are stuck, unable to create the DEX file for the APK. You have no idea of its meaning, nor the slightest clue about how to get around it. All you can do is going all in for the most logic option: panic.

Getting to know your enemy

After you try all the usual tricks (which involve, in order: cleaning the project, restarting the IDE, building via command line, restarting your laptop and your friend’s too, and going out to see if in 10 minutes the problem disappears on its own), you face the hard truth: the problem is here to stay. So it might as well be worth knowing what is causing it.

There are a ton of different posts on the problem: this, this one, this one as well; oh, and that and this one too. So, what’s happening here?

Basically, it seems like you’ve hit something which is commonly (and improperly) referred to as the Dalvik 65K methods limit. In short (thanks fadden):

You can reference a very large number of methods in a DEX file, but you can only invoke the first 65536, because that’s all the room you have in the method invocation instruction.
[…] the limitation is on the number of methods referenced, not the number of methods defined. If your DEX file has only a few methods, but together they call 70,000 different externally-defined methods, you’re going to exceed the limit.
There you go. Your application (or again, your company’s) has too many methods, both written by you or bundled inside the libraries’ JAR files. For this reason, the dx tool cannot write some methods addresses because it simply won’t fit the space reserved for that particular field in the DEX file (which, in turn, contains the compiled Java classes). For this reason, this problem should be referred to as the DEX 65K methods limit.

And yes, you got it right. This issue won’t disappear even when Android will switch to the new ART runtime, unless Google decides to “fix” the DEX format or ditch it for another one.

Let’s do the math

You’re probably wondering how in the holy heavens you managed to shovel more than 65.000 methods in your precious APK. Or, you’re not. Either way, there must be a way of counting those methods and figure out their origin, i.e. their package name.

The DEX header already comes with the number of method references information (just to be clear, this number represents the unique method references, not the sum of every method invocation). But we also would like to see which packages are adding an excessive amount of methods.

The best tool I’ve found so far was made by Mihai Parparita, who coded a simple bash script called dex-method-counts. It is extremely fast and offers, in addition to the overall methods count, an XML-ish view of the packages together with their number of methods. Other solutions, such as dex.sh by the almighty Jake Wharton, gives the exact same result, but takes much more time due to its recursive approach (Jake also wrote an extremely interesting article on this very subject, you should read it as well).

Now that we have the right tool, why don’t we take a look at a sample application, built specifically for the occasion? Called SampleApp, its code consists in this simple Activity:

package com.whyme.example;
import android.app.Activity;
import android.os.Bundle;
public class SampleActivity extends Activity {
@Override
public void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);
dummyMethod();

}
private void dummyMethod() {

System.out.println(“Oh gosh.”);

}
}
I have willingly included the Google Play Services 5.0.77 library, even though I’m not using it at all, and the reason behind this will be crystal clear in a few lines time. Now, let’s analyze the output of the dex-method-counts script (collapsed, for the sake of simplicity):

Read in 20332 method IDs.
<root>: 20332
: 2
android: 834

 […]

com: 18802

 google: 18788
   […]
 whyme: 14
   example: 14

dalvik: 2
system: 2
java: 624

 […]

javax: 5

 […]

org: 63

 apache: 24
   […]
 json: 39

Wait, what? I just threw one method for my Activity and I already have 20.000 methods on my DEX file?

It’s fairly easy to spot the culprit here.

Google Play Services: a mixed blessing

There are more than a few chances that you are familiar with Google Play Services. A clever way for Google to provide support for its many services APIs, all the way back to Android 2.3 Gingerbread, with an iteration time of 6 weeks between one release and the other.

However, such achievement comes at a price, and that would be the ENORMOUS amount of methods that the Play Services library comes with. We’re talking of nearly 19.000 methods here. And given the 65.536 limit, it means that 1/3 of the methods that we can include is gone. Just so.

Now, before moving towards “the solution” of this problem, I think that a brief consideration is due here. Some developers, both inside and outside of Google, share a common opinion, according to which “if you hit the method limit you are a bad, bad developer and you deserve this punishment [plus, you should burn in hell]”. In a more objective view, this means that you have recklessly included too many external libraries, either because you are lazy or because your app does too many things. Personally, I do not share this point of view. I think this is just an excuse which serves two key points:

I do not want to admit the problem/I don’t want to deal with it
I didn’t reach the limit, this means I’m a hardcore developer, and you are obviously doing something wrong
As if the limit was being put there on purpose thinking “This limit will serve as the line between skilled and unskilled developers, for all the future generations. So be it.”. No, that’s not the way I see it.

Strip it till you make it

In a world where justice rules over everything, Google acknowledges the problem and decides to reduce this issue by splitting the gigantic Play Services library into submodules, which can be included depending on the functionalities that we need in our app. For instance, if my app doesn’t deal with gaming, there is no point for me in including the whole Play Games APIs. This is probably the most reasonable (and doable) approach and, as it will turn out, it can be feasible too.

However, Google didn’t (even internally) fully acknowledge this problem, so we better find a way to fix it ourselves. That is to say, manually stripping the google-play-services.jar file, removing the unnecessary parts of the APIs that we do not need for sure.

There are a couple of ways of doing so. You can do it manually by yourself, but keep in mind that it is something that you need to do every six weeks.

You can rely on the JarJar library, which “makes it easy to repackage Java libraries and embed them into your own distribution”. Essentially, it removes the desired packages from the jar file. Plain and simple.

Or, if you like me prefer the good ol’ command line way, you can use the same script that I’m using for my company, which I called (guess what) strip_play_services.sh. Together with the script comes a configuration file called strip.conf, which you can use to disable the desired modules of the Play Services library. What this script does is essentially the following:

Extracts the content of the google-play-services.jar file
Checks if strip.conf already exists: if so, it uses that configuration, otherwise it creates one by reading all the modules from the Play Services library
Based on the strip.conf content, it removes the desired modules (in the form of directories)
Repackages the remaining directories in a google-play-services-STRIPPED.jar file
This is a sample configuration for the strip.conf file:

actions=true
ads=true
analytics=true
appindexing=true
appstate=true
auth=true
cast=true
common=true
drive=false
dynamic=true
games=false
gcm=true
identity=true
internal=true
location=false
maps=false
panorama=false
plus=true
security=true
tagmanager=true
wallet=false
wearable=true

That’s it. Now, for the sake of the example, let’s count the methods one more time to see what has changed:

Read in 11216 method IDs.
<root>: 11216
[...]
Now that is something. With a proper configuration, tailored on the needs of your application, you can manage to shrink the Google Play Services library and make more room for the methods that you (probably) really need.

What you also need is to be careful, though. The Play Services library is very well built (in terms of modules decoupling), and you can safely strip the “games” module without hindering the “maps” module. But, if you strip a module which is used by others (say common, or internal), nothing will work. Testing is advised!

Is there any other way?

There sure is. Running ProGuard on your application can help, since it strips the unnecessary methods from your code and, in addition, shrinks the size of your final APK. But don’t expect huge reductions, because there won’t be.

Another solution consists in creating a secondary DEX file, which contains parts of your code that you will access via interfaces/Reflection and which you will load via a custom ClassLoader (more info). This approach can prove to be more effective in certain circumstances (e.g. when you can easily isolate a module of your application, or when you only need a few methods from an external library). However, setting it up is not trivial, especially if you (like me) need to access it via Reflection. If you’re interested on this topic as well, I can post more material or share my solution.

UPDATE 22/11/2014

We have multiple news on the DEX method limit problem! First of all, Google has officially acknowledged the problem and has provided us with the Multidex option. For a full coverage on this, read the official page. For an example, read this post by Alex Lipov.

Moreover, Google realized (oh my god, finally) that Google Play Services has become way too big. So, starting from the upcoming 6.5 version, we will have modular Play Services!

As we’ve continued to add more APIs across the wide range of Google services, it can be hard to maintain a lean app, particularly if you’re only using a portion of the available APIs. Now with Google Play services 6.5, you’ll be able to depend only on a minimal common library and the exact APIs your app needs. This makes it very lightweight to get started with Google Play services.
Read the full announcement and rejoice, Internet!

UPDATE 09/12/2014

Here we go people! The new Google Play Services 6.5 with a modular structure is now up and running! Check the official documentation, the blog post and one of the funniest DevBytes video ever!

Conclusions

I did choose this as the subject of my first article because, having found myself right in the middle of this issue, I had to find all the pieces and put them together. I don’t have the claim of having written a complete guide to this problem, I sincerely hope that the contents can be uses as a close-to-straight path for developers to put this problem at bay.. for some time.

Feedbacks and opinions are extremely welcome. If you think I’ve made a mistake (probably more than one) or that I haven’t been clear enough on some subjects, please feel free to comment, I will happily correct it!

Hope you enjoyed the reading. ☺

EDIT: I’ve just found out that my good friend Dario Marcato has created a Gradle task to accomplish the same stripping. If you’re using Gradle, be sure to check it out!

AndroidGoogle Play Services