Crashlytics Generate Symbols gradle step fails: GC Overhead Limit Exceeded

Question

I'm converting a project from gradle version 3.3 to 4.10.1. This project is mostly C++ code built using a custom build step - not CMake (externalNativeBuild) or Android.mk(ndkBuild). It produces the following libraries:

armeabi-v7a unstripped: 883.2MB
arm64-v8a unstripped: 864.6MB
armeabi-v7a stripped: 15.6MB
arm64-v8a stripped: 23.9MB

The build process fails at the crashlyticsGenerateSymbolsGoogleDistribution step, with the error OutOfMemoryError: GC Overhead Limit Exceeded.

Is there a method or argument for providing more memory to the crashlytics plugin step?

---

The top level gradle.properties file contains the following jvm args: org.gradle.jvmargs=-Xmx4g -XX:MaxPermSize=512m - though I have tried as much as org.gradle.jvmargs=-Xmx12g:MaxPermSize=2g. No changes to these values seem to affect the crashlyticsGenerateSymbols step.

If I run an assembleGoogleDistribution build, skipping the crashlytics step, the build completes with no issues.

Build stack trace:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':application:crashlyticsGenerateSymbolsGoogleDistribution'.
> GC overhead limit exceeded

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':application:crashlyticsGenerateSymbolsGoogleDistribution'.
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
    at org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
    at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
    at org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
    at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
    at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
    at org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
    at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
    at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
    at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:300)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:292)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:174)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
    at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
    at org.gradle.execution.taskgraph.LocalTaskInfoExecutor.execute(LocalTaskInfoExecutor.java:42)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:277)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareWorkItemExecutor.execute(DefaultTaskExecutionGraph.java:262)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:135)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.execute(DefaultTaskPlanExecutor.java:200)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.executeWithWork(DefaultTaskPlanExecutor.java:191)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$ExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.interleaveRanges(DwarfDataParser.java:321)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:275)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.processCompilationUnit(DwarfDataParser.java:194)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.readCompilationUnit(DwarfDataParser.java:173)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.readCompilationUnit(DwarfDataParser.java:139)
    at com.crashlytics.tools.utils.dwarf.DwarfDataParser.parse(DwarfDataParser.java:62)
    at com.crashlytics.tools.utils.elf.ElfDataParser.parseElf(ElfDataParser.java:131)
    at com.crashlytics.tools.utils.elf.ElfDataParser.parse(ElfDataParser.java:101)
    at com.crashlytics.tools.utils.elf.ElfDataParser.parse(ElfDataParser.java:39)
    at com.crashlytics.tools.android.project.codemapping.csym.ElfCSymFactory.createCSymFromFile(ElfCSymFactory.java:99)
    at com.crashlytics.tools.android.project.codemapping.csym.NdkCSymGenerator.generateCodeMappings(NdkCSymGenerator.java:70)
    at com.crashlytics.tools.android.project.CSymManager.generate(CSymManager.java:96)
    at com.crashlytics.tools.android.DeveloperTools.processProperties(DeveloperTools.java:692)
    at com.crashlytics.tools.android.DeveloperTools.processArgsInternal(DeveloperTools.java:348)
    at com.crashlytics.tools.android.DeveloperTools.gradleMain(DeveloperTools.java:292)

---

Gradle and Java info:

./gradlew --version

------------------------------------------------------------
Gradle 4.10.1
------------------------------------------------------------

Build time:   2018-09-12 11:33:27 UTC
Revision:     76c9179ea9bddc32810f9125ad97c3315c544919

Kotlin DSL:   1.0-rc-6
Kotlin:       1.2.61
Groovy:       2.4.15
Ant:          Apache Ant(TM) version 1.9.11 compiled on March 23 2018
JVM:          1.8.0_121 (Oracle Corporation 25.121-b13)
OS:           Mac OS X 10.13.6 x86_64

top level build.gradle

...
buildscript {
    dependencies {
        classpath 'com.android.tools.build:gradle:3.3.0'
        classpath 'com.google.gms:google-services:4.1.0'
        classpath 'io.fabric.tools:gradle:1.31.0'
    }
}
...

application build.gradle

apply plugin: 'com.android.application'
apply plugin: 'io.fabric'
...
crashlytics {
    enableNdk true
    androidNdkOut './../../../build/crashlytics'
    androidNdkLibsOut './../../../build/crashlytics/lib'
}
...
dependencies {
    // Crashlytics Kit
    implementation('com.crashlytics.sdk.android:crashlytics:2.10.1@aar') {
        transitive = true
    }
    // NDK Kit
    implementation('com.crashlytics.sdk.android:crashlytics-ndk:2.1.0@aar') {
        transitive = true
    }
}
...

(I have confirmed that the androidNdkOut path contains armeabi-v7a and arm64-v8a folders which contain our unstripped libnative.so files, and androidNdkLibsOut contains the same folders with our stripped libnative.so files)

---

How can I ensure that the crashlytics plugin is getting enough memory for its crashlyticsGenerateSymbols step so that it doesn't exceed the garbage collector overhead?

Mathamathe

Does trying a few older versions of the Fabric Gradle plugin get rid of the OOM? Might be worth trying as far down as 1.28.0. docs.fabric.io/android/changelog.html#fabric-gradle-plugin – Danford

I tried every version down to and including 1.28.0 with no success. The project was upgraded from 1.24.2 (where it worked, but that version is too old now) – Mathamathe

Thanks, so looks like an OOM happens regardless of the plugin version you use when you're generating symbols. Could you run the following command on your unstripped binary and see how big the output is? readelf --debug-dump=info <path_to_.so> -> ~/sample/directory/dump.txt the file size will approximately be the size of the raw debug information that we read to generate symbols in the first place. – Danford

Were you able to resolve this issue as I am also getting the same error? – Neigh

I did not, the issue is that our native symbols end up being ~6.5GB per architecture, and the Crashlytics gradle plugin doesn't handle that at all. The plugin should probably stream the symbols in/out as needed, rather than loading them into an in-memory structure all at once. How easy that is to do, and how much work it is to fix, I'm not sure. That is unfortunately in the hands of the plugin developers. – Mathamathe

Encountered the same problem, created issue github.com/fabric/fabric/issues/2057 – Maryleemarylin

Answer 1

Try putting a file fabric.properties in the app directory with the following content

disableEnhancedSymbolGeneration=true

It disables DWARF information processing but retains symbols information in mapping. You will not get file names and lines in resulting stacktraces in this case. Still you will get full function names (including namespaces) instead of function addresses.

Mutate

Anyone figure out the trick here for Firebase Crashlytics -- which clearly has the same problem? – Instill

This isn't going to meaningfully impact the memory usage of the Crashlytics plugin - we have a sum total of 13GB of symbols expanded into memory, and then used to produce output (also in memory). The problem is architectural. The symbols should be streamed in and processed in chunks, not all loaded into memory for a factory to process beginning to end. – Mathamathe

Answer 2

Please try to downgrade the Fabric Plugin from 1.31.0 to 1.25.4 in build.gradle

classpath 'io.fabric.tools:gradle:1.25.4'

Outsell

Not compatible with newer libraries and tools, hence why the project was upgrading in the first place. – Mathamathe

I still use the fabric 1.25.4 in my project and everything still fine. I got the same trouble before so that's why I suggest to downgrade it. Do you have the log error when you use 1.25.4 – Outsell

I can't remember all the changes I had to make, and the error messages I saw along the way. It was over 9 months ago now. Do you use the Android X libraries in your project? What version of Gradle, and the Android Plugin do you use? – Mathamathe

I use Android X, Gradle 4.6, here is the depedencies classpath 'com.android.tools.build:gradle:3.2.1' classpath 'io.fabric.tools:gradle:1.25.4` classpath 'com.google.gms:google-services:4.1.0 My SO Size is similar with yours, armeabi-v7a: 750MB, arm64-v8a: 780MB – Outsell

Answer 3

I run into this issue and found two walk around. 1: use NDK r18 and Android Gradle plugin 3.4.2. (19 and later NDK will cause java heap issue when generate symbol for arm64-v8a) 2: use most recent NDK r21d and set linker flag -fuse-ld=lld If use ndkBuild, set APP_LDFLAGS += -fuse-ld=lld in application.mk

Those two worked for me to get pass this issue.

Itinerary