Crashlytics NDK: No symbols for crashes since update to new sdk

Question

It worked fine until we updated to the new SDK, and now it only shows the last java stack frame on the report. The task uploadCrashlyticsSymbolFileFlavorDebug gets called right after externalNativeBuildFlavorDebug, but no symbols on the reports. I have tried running it by cli after the build, but still no luck. I'm out of ideas and can't find where did I go wrong

Project-Level build.gradle

buildscript {
    repositories {
        jcenter()
        google()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:3.3.2'
        classpath 'com.google.gms:google-services:4.3.3'
        classpath 'com.google.firebase:perf-plugin:1.3.1'
        classpath 'com.google.firebase:firebase-crashlytics-gradle:2.0.0-beta02'
    }
}

App-Level build.gradle

apply plugin: 'com.android.application'
apply plugin: 'com.google.gms.google-services'
apply plugin: 'com.google.firebase.crashlytics'
apply plugin: 'com.google.firebase.firebase-perf'

android {
    buildTypes {
      release {
         debuggable false
         firebaseCrashlytics {
             nativeSymbolUploadEnabled true
         }
      }
      debug {
         debuggable true
         firebaseCrashlytics {
             nativeSymbolUploadEnabled true
         }
      }
    }
  }

dependencies {
    implementation 'com.google.firebase:firebase-analytics:17.2.3'
    implementation 'com.google.firebase:firebase-crashlytics:17.0.0-beta01'
    implementation 'com.google.firebase:firebase-crashlytics-ndk:17.0.0-beta01'
}
afterEvaluate {
  android.applicationVariants.all { variant ->
    tasks."externalNativeBuild${variant.name.capitalize()}".finalizedBy "uploadCrashlyticsSymbolFile${variant.name.capitalize()}"
  }
}

I've also removed the following line from onCreate

Fabric.with(this.spawner, new Crashlytics(), new CrashlyticsNdk());

EDIT 1 I've checked the debug log to see if everything was working as intended, and it seems to be:

Symbols get generated

14:24:44.950 [DEBUG] [com.google.firebase.crashlytics] Generating symbols for <BUILD_PATH>\intermediates\ndkBuild\appName\debug\obj\local\armeabi-v7a\libmain.so
14:24:44.950 [DEBUG] [com.google.firebase.crashlytics] Using DWARF data for cSYM generation.
14:25:14.992 [DEBUG] [com.google.firebase.crashlytics] Generating symbols for <BUILD_PATH>\intermediates\ndkBuild\appName\debug\obj\local\armeabi-v7a\libSDL2.so
14:25:14.992 [DEBUG] [com.google.firebase.crashlytics] Using DWARF data for cSYM generation.

AppId & Crashlytics Org Id are found

14:25:15.109 [DEBUG] [com.google.firebase.crashlytics] Getting appId from output of the Google Services plugin at <BUILD_PATH>\generated\res\google-services\appName\debug\values\values.xml
14:25:15.139 [DEBUG] [com.google.firebase.crashlytics] Found Google appId: #:############:android:xxxxxxxxxxxxxxxx
14:25:15.140 [DEBUG] [com.google.firebase.crashlytics] Uploading native symbol files from directory: <BUILD_PATH>\crashlytics\AppNameDebug\nativeSymbols
14:25:15.144 [DEBUG] [com.google.firebase.crashlytics] Requesting Legacy Crashlytics settings for #:############:android:xxxxxxxxxxxxxxxx from: http://firebase-settings.crashlytics.com/spi/v2/gmp/#:############:android:xxxxxxxxxxxxxxxx/fabric_ids
14:25:15.144 [DEBUG] [com.google.firebase.crashlytics] Request Headers:
14:25:15.144 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-DEVELOPER-TOKEN : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
14:25:15.144 [DEBUG] [com.google.firebase.crashlytics]  User-Agent : crashlytics-gradle/2.0.0-beta02
14:25:15.144 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-API-CLIENT-TYPE : crashlytics-gradle
14:25:15.144 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-API-CLIENT-VERSION : 2.0.0-beta02
14:25:15.144 [DEBUG] [com.google.firebase.crashlytics]  Accept : application/json
14:25:15.147 [DEBUG] [com.google.firebase.crashlytics] REQUEST: http://firebase-settings.crashlytics.com/spi/v2/gmp/#:############:android:xxxxxxxxxxxxxxxx/fabric_ids
14:25:15.469 [DEBUG] [com.google.firebase.crashlytics] RESPONSE: 200 [reqId=null]
14:25:15.469 [DEBUG] [com.google.firebase.crashlytics] Crashlytics settings response: Optional.of({"fabric_org_external_id":"xxxxxxxxxxxxxxxxxxxxxxxx","collector_endpoint":1,"fabric_app_external_id":"xxxxxxxxxxxxxxxxxxxxxxxx","fabric_bundle_id":"package.name"})
14:25:15.469 [DEBUG] [com.google.firebase.crashlytics] Using fetched Crashlytics Org Id: xxxxxxxxxxxxxxxxxxxxxxxx for #:############:android:xxxxxxxxxxxxxxxx

Main lib symbols get uploaded

14:25:23.677 [DEBUG] [com.google.firebase.crashlytics] POST file: <BUILD_PATH>\crashlytics\AppNameDebug\nativeSymbols\main-armv7-084400b7e721bf5dd66c8c5877626739802bc470.cSYM.gz to URL: https://cm.crashlytics.com/api/v3/platforms/android/code_mappings
14:25:23.677 [DEBUG] [com.google.firebase.crashlytics] POST params:
14:25:23.677 [DEBUG] [com.google.firebase.crashlytics]  project[identifier] = package.name
14:25:23.677 [DEBUG] [com.google.firebase.crashlytics]  code_mapping[executables][][arch] = armv7
14:25:23.677 [DEBUG] [com.google.firebase.crashlytics]  code_mapping[type] = csym
14:25:23.677 [DEBUG] [com.google.firebase.crashlytics]  code_mapping[executables][][identifier] = 084400b7e721bf5dd66c8c5877626739802bc470
14:25:23.678 [DEBUG] [com.google.firebase.crashlytics] POST headers:
14:25:23.678 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-ORG-ID = xxxxxxxxxxxxxxxxxxxxxxxx
14:25:23.678 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-DEVELOPER-TOKEN = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
14:25:23.678 [DEBUG] [com.google.firebase.crashlytics]  User-Agent = crashlytics-gradle/2.0.0-beta02
14:25:23.678 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-API-CLIENT-TYPE = crashlytics-gradle
14:25:23.678 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-API-CLIENT-VERSION = 2.0.0-beta02
14:25:27.780 [DEBUG] [com.google.firebase.crashlytics] POST response: [reqId=671299695985d1282834f3b17cc7375966ed349acfb42410] 202
14:25:27.781 [DEBUG] [com.google.firebase.crashlytics] Crashlytics symbol file uploaded successfully; deleting local csym: <BUILD_PATH>\crashlytics\AppNameDebug\nativeSymbols\main-armv7-084400b7e721bf5dd66c8c5877626739802bc470.cSYM

SDL lib symbols get uploaded

14:25:29.643 [DEBUG] [com.google.firebase.crashlytics] POST file: <BUILD_PATH>\crashlytics\AppNameDebug\nativeSymbols\SDL2-armv7-7d36271372e5e78fdb06018f776f944d0761b6f9.cSYM.gz to URL: https://cm.crashlytics.com/api/v3/platforms/android/code_mappings
14:25:29.643 [DEBUG] [com.google.firebase.crashlytics] POST params:
14:25:29.643 [DEBUG] [com.google.firebase.crashlytics]  project[identifier] = package.name
14:25:29.643 [DEBUG] [com.google.firebase.crashlytics]  code_mapping[executables][][arch] = armv7
14:25:29.643 [DEBUG] [com.google.firebase.crashlytics]  code_mapping[type] = csym
14:25:29.643 [DEBUG] [com.google.firebase.crashlytics]  code_mapping[executables][][identifier] = 7d36271372e5e78fdb06018f776f944d0761b6f9
14:25:29.644 [DEBUG] [com.google.firebase.crashlytics] POST headers:
14:25:29.644 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-ORG-ID = xxxxxxxxxxxxxxxxxxxxxxxx
14:25:29.644 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-DEVELOPER-TOKEN = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
14:25:29.644 [DEBUG] [com.google.firebase.crashlytics]  User-Agent = crashlytics-gradle/2.0.0-beta02
14:25:29.644 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-API-CLIENT-TYPE = crashlytics-gradle
14:25:29.644 [DEBUG] [com.google.firebase.crashlytics]  X-CRASHLYTICS-API-CLIENT-VERSION = 2.0.0-beta02
14:25:31.199 [DEBUG] [com.google.firebase.crashlytics] POST response: [reqId=ed1547538ba4ccc96747c9b2c4d3900fcb604606713e293f] 202
14:25:31.199 [DEBUG] [com.google.firebase.crashlytics] Crashlytics symbol file uploaded successfully; deleting local csym: <BUILD_PATH>\crashlytics\AppNameDebug\nativeSymbols\SDL2-armv7-7d36271372e5e78fdb06018f776f944d0761b6f9.cSYM

The only strange thing I found was this early line

14:18:56.974 [DEBUG] [com.google.firebase.crashlytics] Mapping File Upload Enabled: false; id: 00000000000000000000000000000000

... but is quickly followed by this one, so i don't think it matters

14:18:57.013 [DEBUG] [com.google.firebase.crashlytics] Crashlytics native symbol uploading enabled: true

Ringtail

I am having the same problem, and I am in a process of investigating this with firebase helpdesk. Actually we were having completely wrong NDK callstack since a year or two, even with fabric.io, thats why we tried with beta firebase crashlytics. We will give it a try for a few more weeks and try switching to backtrace.io . – Dorpat

@Berinhardt, any updates on this? I'm running into a similar issue and I'm curious if you were able to resolve this. Thanks! – Thai

Answer 1

SOLVED

Making transformNativeLibsWithStripDebugSymbolForRelease task being finalizedBy uploadCrashlyticsSymbolFileRelease solved the issue

Ringtail

Where did you exactly set transformNativeLibsWithStripDebugSymbolForRelease? – Warthog

Answer 2

Firebaser/Crashlytics team here: To see debugging output of the Firebase Crashlytics NDK SDK I recommend you try the following:

./gradlew app:yourCustomTask --console=plain --debug | grep "\[com.google.firebase.crashlytics\]"

This will show some more output that might help you self-resolve.

Your setup looks correct so far, but it's possible your symbol files were never generated and uploaded to Crashlytics in the first place. I recommend checking out this Firebase documentation on how to provide paths to your stripped and unstripped native libraries: https://firebase.google.com/docs/crashlytics/ndk-reports-new-sdk. Here's the example from the doc:

firebaseCrashlytics {
                nativeSymbolUploadEnabled true
                strippedNativeLibsDir ‘path/to/stripped/parent/dir’
                unstrippedNativeLibsDir ‘path/to/unstripped/parent/dir’
            }

If you're still having trouble feel free to submit a case to Firebase support with the logs from above and all relevant Firebase + Crashlytics setup information for deeper investigation.

Achilles

I've checked the debug logs as you suggested but found nothing out of the ordinary there – Ringtail

Thanks for posting the console command! It helped me to find strippedNativeLibsDir and unstrippedNativeLibsDir for the native code compiled in a library project. – Hearn

Hello, the documentation link above gives me a 404. – Iene

Your solution doesn't work. Are you sure you work in Crashlytics?! If even you guys don't understand how it's working, then who does?! – Warthog