Google Cloud Endpoints: verifyToken: Signature length not correct

Question

This morning the following exception has started occuring on every API request to my Google Cloud Endpoint from my Android app:

com.google.api.server.spi.auth.GoogleIdTokenUtils verifyToken: verifyToken: Signature length not correct: got 256 but was expecting 128

The call still works perfectly from my javascript web clients. I have changed nothing on the server side code or client code.

Has anything changed with the service recently that might make this occur?

UPDATE: The first occurrence of this appears to have been at 11:17:07 UTC

UPDATE: Things that don't work include generating a new Client ID for android & updating to App Engine SDK 1.9.22

Chute

so it's not just me... looking into it – Friedcake

This just happened with me... was working fine 3 hours ago. – Pterosaur

We are not alone The link to the official google community for downtime issues. groups.google.com/forum/#!topic/google-appengine/B5RmtfLS6HM – Pterosaur

I got the same issue. This problem is causing thousands of android users not to be able to access our system. – Paracasein

We are having the same issue if someone gets a solution – Rosenquist

Thank God i am not alone :) happened to me after lunch break :( – Joashus

We contacted Google Enterprise Support team. They told they are already working to fix this issue. – Paracasein

Thanks @Paracasein - I've been trying to work out a way of getting in touch with them! Can you keep us updated please? – Chute

They told they will provide further update at 10:00 US/Pacific. As soon as I receive any response back from them I post here. – Paracasein

Any updates? The only thing I can find is code.google.com/status/appengine which seems to show some problem with the java and python gae. – Rosenquist

The issue should only affect the mixed-key-length key combinations. The current sets of keys being served only include 2048-length keys. You may still be experiencing outage due to cached key sets, but likely if you re-start your application the issue will have been resolved. – Bethelbethena

Thanks seems to be good now! That as a nice 7 hour panic :) Thanks everyone for updates – Rosenquist

@Bethelbethena restarting changes nothing to me, still the same issue... – Plott

@Bethelbethena I still have the same problem as well – Chute

Hi all, please let us know if you are still dealing with issues. – Aldredge

It all seems to be working for me now, thanks sorting it out! @Aldredge Could someone post something that I can mark as the correct answer for this? – Chute

Answer 1

The causes

• RSA has variable length signatures, depending on the key size.
• Google updated the key pairs it uses for signing, and now one of the key pairs generates a different length signature from the other
• java.security.Signature.verify(byte[] signature) throws an exception if a signature of the wrong length is passed (instead of returning false which is normally done when a signature does not match the key)

For me the solution was to wrap the verify call (try...catch), and return false instead. You could also do an early check on the public key yourself, checking if the length of the signature matches the length of the public key modulus.

If you use a library to check the signature, make sure you use the latest version.

Looking at the example code on http://android-developers.blogspot.nl/2013/01/verifying-back-end-calls-from-android.html, you would have to change this:

GoogleIdToken token = GoogleIdToken.parse(mJFactory, tokenString);

to

JsonWebSignature jws = JsonWebSignature.parser(mJFactory).setPayloadClass(Payload.class).parse(tokenString);
GoogleIdToken token = new GoogleIdToken(jws.getHeader(), (Payload) jws.getPayload(), jws.getSignatureBytes(), jws.getSignedContentBytes()) {
   public boolean verify(GoogleIdTokenVerifier verifier)
  throws GeneralSecurityException, IOException {
       try {
           return verifier.verify(this);
       } catch (java.security.SignatureException e) {
           return false;
       }
   }
};

I unfortunately don't have an exact setup to test this.

For those using Google Cloud Endpoint, like the question states, I think there was very little you could do except wait until Google fixes it. Luckily it's fixed now. (Technically, you could argue changing the keys, as is done now, is a workaround, and the library Google provides needs to be fixed. But it works, so that's a good start)

Friedcake

Thanks for the info. Any idea where to start trying to fix it? – Chute

@SanketBerde Google generates the keys, and then they just sign the message... if Google generates new keys, maybe it will help, but for us that's no solution – Friedcake

This is definitely not a problem from our end right ? Did we do something wrong with the config ? – Pterosaur

@Friedcake where did you wrap the verify call? I'm using Endpoints annotations to set up my API so I think that call may be hidden away somewhere in the app engine sdk – Chute

@Tom, I don't use the app engine sdk, so that makes it a bit different... Do you have some control over which public key is used to verify the signature? – Friedcake

@Friedcake Do you mean on the server side or android side? This is a play store deployed application so I can't quickly change anything on the client side as it will take hours to push through to users. – Chute

@Tom, on the server side – Friedcake

@Friedcake Yes, I can change the client id for android applications. I'm going to try generating a new client Id and pushing a new version of the server. – Chute

@Tom, the client id is not related to the public key... The public key is provided by Google. But pushing a new version of the server without changing anything might solve it simply by updating the Google Libraries. – Friedcake

I Tried generating new Android client ID and Audience. Didn't work, Same error. What i've noticed is that the Endpoints calls made from the Google Endpoints explorer work. So this is a problem with the Client ids only. – Pterosaur

@SanketBerde Same here, also my javascript clients are still working. – Chute

Do we know any big products that work on endpoints ? if they are failing too, we are experiencing a once-in-a-lifetime phenomena. – Pterosaur

@Tom, does your server side contain code from the example I just added? Could you test my suggestion? – Friedcake

@Friedcake Sorry, I can't verify that works. My API methods are all automagically created by the SDK - I only declare them with the API annotation and add authentication as described here cloud.google.com/appengine/docs/java/endpoints/auth – Chute

The verification is supposed to happen automatically behind the scenes. thats what they say when they advertise endpoints. – Pterosaur

I tired this solution but In my case return verifier.verify(this); is never called.. – Joashus

I tested the java code posted to replace the creation of the GoogleIdToken object. Unfortunately, I get the exact same exception as before!! – Plott

The issue should only affect the mixed-key-length key combinations. The current sets of keys being served only include 2048-length keys. You may still be experiencing outage due to cached key sets, but likely if you re-start your application the issue will have been resolved. – Bethelbethena

Answer 2

Same problem here, as far as I can tell the public cert URL (now? I guess this wasn't the case before or the order changed) returns two keys:

https://www.googleapis.com/oauth2/v1/certs

inspecting those, the first has a 1024 bit key and the second a 2048 bit key. I believe my incoming tokens from android clients were signed by the second cert with the 2048 bit key, hence the "Signature length not correct: got 256 but was expecting 128".

Looking at the Google verifier source (GoogleTokenVerifier.java) it does appear to iterate multiple keys:

// verify signature
for (PublicKey publicKey : publicKeys.getPublicKeys()) {
  if (googleIdToken.verifySignature(publicKey)) {
    return true;
  }
}

assuming the keys got parsed correctly (that code looks reasonable but haven't actually inspected the results).

As beestra pointed out the, this code expects false to be returned in case it could not be verified but instead it's throwing an exception. Ideally it should keep iterating after a failure and use the second public key to verify, which should work.

To fix this there appears to be two options:

1. Fork the google api client library and fix
2. Duplicate the verification (GoogleIdTokenVerifier.verify(GoogleIdToken)) in (your) calling code

I don't know how realistic 2. is, some super functionality is used and a lot of internal state is private, would have to duplicate all of it. Busy investigating...

UPDATE: Ok, looks to be fixed in my tests using production data, though haven't deployed it to production just yet. Here's the Scala

  val jsonFactory = new JacksonFactory()
  val transport = new NetHttpTransport()
  val googleIdTokenVerifier = new GoogleIdTokenVerifier(transport, jsonFactory)

  class DuplicateVerifier(builder: GoogleIdTokenVerifier.Builder) extends IdTokenVerifier(builder)
  val topIdTokenVerifier = new DuplicateVerifier(new GoogleIdTokenVerifier.Builder(transport, jsonFactory))
  val publicKeysManager = new GooglePublicKeysManager(transport, jsonFactory)
  def duplicateGoogleVerify(token: GoogleIdToken): Boolean = {
    // check the payload
    if (!topIdTokenVerifier.verify(token)) {
      false
    } else {
      // verify signature
      import scala.collection.JavaConverters._
      publicKeysManager.getPublicKeys.asScala.map { k =>
        Try(token.verifySignature(k))
      }.foldLeft(false)((c, x) => c || x.getOrElse(false))
    }
  }

Just to be clear if it isn't obvious, using this method instead of Google's:

// if (googleIdTokenVerifier.verify(token)) {
if (duplicateGoogleVerify(token)) {

I'll try to write the Java equivalent later if anyone needs it.

Prison

I think you're right about the public cert URL - as of March 15th 2015 it had 2 certificates of the same size web.archive.org/web/20150315022911/https://www.googleapis.com/… – Chute

please provide JAVA equivalent if it is successful.. i need it!! – Joashus