InvalidKeyException: Keystore operation failed on RSA decrypt on Android device

Question

I'm trying to perform RSA encryption and decryption on a key that's within AndroidKeyStore. Encryption completes successfully, but when I try to decrypt, it throws an InvalidKeyException: Keystore operation failed on Cipher.init().

Here's my keygeneration code:

KeyPairGenerator kpGenerator = KeyPairGenerator.getInstance("RSA", "AndroidKeyStore");
AlgorithmParameterSpec spec = null;
spec = new KeyGenParameterSpec.Builder(mAlias,
        KeyProperties.PURPOSE_ENCRYPT | KeyProperties.PURPOSE_DECRYPT)
        .setDigests(KeyProperties.DIGEST_SHA256, KeyProperties.DIGEST_SHA512)
        .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_RSA_PKCS1)
        .build();
kpGenerator.initialize(spec);
KeyPair kp = kpGenerator.generateKeyPair();

Here's my encrypt code:

KeyStore keyStore = KeyStore.getInstance("AndroidKeyStore");
keyStore.load(null);
KeyStore.PrivateKeyEntry entry = (KeyStore.PrivateKeyEntry) keyStore.getEntry(mAlias, null);
Cipher cip = null;
RSAPublicKey pubKey = (RSAPublicKey) entry.getCertificate().getPublicKey();
cip = Cipher.getInstance("RSA/ECB/NoPadding");
cip.init(Cipher.ENCRYPT_MODE, pubKey);
byte[] encryptBytes = cip.doFinal(challenge.getBytes());
String encryptedStr64 = Base64.encodeToString(encryptBytes, Base64.DEFAULT);

Here's my decrypt code:

KeyStore keyStore = KeyStore.getInstance("AndroidKeyStore");
keyStore.load(null);
KeyStore.PrivateKeyEntry entry = (KeyStore.PrivateKeyEntry) keyStore.getEntry(mAlias, null);
Cipher cip = null;
cip = Cipher.getInstance("RSA/ECB/NoPadding");
cip.init(Cipher.DECRYPT_MODE, entry.getPrivateKey());
byte[] decryptedBytes = cip.doFinal(Base64.decode(encrypted64, Base64.DEFAULT));
String plainText = new String(decryptedBytes);

I'm aware that the padding in the key generation code is different from my encrypt/decrypt code. However, when I change the padding of my key generation code to KeyProperties.ENCRYPTION_PADDING_NONE, I get an InvalidAlgorithmParameterException on kpGenerator.initialize(spec) instead. With "RSA/ECB/PKCS1Padding" in decrypt, it works. Regardless of padding in encrypt, it always works.

Yes, I'm also aware of the security implications of using no padding, but for my application, I need a deterministic ciphertext.

Here's the stacktrace with InvalidKeyException:

03-06 09:10:32.710  5058  5058 W System.err: java.security.InvalidKeyException: Keystore operation failed
03-06 09:10:32.713  5058  5058 W System.err:    at android.security.KeyStore.getInvalidKeyException(KeyStore.java:692)
03-06 09:10:32.713  5058  5058 W System.err:    at android.security.KeyStore.getInvalidKeyException(KeyStore.java:712)
03-06 09:10:32.713  5058  5058 W System.err:    at android.security.keystore.KeyStoreCryptoOperationUtils.getInvalidKeyExceptionForInit(KeyStoreCryptoOperationUtils.java:54)
03-06 09:10:32.713  5058  5058 W System.err:    at android.security.keystore.KeyStoreCryptoOperationUtils.getExceptionForCipherInit(KeyStoreCryptoOperationUtils.java:89)
03-06 09:10:32.713  5058  5058 W System.err:    at android.security.keystore.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:263)
03-06 09:10:32.713  5058  5058 W System.err:    at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:108)
03-06 09:10:32.713  5058  5058 W System.err:    at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:612)
03-06 09:10:32.713  5058  5058 W System.err:    at javax.crypto.Cipher.tryCombinations(Cipher.java:532)
03-06 09:10:32.714  5058  5058 W System.err:    at javax.crypto.Cipher.getSpi(Cipher.java:437)
03-06 09:10:32.714  5058  5058 W System.err:    at javax.crypto.Cipher.init(Cipher.java:815)
03-06 09:10:32.714  5058  5058 W System.err:    at javax.crypto.Cipher.init(Cipher.java:774)
03-06 09:10:32.714  5058  5058 W System.err:    at dfpidentifiers.my.test.app.MainActivity.decrypt(MainActivity.java:950)
03-06 09:10:32.714  5058  5058 W System.err:    at dfpidentifiers.my.test.app.MainActivity.onCreate(MainActivity.java:117)
03-06 09:10:32.714  5058  5058 W System.err:    at android.app.Activity.performCreate(Activity.java:6251)
03-06 09:10:32.714  5058  5058 W System.err:    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1107)
03-06 09:10:32.714  5058  5058 W System.err:    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2369)
03-06 09:10:32.714  5058  5058 W System.err:    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2476)
03-06 09:10:32.714  5058  5058 W System.err:    at android.app.ActivityThread.-wrap11(ActivityThread.java)
03-06 09:10:32.714  5058  5058 W System.err:    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1344)
03-06 09:10:32.714  5058  5058 W System.err:    at android.os.Handler.dispatchMessage(Handler.java:102)
03-06 09:10:32.714  5058  5058 W System.err:    at android.os.Looper.loop(Looper.java:148)
03-06 09:10:32.714  5058  5058 W System.err:    at android.app.ActivityThread.main(ActivityThread.java:5417)
03-06 09:10:32.714  5058  5058 W System.err:    at java.lang.reflect.Method.invoke(Native Method)
03-06 09:10:32.714  5058  5058 W System.err:    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
03-06 09:10:32.714  5058  5058 W System.err:    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)
03-06 09:10:32.714  5058  5058 W System.err: Caused by: android.security.KeyStoreException: Incompatible padding mode
03-06 09:10:32.714  5058  5058 W System.err:    at android.security.KeyStore.getKeyStoreException(KeyStore.java:632)
03-06 09:10:32.714  5058  5058 W System.err:    ... 24 more

From the stacktrace, it seems to be an incompatible padding mode, but how do I generate a keypair that supports no padding?

Mia

Could you please post the full stacktrace? – Subaudition

Note that deterministic asymmetric cryptography is usually considered a no-go area; an attacker can simply try to encrypt several messages to find one that matches your ciphertext. If you already have enough random bytes in the message then I would considered expanding this to the full modulus size (-1 most significant bit) using e.g. HKDF or a XOF and RSA encrypt this value. Then you can again use HKDF to derive a symmetric key from the value (before encryption / after decryption) and finally use that to encrypt the ciphertext with e.g. AES-GCM. – Subaudition

Maybe you need to pass a call back handler for the password. If this is your problem please allow me to answer the question; without stack trace or key store we cannot verify any solutions. – Subaudition

I've added a stack trace. – Mia

I was getting a similar error. At the end of the day, it had to deal with the size of object I was trying to encrypt and decrypt. When I made the object smaller it worked. – Anthia

Answer 1

The InvalidKeyException was indeed caused by a difference in padding during key generation and decryption. Not sure why encryption didn't throw the same exception though.

I was also initially unable to use NoPadding during key generation as IND-CPA is required. I have to set setRandomizedEncryptionRequired(true) to override this.

Mia

You mean - you had to set setRandomizedEncryptionRequired(false) to use NoPadding, right? – Checkered