OpenSSL, decrypting with a private key

Question

Okay, so I have a text file named Kryptert that is encrypted. A key file named private with the private key. I want the output to be in a text file named Klartext.

I am about to rip my hair out, because I cannot seem to figure this out.

openssl rsautl -decrypt -inkey C:\private.key -in C:\Kryptert.txt -out C:\Klartext.txt

The command above is what I use, and I get the following output in the CMD windows:

C:\Users\Marco>openssl rsautl -decrypt -inkey C:\private.key -in C:\Kryptert.txt -out C:\Klartext.txt
Loading 'screen' into random state - done
RSA operation error
8560:error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02:.\crypto\rsa\rsa_pk1.c:190:
8560:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:.\crypto\rsa\rsa_eay.c:592:

Anyone able to help me understand what is wrong, and how I could fix it? Thank you.

Felicio

What command was used to encrypt the file? – Relativity

Answer 1

Here you have the commands you need to encrypt or decrypt using openssl:

Decrypt:

$ openssl rsautl -decrypt -in $ENCRYPTED -out $PLAINTEXT -inkey keys/privkey.pem

Encrypt:

$ openssl rsautl -encrypt -in $PLAINTEXT -out $PLAINTEXT.encrypt -pubin -inkey keys/pubkey.pem

Hope this helps! :)

Kippy

It would appear (from the error message) that the format of the input is wrong, i.e. has the wrong padding. – Amundsen

Answer 2

For encryption:

openssl rsautl -encrypt -in /path/to/your/file -out /path/to/your/encrypted -pubin -inkey /path/to/your/public_key.pem

For decryption:

openssl rsautl -decrypt -in /path/to/your/encrypted -out /path/where/you/want/your/decrypted.txt -inkey /path/to/your/private_key.pem

Note: If you have this decryption error: RSA_EAY_PRIVATE_DECRYPT:data greater than mod len try this command before decrypt your file:

cat yourEncryptedFile| base64 -D > yourEncryptedRawFile

More information here

Rhomb

You have to use the base64-command with a lowercase -d, uppercase -D is invalid – Grindery

Answer 3

For versions 3.0 and above rsautil is deprecated. Use pkeyutl instead. The usage stays the same for this example, i.e. (on Windows):

.\openssl.exe pkeyutl -decrypt -in .\encryptedfile -out decryptedfile -inkey .\private-key.pem

Side note: I was playing around with TLS and wanted to decrypt the premaster key sent by the client in a TLS 1.0 (yes, TLS 1.0) handshake. In case someone tries the same: Copy the premaster key (which will be in a hexadecimal representation) into a text file, save it, and then convert the text file to a binary file using certutil. The output can then be used with openssl.

Example:

1. Convert premaster key's hexadecimal representation to binary:

certutil -decodehex -f .\premasterkey.txt premasterkey.bin

2. Decrypt using openssl

.\openssl.exe pkeyutl -decrypt -in .\premasterkey.bin -out decrypted.bin -inkey .\private-key.pem

3. (Optional) Convert the resulting binary into a hexadecimal representation again for viewing.

certutil -encodehex -f decrypted.bin decryptedinhex.txt

Sixpence