RSA_public_decrypt and MS Crypto API equivalent
Asked Answered
S

3

9

I'm trying to develop a license verification solution. Licenses are encoded on server using OpenSSL's RSA_private_encrypt function.

For Mac OX X I use RSA_public_decrypt and it works like a charm. On Windows I must use very tiny bit of code, so I can not link with OpenSSL or other lib AND I have to use MS Crypto API.

I have spent several days trying to figure out what is wrong, but with no luck. I can successfully import public key, but here my success ends. I'm aware that I need to reverse byte order with CAPI so this might not be the issue.

I have tried everything, including CryptVerifyMessageSignatureWithKey and CryptDecodeObject to load the blob with different params, but still no luck.

It always ends up with GetLastError() == CRYPT_E_ASN1_BADTAG, which I assume means that the BLOB is not ASN1 formatted... Google does not tell anything on the output format of RSA_private_encrypt... so I am completely lost here.

Here is the OS X code based on OpenSSL:

void cr_license_init(const char* lic) {
    __cr_license_ = lic;
    unsigned char lic_encoded[CR_LIC_LEN];

    BIO* b64 = BIO_new(BIO_f_base64());
    BIO* licIn = BIO_new_mem_buf((void*)lic, -1);
    licIn = BIO_push(b64, licIn);

    if(BIO_read(licIn, lic_encoded, CR_LIC_LEN) == CR_LIC_LEN) {

        const unsigned char* key_data = license_pub_der;
        RSA* r = d2i_RSA_PUBKEY(NULL, &key_data, sizeof(license_pub_der));

        if(r != NULL) {
            if(__cr_license_data_ != NULL) {
                free((void*)__cr_license_data_);
            }
            __cr_license_data_ = malloc(CR_LIC_LEN);
            if(RSA_public_decrypt(CR_LIC_LEN, lic_encoded,
    (unsigned char*)__cr_license_data_, r, RSA_PKCS1_PADDING) &lt= 0) {
                free((void*)__cr_license_data_);
                __cr_license_data_ = NULL;
            }
            RSA_free(r);
        }
    }
    BIO_free_all(licIn);
}

This part of code on windows works well, so I assume public key is not an issue.

__cr_license_ = lic;
unsigned char lic_encoded[CR_LIC_LEN];

DWORD dwSize;
if(CryptStringToBinaryA(__cr_license_, 0/*autocalculate*/, CRYPT_STRING_BASE64, lic_encoded, &dwSize, NULL, NULL) && dwSize == CR_LIC_LEN) {
HCRYPTPROV hProv;
if(CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
    PCERT_PUBLIC_KEY_INFO pki = NULL;
    DWORD dwKeySize;
    if(CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, license_pub_der, sizeof(license_pub_der), CRYPT_ENCODE_ALLOC_FLAG, NULL, &pki, &dwKeySize)) {
        HCRYPTKEY hKey = 0;
        if(CryptImportPublicKeyInfo( hProv, X509_ASN_ENCODING, pki, &hKey)) {

But after that anything I try to do with message leads to CRYPT_E_ASN1_BADTAG. I tried CryptMsgOpenToDecode with CryptMsgUpdate, CryptDecodeObject, CryptVerifyMessageSignatureWithKey - nothing works.

Basically I think that the problem is in pkcs1 and pkcs7 incompatibility as owlstead mentioned. Does anyone has experience working with pkcs1 format importing/converting/etc with MS CAPI?

Any help or even a clue is appreciated a lot! Thanks in advance!

Shortbread answered 25/1, 2013 at 18:15 Comment(18)
You might want to include some code in the question.Gagliano
also show header of keys and encoded files. That error is for invalid ASN tag headerBudwig
Does this help? https://mcmap.net/q/1318954/-how-to-use-capi-39-s-cryptimportkey-with-pem-encode-public-key-from-openssl/645583Pietra
Yeah, I have seen this thread. The problem in this case is not the key. I can successfully import the key with CryptImportPublicKeyInfo, so I assume the key is not the case. It seems that the problem is with encoded BLOB. I believe that the BLOB is not ASN1 compatible and should be converted somehow.Shortbread
Did you check the comment below this article? msdn.microsoft.com/en-us/library/windows/desktop/…Mutineer
Thanks @owlstead, I have seen this comment of course. I doubt this is the case, because I can import the public key into the context with CryptImportPublicKeyInfo successfully.Shortbread
@Daniel I was hoping for the Windows code which threw the CRYPT_E_ASN1_BADTAG error you are refering to.Gagliano
@DanielRoethlisberger, sorry I have no Windows code at home, but I will definitely add some tomorrow at office.Shortbread
Daniel, would you know the padding mechanism used by CryptVerifyMessageSignatureWithKey? For some reason it is not specified by the API (not that I'm surprised, Microsoft regularly underspecifies)Mutineer
@owlstead, Microsoft tells you can choose one: X509_ASN_ENCODING | PKCS_7_ASN_ENCODING. So BLOB certainly must be ASN1 formatted and seems that it works with PKCS padding.Shortbread
PKCS#7 or CMS is a high level cryptographic message syntax. OpenSSL code above seems to use a low level PKCS#1 signature. This signature format can be part of PKCS#7 structure, but is certainly not the same. PKCS stands for public-key cryptography standards and requires a number such as #1 #7 to point to a specific standard (and this should hopefully be followed by a version and the name of the specific message syntax or algorithm). PKCS in itself doesn't define anything. This might answer your question. Note that CMS is specified using ASN.1 BER notation....Mutineer
X509_ASN_ENCODING is for certificates by the way, they are also specified by ASN.1 BER notation. Certificates are signed in much the same way as CMS, basically they are a container format that is signed, again, the signature itself is likely to be PKCS#1 (using the v1.5 signature format).Mutineer
@owlstead, OK, got it, thanks for so detailed explanation! Any suggestions how could PKCS1 be converted to PKCS7?Shortbread
Conversion may not be possible. PKCS#7 may use a PKCS#1 signature format, but it is normally not directly calculated over the data itself. Either let the server create PKCS7 (it is OpenSSL functionality) or try and find a function that will verify PKCS#1 signatures.Mutineer
@DanielRoethlisberger, I posted the windows code part.Shortbread
@Daniel: Two question. I saw that you was asking this 3 month ago and just created a bounty. Any progress on this. Can you please post public, private key and encrypted license (for testing). So, I can take a look at it and try to write a code which will decrypt it on Windows.Labana
@Daniel: One more question. Where this requirement of tiny code come from? Is this for a driver or something like it?Labana
Also see OpenSSL and MS CryptoAPI: different digital signatures.Jammie
M
4

You are mixing higher and lower level signature formats. OpenSSL asumes PKCS#1 v1.5 signatures by default, which contains of only the signature data. Windows seems to asume PKCS#7 containers. These may contain a PKCS#1 v1.5, but those and other data are wrapped using ASN.1 BER tag/length format. If the Microsoft API tries to decode this it will assume that the raw signature is the container format, and decoding will fail.

Mutineer answered 25/1, 2013 at 23:28 Comment(3)
Is it possible to add higher level encoding (PKCS7) to the signature? I mean may be something like CryptEncodeObject?Shortbread
See my latest comment on your questionMutineer
thank you very much for the clarification! Will continue the research. Anyone is aware of possible solution?Shortbread
G
1

Unless this is so obvious that you've tried but omitted listing it or I misunderstand your question otherwise, I think you should be using CryptDecrypt to decrypt the license, not the functions you mention in the question. Note that since you seem to be using OpenSSL with PKCS#1 v1.5 padding and CryptoAPI does not seem to support that (haven't tested, but specs only list PKCS#1 v2 OAEP), you will probably have to use CRYPT_DECRYPT_RSA_NO_PADDING_CHECK and verify and remove the PKCS#1 v1.5 padding manually after decryption.

Gagliano answered 26/1, 2013 at 14:59 Comment(5)
Isn't CryptDecrypt intended to be used with the private key?Shortbread
As I thought it returns NTE_NO_KEY when used with public key.Shortbread
Have you tried or do you just think it does? I honestly don't know, but all the other functions operate on higher level cryptographical containers, not raw RSA blocks. If it does not work, you might either be able to trick it into believing the public key is a private key by flipping some bit (not sure how the public key is marked as such), or trick it using CryptEncrypt without any padding (not specifying CRYPT_OAEP), since if it does not do any padding, encrypt and decrypt is the equivalent operation.Gagliano
Yeah, I just tried. It does not work. I'm not sure if it is possible (and how) to trick it to believe that it's a private key... Basically I think this might not be possible, because encrypting with private key (also known as "signing") should differ a lot from encrypting with public key.Shortbread
The encrypt and decrypt operations only differ in adding the padding / verifying&removing padding. On the raw RSA level, the operations are identical, just with a different key. Also, the raw signing/verifying operations only differ from encryption/decryption in the different padding, the actual RSA operations on the bit level are identical.Gagliano
M
0

OpenSSL exports keys with extra header which is not expected by CryptoAPI.

Header for private key (in ASN.1 notation):

Offset| Len  |LenByte|
======+======+=======+======================================================================
     0|   630|      3| SEQUENCE : 
     4|     1|      1|    INTEGER : 0
     7|    13|      1|    SEQUENCE : 
     9|     9|      1|       OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1]
    20|     0|      1|       NULL : 
    22|   608|      3|    OCTET STRING : 
                             ... actual key data go here ...

Header for public key (in ASN.1 notation):

Offset| Len  |LenByte|
======+======+=======+======================================================================
     0|   159|      2| SEQUENCE : 
     3|    13|      1|    SEQUENCE : 
     5|     9|      1|       OBJECT IDENTIFIER : rsaEncryption [1.2.840.113549.1.1.1]
    16|     0|      1|       NULL : 
    18|   141|      2|    BIT STRING UnusedBits:0 : 
                              ... actual key data go here ...

These headers are what causes CryptDecodeObjectEx to choke. It expects RAW key data, without any header.

So, basically, you need:

  1. (Optional) Convert .PEM to .DER with CryptStringToBinary.
  2. Check if DER starts with the above mentioned headers. For that you need to read ASN.1-encoded data.
  3. (Optional) Skip the above mentioned header and seek directly to key's data (starts with SEQUENCE which includes 2 INTEGER for public key or 9 INTEGER for private key).
  4. Feed the result to CryptDecodeObjectEx(X509_ASN_ENCODING, RSA_CSP_PUBLICKEYBLOB/PKCS_RSA_PRIVATE_KEY).
  5. Import keys with CryptImportKey.
Materials answered 4/12, 2015 at 16:43 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.