CryptoAPI: Using CryptVerifySignature to verify a signature from openssl with public key

Question

I am trying to port the AquaticPrime framework for Mac to Windows.

On the Mac, it uses the opensll library, and I try to understand how to port this to Windows, where I have to use the CryptoAPI, I guess.

I mainly need the code for validation of the generated signature with a given public key.

Here's how verification is done with openssl:

1. inputs: license data, public key and signature, both 128 bytes long.
2. A SHA1 digest is calculated from the license data.
3. A RSA context is set up with the public key data
4. RSA_public_decrypt() is called, given the RSA key and the signature, which returns a 20 byte long SHA1 digest - is this digest equal the one from step 2, the signature is valid.

So, how do I do this with CryptoAPI? I've gotten this far:

1. Start with CryptAcquireContext(ctx, 0, 0, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)
2. Use CryptImportKey with the help of this posting, with pubexp=3 and bitlen=1024. That all works, i.e. I get no errors, and I looked at the binary data to verify that it matches what the MSDN article shows.
3. Create a SHA1 digest from the license data. I've retrieved the resulting 20 byte hash value and see that it matches what I get with openssl on the Mac.

At this point, I call:

CryptVerifySignature (hashHdl, sig, sigLen, keyHdl, 0, 0)

This fails with error code ERROR_INVALID_PARAMETER.

The odd thing is that when I first accidentally had stored a twice as large public key into the PUBLICKEYBLOB structure, I received a NTE_BAD_SIGNATURE error instead. This might suggest that now the public key I am passing is correct.

Why the ERROR_INVALID_PARAMETER error now, then? I've verified that the hash value is correct, and the key appears to be accepted, too. And the "sig" parameter is just a pointer to the 128 bytes of the signature, and sigLen is 128.

So, what am I missing here?

Bott

Question: Why don't you just use some Windows ported version of OpenSSL? I mean there's nothing wrong in using CryptoAPI but openSSL already exists and it would mean less work... – Bickel

I don't want to require people to install new system components just to run an app that doesn't even require an installer otherwise. – Bott

Answer 1

OK, I solved the problem after lots of trial-and-error.

Both the signature and the public key data, when in their pure byte string form, need to be reversed, i.e. first byte to last position, and so on. Then the above works.

Bott

I think I'm running into a similar problem, how did you find out that the signature and public key data needed to be reversed? – Dias

Got it!, the output of the CryptoAPI signing and encryption functions is in little endian format! – Dias

Glad to see you could verify my findings. Then my post here wasn't in vain :) – Bott

On a side note if you RSA encrypt an sha1 hash with NSS and decrypt that with CyrptoAPI you need to reverse just the signed buffer bytes. – Whittington

Documentation for the struct: msdn.microsoft.com/en-us/library/ee442238.aspx – Accession

@ThomasTempelmann I am stuck on the same, after which step do you reverse the strings, i have posted the code here #20776292 – Mam

At what point are you byte-swaping public key, I couldn't get it to work .. see comments at #64149598 – Sulphide

@Sulphide byte-swapping reverses the order of 2 or 4 adjacent bytes. What I mean by reversing is that you make the last byte the new first, the second-to-last the new second, and so on. – Bott

Answer 2

Compile and link the OpenSSL libCrypto statically. It can be done, I've seen this at a former employer.

Erastian