MD5-SHA1 hash using CryptoAPI

Question

I have requirement of signing a data using MD5-SHA1 hash (MD5 hash and SHA1 hash of the data are combined and then signed).

Primary requirement:

MD5-SHA1 hash is provided by OpenSSL in some way that is not exposed. This hash is complete.

Now my requirement is to sign this hash using Crypto API without hashing (only signing is required, not hashing again). Why CryptoAPI, not OpenSSL? Because, I am dealing with a certificate with non-exportable private key. Its private key can only be used by CryptoAPI, not by OpenSSL.

This scenario is from SSL handshake where send Client verify part (under the case of non-exportable private key) to my layer and is trying to sign the data using CryptoAPI.

Also, I would like to know about other ways of making the use of non-exportable private key certificates using OpenSSL.

Sestertium

If the requirement is defined by you, please avoid using MD5, en.wikipedia.org/wiki/MD5#Security – Serrated

This requirement is not defined by me. I know MD5 is not secure. But in this case, it is to be used along with SHA1 and will be signed. This is be standard. See the section 7.4.3 in ietf.org/rfc/rfc2246.txt – Sestertium

Answer 1

By using CALG_SSL3_SHAMD5 in CryptCreateHash and after that calling CryptSetHashParam and then calling the CryptSignHash did the job for me.

Let me detailed out the requirement: 1. Generally, I gave private key using EVP_PKEY to OpenSSL function so that it can do the signing part. I used to extract the private key from Windows Certificate store (when key is marked as exportable). But in one scenario, private key is not marked as exportable and I was not able to get private key to EVP_PKEY. So, I did modification to OpenSSL code on the case when private key is not available, then the signing part is implemented using CryptoAPI (since CryptoAPI allows the signing using such non-exportable key).

So, in my case, data is completely hashed by OpenSSL (using md5-sha1 hash). Only thing left for me is to do signing.

So, following steps (under signing part) did the job:

1. Created hash using Certificate key handle provider and use CALG_SSL3_SHAMD5 algorithm.
2. Using CryptSetHashParam, I set the initial hash value
3. Signed the hash using CryptSignHash.
4. After that, reverse the bytes order (as OpenSSL signature is Big endian mentioned [here][1] which I found after lots of experimenting).

Rest of code of OpenSSL seems happy with this signature.

Sestertium