How to encrypt a string using OpenSSL C library and a public key file?

Question

What is the recommended way of encrypting a short std::string into another std::string using the openssl C library (not the command-line tool of the same name) using a public keyfile, and knowing the algorithm? (in this case the string is no larger than ~100 bytes, keyfile is in .pem format, algorithm can be any asymmetric one like RSA/ECDSA/etc).

I am looking to write a function with an interface like so: bool EncryptString(const std::string& InStr, const std::string& InPublicKey, std::string& OutString).

Looking through documentation it seems like EVP_PKEY_encrypt is the function to use, which requires an EVP_PKEY_CTX *ctx variable. My assumption is, this variable should be initialized with EVP_PKEY_CTX_new, which in turn requires an EVP_PKEY *pkey and an ENGINE *e. Those I don't know how to initialize, and searching through the documentation leaves me very confused.

Maybe these functions are not the easiest approach, I am not familiar with this library at all and have no cryptography knowledge. I simply care about a black-box way of converting a string to an encrypted string.

Thank you

Manifold

X25519 is not used for encryption in the classical sense. It is a key agreement (based on ec curve 25519) with which a shared secret can be generated on both sides using an insecure channel. This shared secret can then be used to derive a symmetric key for the actual (symmetric) encryption (e.g. AES). – Scandian

Thank you for the explanation @Topaco! As I mentioned I have no in depth knowledge of cryptography. Which algorithm should I use instead? I was thinking to use either RSA or ECDSA. In any case the specific algorithm isn’t so relevant, what matters is that a string should get encrypted with a given key file. – Manifold

If you have a PEM key, we are talking about asymmetric algorithm, e.g. RSA (btw, if you already have a PEM key, the algorithm is already defined by it). With RSA you can encrypt only small data. For large data you would have to use an additional symmetric algorithm like AES (also because of the poor performance of asymmetric algorithms), s. hybrid encryption. – Scandian

In hybrid encryption, only the symmetric key is encrypted with RSA (if the key does not need to be exchanged, AES alone can simply be applied). Alternatives to RSA are elliptic curves (ECDH, X25519) for key agreement. Please describe your requirements in more detail. Encrypting a string with a key (from a file) is not a strong decision criterion, since this is practically possible with all these algorithms. – Scandian

@Scandian I see, thank you. The InStr that I need to encrypt is very short ~100bytes, no longer than an AES key, so speed should not be a concern. It can be any short string, including an AES key. Therefore I assume it's appropriate to use RSA/ECDSA on it directly. – Manifold

I assumed that this use case would be very basic, as far as encryption and the openssl library go. I am quite surprised how hard it is to find information on a presumably simple task, in one of the most used encryption libraries. But it might be that my comprehension of the topics involved was too surface-level. – Manifold

ECDSA is not an option as it creates a digital signature and is not for encryption. For RSA, I posted an answer with a code snippet that describes how to use the functions. – Scandian

Answer 1

As it turned out in the comments, RSA is an acceptable option for you.

When implementing RSA with OpenSSL, the following steps are required for encryption:

• Loading the public key
• Creating and initializing the context
• Specifying the padding
• Encryption

The implementation below for encryption with RSA and OpenSSL runs successfully on my machine and shows how the functions are called (without exception handling for simplicity):

#include <openssl/pem.h>
#include <string>
...

bool EncryptString(const std::string& InStr /*plaintext*/, const std::string& InPublicKey /*path to public key pem file*/, std::string& OutString /*ciphertext*/) {
    
    // Load key
    FILE* f = fopen(InPublicKey.c_str(), "r");
    EVP_PKEY* pkey = PEM_read_PUBKEY(f, NULL, NULL, NULL);
    fclose(f);
    
    // Create/initialize context
    EVP_PKEY_CTX* ctx;
    ctx = EVP_PKEY_CTX_new(pkey, NULL);
    EVP_PKEY_encrypt_init(ctx);

    // Specify padding: default is PKCS#1 v1.5
    // EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_OAEP_PADDING); // for OAEP with SHA1 for both digests

    // Encryption
    size_t ciphertextLen;
    EVP_PKEY_encrypt(ctx, NULL, &ciphertextLen, (const unsigned char*)InStr.c_str(), InStr.size());
    unsigned char* ciphertext = (unsigned char*)OPENSSL_malloc(ciphertextLen);
    EVP_PKEY_encrypt(ctx, ciphertext, &ciphertextLen, (const unsigned char*)InStr.c_str(), InStr.size());
    OutString.assign((char*)ciphertext, ciphertextLen);

    // Release memory
    EVP_PKEY_free(pkey);
    EVP_PKEY_CTX_free(ctx);
    OPENSSL_free(ciphertext);

    return true; // add exception/error handling
}

---

With the ciphertext it must be considered that it is a byte sequence that does not represent real text (it may contain 0x00 values, for example). If the ciphertext is to be represented as real text, an additional binary-to-text encoding such as Base64 encoding must be performed.

The code requires a PEM encoded public key in X.509/SPKI format. If the public key has the PKCS#1 format, it can be imported using PEM_read_RSAPublicKey() and EVP_PKEY_set1_RSA().

Scandian

In my previous attempts to make this work I didn't understand how to import the key, because for some reason I didn't find the PEM_read_PUBKEY in the openssl docs. You example makes it very clear how to do this. Thank you for the details and considerations as well. – Manifold

Nice code. Few questions: 1) Is this doing RSA v1.5? 2) You mention SHA1 in the commented-out code. Why is hashing mentioned, I thought we're just encrypting? – Who

@Who - 1) Yes, the code encrypts using RSA with PKCS#1 v1.5 padding by default. 2) The comment you addressed refers to a 2nd padding variant for encryption, namely OAEP, which must be configured via various parameters, including digests. For more details on PKCS#1 v1.5 padding and OAEP, see RFC 8017, 7. Encryption Schemes. – Scandian