"decrypt error" of TLS 1.2 change-cipher-spec, but reads MAC correctly

Question

I'm trying to bring an old TLS 1.0 implementation (that I did not write) up to date to speak TLS 1.2.

As a first step I integrated the TLS 1.1 change of putting the plaintext initialization vector in the record. That was no problem. It seemed to work well enough that I could read https://example.com in TLS 1.1, as well as SSL Labs viewMyClient.html.

Then I adapted to the TLS 1.2 change of the pseudorandom function to (for most practical purposes) P_SHA256 instead of the (more complex and bizarre) half and half MD5/SHA1 rigamarole. I did it wrong the first time and got an invalid MAC error, but it was more or less a typo on my part and I fixed it. Then the invalid MAC error went away.

But despite that, after sending the ClientKeyExchange->ChangeCipherSpec messages, I'm getting a "Decrypt Error" back from the server(s) (same Alert regardless, https://google.com or anything I try). I gather the ChangeCipherSpec message is encrypting just one byte, putting it into a message with padding and the MAC, etc.

If I tweak the MAC randomly by one byte, it goes back to complaining about invalid MAC. What confuses me is that the MAC itself is encrypted as part of GenericBlockCipher:

struct {
    opaque IV[SecurityParameters.record_iv_length];
    block-ciphered struct {
        opaque content[TLSCompressed.length];
        opaque MAC[SecurityParameters.mac_length]; // <-- server reads this fine!
        uint8 padding[GenericBlockCipher.padding_length];
        uint8 padding_length;
    };
} GenericBlockCipher;

UPDATE: FWIW, I've added a Wireshark log of the failing 1.2 read of https://example.com, as well as a log of a functioning 1.1 session running what is the same code, not counting the P_SHA256 MAC update:

http://hostilefork.com/media/shared/stackoverflow/example-com-tls-1.2.pcapng (fails) http://hostilefork.com/media/shared/stackoverflow/example-com-tls-1.1.pcapng (succeeds)

So, what exactly is it having trouble decrypting? The padding seems correct, as if add or subtract 1 to the byte I get an invalid MAC error. (The spec says "The receiver MUST check this padding and MUST use the bad_record_mac alert to indicate padding errors.", so that is to be expected.) If I corrupt the client-iv in the message from what I used to encrypt (just put a bad byte in the transmitted version), doing so also gives me Bad Record MAC. I'd expect that to wreck the decryption also.

So I'm puzzled on what could be the problem:

• The server demonstrates discernment of valid MAC vs. not, so it must have decrypted. How's it getting the right MAC -and- having a decrypt error?
• Cipher suite is an old one (TLS_RSA_WITH_AES_256_CBC_SHA) but I'm just tackling one issue at a time...and if I'm not mistaken, that shouldn't matter.

Does anyone with relevant experience have a theory of how TLS 1.2 could throw a wrench into code that otherwise works in TLS 1.1? (Perhaps someone who's done a similar updating to a codebase, and had to change more than the two things I've changed to get it working?) Am I missing another crucial technical change? What recourse do I have to find out what is making the server unhappy?

Barkley

Hm...just a guess, but are you using correct padding type? if you use incorrect padding, you get error on decryption, but the decrypted data is correct. you your MAC will verify, but decrypt creates error. – Havenot

@Havenot Did the padding change between 1.1 and 1.2? As I say, I've really only changed two things so far (I know cert validation needs a couple of things but I'm not attacking that just yet)...I changed the initialization vector and the PRF. I haven't touched the padding code – Barkley

Just checked myself, padding mode itself is not changed. but terms for SHOULD is changed to MUST. ietf.org/rfc/rfc4346.txt tools.ietf.org/html/rfc5246 in case of TLS 1.0 these is no rule for returning error on incorrect padding. you started at TLS 1.0 so maybe something is missing in middle way. But in general it was just a guess. – Havenot

anyway, padding mechanism is simple. so you can quickly check it. – Havenot

BTW, you may have same problem, if you send IV incorrectly to server side. because you are using CBC encryption mode and if you send incorrect IV, only 1st block will be incorrect. so in decryption you will have correct MAC(assuming content is more than 16 bytes). – Havenot

@Havenot Wouldn't all blocks be incorrect with a bad IV, as the error would compound? I guess the main thing I'm wondering is just how it could work in 1.1 but then not 1.2; I don't know if I need to build and debug a server so I can see what it expected that it is erroring on... I can check the padding. But it would be nice if someone set up an https server somewhere that gave back something more useful than "decrypt error" (!) – Barkley

in CBC mode, incorrect IV only cause incorrect decryption on 1st block. it is true that it is strange that it is working in 1.1 but not 1.2, but without debugging, I can only think of these 2 problem. – Havenot

Answer 1

There's actually not anything wrong with the ChangeCipherSpec message. It's actually the Finished message that has the problem. It is complaining about the decrypted verify_data inside that message, which is not matching an expected hash (despite the encryption/decryption itself being correct).

But what's confusing in the Wireshark log is that the Finished message shows up on the same log line, but under the name "EncryptedHandshakeMessage" This makes it look like some kind of tag or label describing ChangeCipherSpec, but it's not. That message actually isn't encrypted at all.

• TLS finished packet renamed encrypted handshake message
• HTTPS over TLS - encrypted type

From the second link:

In practice, you will see unencrypted Client Hello, Server Hello, Certificate, Server Key Exchange, Certificate Request, Certificate Verify and Client Key Exchange messages. The Finished handshake message is encrypted since it occurs after the Change Cipher Spec message.

---

"Hoping someone has experience updating TLS 1.0 or 1.1 to 1.2, and might have seen a similar problem due to not changing more than the P_SHA256 MAC and bumping the version number"

They only mention two of the three places that you need to update the MD5/SHA1 combination in the "changes from TLS 1.1" section of RFC 5246:

• The MD5/SHA-1 combination in the pseudorandom function (PRF) has been replaced with cipher-suite-specified PRFs. All cipher suites in this document use P_SHA256.

• The MD5/SHA-1 combination in the digitally-signed element has been replaced with a single hash. Signed elements now include a field that explicitly specifies the hash algorithm used.

(Note: The second applies to certificates, and if you haven't gotten to certificate checking you wouldn't be at that point yet.)

What they don't mention in that section is the third place the MD5/SHA-1 combination changes, which is a hash used in the seed for the verify_data of the Finished message. However, this point is also a change from TLS 1.1, described much further down the document in section 7.4.9:

"Hash denotes a Hash of the handshake messages. For the PRF defined in Section 5, the Hash MUST be the Hash used as the basis for the PRF. Any cipher suite which defines a different PRF MUST also define the Hash to use in the Finished computation."

For a formal spec they're being a bit vague on "hash used as the basis for the PRF" (is it the HMAC or just the plain hash?) But it's the plain hash. So SHA256, unless the cipher suite's spec says otherwise.

(Note also the cipher suite can dictate the length of the verify_data as more than 12 bytes, though none mentioned in the spec do so.)

---

"What recourse do I have to find out what is making the server unhappy?"

YMMV. But what I did was just build OpenSSL as a static debug library, and linked it to a simple server. Then I added breakpoints and instrumentation to see what it was upset about. (GDB wasn't letting me step into the shared library, for some reason.)

Circa 30-Sep-2018, on a plain linux machine:

• git://git.openssl.org/openssl.git
• ./config no-shared no-asm -g3 -O0 -fno-omit-frame-pointer -fno-inline-functions no-ssl2 no-ssl3
• make

The simple server I used came from Simple TLS Server. Compile against the static library with:

• gcc -g -O0 simple.c -o simple -lssl -lcrypto -ldl -lpthread

I followed the instructions for generating certificates here, but changed the AAs to localhost

openSSL sign https_client certificate with CA

Then I changed the cert.pem => rootCA.pem and key.pem => rootCA.key in the simple server code. I was able to do:

wget https://localhost:4433 --no-check-certificate

And successfully get back test as a response. So then it was just a matter of seeing where my client caused a failure.

Barkley

excellent example of sharing knowledge acquired by experience! – Alb

Answer 2

I can think of 2 different situations that creates this problem:

1. Sending incorrect IV. IV affects only 1st block in decryption of CBC mode, so if your content is more than 16 bytes (AES block size), MAC part of your data will be decrypted correctly.
2. If you are using incorrect padding structure, you may get error in decryption(because padding verification fails), but content will be decrypted correctly.

Havenot

It seems the padding is correct...if I change it I get an invalid mac error (it says "The receiver MUST check this padding and MUST use the bad_record_mac alert to indicate padding errors.", so that is to be expected.) I corrupted the client-iv in the message from what I used to encrypt (just put a bad byte in the transmitted version) and doing so also gives me Bad Record MAC, which is what I'd expect. :-/ Grrr. – Barkley

Thanks for your helpfulness. I had bountied the question, but since I solved it myself, you get the points just for being a nice guy. Occasionally it pays off. :-) – Barkley

@HostileFork lol, thanks, I suddenly surprised to see a huge amount of points :D anyway, I like to know what the problem was at least, anything related to my reply? – Havenot