Amazon SES 535 Authentication Credentials Invalid trying to rotate access key

Question

We have an Amazon SES setup that works well and sends thousands of emails a day via SMTP. Trying to follow a best practice of "rotating" access keys we went to https://console.aws.amazon.com/iam/home and creating a new access key for the exact same user which is used to send emails. The new key is supposedly active but when trying to email with the access keys, we keep getting

535 Authentication Credentials Invalid

Switching to the old access keys works well and emails are sent. Tried a couple of times to delete the new access keys and create others. Same machine, same software. We have proper copy+paste skills to ensure we're using the same ID/Password provided in the CSV coming from Amazon. Here the dialog from Amazon:

So what's going on? Is there a time limit till the new key becomes active? Is there some other hidden limitation somewhere?

Bellanca

Answer 1

You are confusing the SMTP credentials with access_key and secret. They are different.

• access_key/secret --> Use in SDK and CLI

• SMTP credentials --> Use to configure SES SMTP

• You are creating a new access_key/secret and using it as SMTP credentials

• Instead you create a new SMTP credentials and use it
• Key rotation is different from SMTP credential rotation
• No need to create a new user

It is likely you are using the SMTP credentials that does not change even if you generate another set of access_key/secret. In your case it looks like you are using the SMTP server and not the SDK. So generating a new set of access_key/secret has no effect on SMTP credentials.

If you want to create a new set of SMTP credentials, go to AWS SES dashboard and create SMTP credentials.

For more information: Obtaining Your Amazon SES SMTP Credentials

Vaginal

We've added a screen shot of the dialog we've used. This is the user that is actively used to send emails via SMTP so, if understood correctly, the user has SMTP credentials. We're trying to rotate the access keys on the user, not create another user. – Bellanca

Try reading my answer again. You don't seem to understand the difference. You are creating new keys and using it as SMTP credentials which won't work. As I said in my answer, SMTP credentials and access key are different. Nowhere in my answer I said you have to create a new user. – Vaginal

Thank you for your reply. This was extremely confusing. Creating SMTP Credentials creates a new user and the access keys (user/pass) appearing under "Security Credentials" for that user are not the user/pass used for sending SMTP. So while Amazon encourages "rotating" access keys, these cannot be be used for SMTP sending and effectively the only way to "rotate" passwords for SMTP sending is to create new SMTP credentials (meaning also a new user). We find that really confusing and hopefully your answer will help others. – Bellanca

Yep, have to create a new SMTP Credentials to rotate the accessKeys which is weird but it works. It won't work if just create access keys for existing ses-smtp-user. Also, Obtaining Your Amazon SES SMTP Credentials the last line of the doc says If you want to change your SMTP password, go to the IAM console and delete your existing IAM user, and then go to the Amazon SES console to re-generate your SMTP credentials. However, you'd like to keep old users for a while till you are sure it's inactive. – Infanta

Thank you! This is exactly the answer I needed. Spent hours trying to figure out why new access keys didn't work. It doesn't help that the SMTP credentials become listed as access keys in that list. – Machellemachete

Answer 2

Yes, there's a hidden limitation in the way AWS approaches the SMTP password for SES. And they are using a very confusing way of handling these credentials.

The answer from helloV is on the right track, but it's not entirely correct. Both AWS and his answer tell us that Access_key/Secret_key and SES SMTP credentials are different things, but:

• If you create fresh SES SMTP credentials, it creates a new IAM User with an Access Key/Secret Key pair
• The Access Key Id is the same as the username for SMTP
• If you delete or disable this key, you lose your SMTP access. So they are clearly very related.
• The password for SMTP is derived from the Secret Key

It turns out that a new access_key/secret_key pair on an existing IAM user, can be used for SMTP, and therefore keys can be rotated without creating new users. AWS converts the Secret Access Key to generate the SMTP password, as they explain in this documentation page:

The following pseudocode shows the algorithm that converts an AWS Secret Access Key to an Amazon SES SMTP password.

key = AWS Secret Access Key;
message = "SendRawEmail";
versionInBytes = 0x02;
signatureInBytes = HmacSha256(message, key);
signatureAndVer = Concatenate(versionInBytes, signatureInBytes);
smtpPassword = Base64(signatureAndVer);

So using the Secret Access key, the SMTP password can be generated
With bash and openssl installed, the following command will output the password for use in SMTP:

(echo -en "\x02"; echo -n 'SendRawEmail' \
  | openssl dgst -sha256 -hmac $AWS_SECRET_ACCESS_KEY -binary) \
  | openssl enc -base64

Just replace $AWS_SECRET_ACCESS_KEY with your key, or set the variable beforehand

Graceless

This is handy! Thanks. – Maxey

AWS Documentation have updated their code sample to use version '0x04', with a different setup. While the code sample still works as of now, please know that in future they might drop the support for version '0x02'. – Chloras

This method is in principle correct. However, it's out of date. The best way is to use the python code given in the official page in this answer. It works for me. – Tradesfolk

Answer 3

Since, both Secret Key and SMTP Password are in different format, you need to convert Secret Key to SMTP Password using algorithm provided by AWS.

You can find it here: https://aws.amazon.com/premiumsupport/knowledge-center/ses-rotate-smtp-access-keys/

Elysha

Answer 4

Here is a working piece of code for transformning your secret key to smtp password using bash :

#!/usr/bin/env bash
# Convert AWS Secret Access Key to an Amazon SES SMTP password
# using the following pseudocode:
#
#   date = "11111111";
#   service = "ses";
#   terminal = "aws4_request";
#   message = "SendRawEmail";
#   version = 0x04;
#
#   kDate = HmacSha256(date, "AWS4" + key);
#   kRegion = HmacSha256(region, kDate);
#   kService = HmacSha256(service, kRegion);
#   kTerminal = HmacSha256(terminal, kService);
#   kMessage = HmacSha256(message, kTerminal);
#   signatureAndVersion = Concatenate(version, kMessage);
#   smtpPassword = Base64(signatureAndVersion);
#
# Usage:
#   chmod u+x aws-ses-smtp-password.sh
#   ./aws-ses-smtp-password.sh secret-key-here
# See: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html
#

if [ "$#" -ne 1 ]; then
  echo "Usage: ./aws-ses-smtp-password.sh secret-key-here"
  exit 1
fi

KEY="${1}"

DATE="11111111"
REGION="eu-west-1"
SERVICE="ses"
TERMINAL="aws4_request"
MESSAGE="SendRawEmail"
VERSION="4"
VERSION_IN_BYTES=$(printf \\$(printf '%03o' "${VERSION}"));

#SIGNATURE_IN_BYTES=$(echo -n "${MESSAGE}" | openssl dgst -sha256 -hmac "${KEY}" -binary);

SIGNATURE_IN_BYTES=$(echo -n "${DATE}" | openssl dgst -sha256 -mac HMAC -macopt "key:AWS4${KEY}" | sed 's/^.* //');
SIGNATURE_IN_BYTES=$(echo -n "${REGION}" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${SIGNATURE_IN_BYTES}" | sed 's/^.* //');
SIGNATURE_IN_BYTES=$(echo -n "${SERVICE}" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${SIGNATURE_IN_BYTES}" | sed 's/^.* //');
SIGNATURE_IN_BYTES=$(echo -n "${TERMINAL}" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${SIGNATURE_IN_BYTES}" | sed 's/^.* //');
SIGNATURE_IN_BYTES=$(echo -n "${MESSAGE}" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${SIGNATURE_IN_BYTES}" -binary | sed 's/^.* //');
SIGNATURE_AND_VERSION="${VERSION_IN_BYTES}${SIGNATURE_IN_BYTES}"
SMTP_PASSWORD=$(echo -n "${SIGNATURE_AND_VERSION}" | base64);

echo "${SMTP_PASSWORD}"

Jonette

Answer 5

Alexandre's answer is great - thank you.

Some improvements include the less awkward version handling (as in Chikitulfo's answer) and DRYness:

#!/bin/bash -e
function HmacSha256 {
    echo -n "$1" | openssl dgst -sha256 -mac HMAC -macopt "$2" $3 | sed 's/^.* //'
}

function key2pwd {

    local kDate=$(HmacSha256 '11111111' "key:AWS4$1")
    local kRegion=$(HmacSha256 'us-east-1' "hexkey:$kDate");
    local kService=$(HmacSha256 'ses' "hexkey:$kRegion")
    local kTerminal=$(HmacSha256 'aws4_request' "hexkey:$kService")
    local kMessage=$(HmacSha256 'SendRawEmail' "hexkey:$kTerminal" -binary)

    (echo -en '\x04'; echo -n "$kMessage") | openssl enc -base64

}

SMTP_PWD=$(key2pwd 'abcdefg/1234')
echo "$SMTP_PWD" # BAnYaXRsbgGIcqAOX4N5ss+aBs5F3AlV80bYtiTZ88f5

Gstring