Using Encrypted EBS Volumes in Auto Scaling Groups with CMK owned by a different AWS account

Question

I'm trying to use Auto Scaling groups in AWS to create and manage instances created from AMIs with encrypted snapshots, which have been encrypted by a CMK owned by a different AWS account.

I keep getting the error "Client.InternalError: Client error on launch". According to Scenario 2 at https://docs.aws.amazon.com/autoscaling/ec2/userguide/ts-as-instancelaunchfailure.html#ts-as-instancelaunchfailure-12, I need to create a grant to the CMK with the Auto Scaling group service-linked role as the grantee principal.

I tried following the guidelines in the AWS documentation and at https://forums.aws.amazon.com/thread.jspa?threadID=277523 for setting up the grant.

However, I keep getting an AccessDeniedException saying that my user is not authorised to perform kms:CreateGrant on the CMK.

I feel like I've followed the instructions perfectly, but it's not working. I'm hoping someone might be able to provide some insight.

Theatrical

Answer 1

I chatted with an AWS employee who ran into the same problem until he re-read the forum post. The key line in Case 2 Step 4 is "The kms:GrantIsForAWSResource condition is not included to allow an IAM user or role in account 111122223333 to create the grant in the next step.".

In other words, you need to remove this condition from the default key policy for a customer managed CMK.

The instructions could've made that requirement much more explicit, but technically it's there and it resolves the problem.

Edit: To clarify, I'm going to include the default and amended JSON below.

The following is the default key policy as shown at https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html#key-policy-default

    {
      "Version": "2012-10-17",
      "Id": "key-consolepolicy-2",
      "Statement": [
        {
          "Sid": "Enable IAM User Permissions",
          "Effect": "Allow",
          "Principal": {"AWS": "arn:aws:iam::111122223333:root"},
          "Action": "kms:*",
          "Resource": "*"
        },
        {
          "Sid": "Allow access for Key Administrators",
          "Effect": "Allow",
          "Principal": {"AWS": [
            "arn:aws:iam::111122223333:user/KMSAdminUser",
            "arn:aws:iam::111122223333:role/KMSAdminRole"
          ]},
          "Action": [
            "kms:Create*",
            "kms:Describe*",
            "kms:Enable*",
            "kms:List*",
            "kms:Put*",
            "kms:Update*",
            "kms:Revoke*",
            "kms:Disable*",
            "kms:Get*",
            "kms:Delete*",
            "kms:TagResource",
            "kms:UntagResource",
            "kms:ScheduleKeyDeletion",
            "kms:CancelKeyDeletion"
          ],
          "Resource": "*"
        },
        {
          "Sid": "Allow use of the key",
          "Effect": "Allow",
          "Principal": {"AWS": [
            "arn:aws:iam::111122223333:user/KMSUser",
            "arn:aws:iam::111122223333:role/KMSRole",
            "arn:aws:iam::444455556666:root"
          ]},
          "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
          ],
          "Resource": "*"
        },
        {
          "Sid": "Allow attachment of persistent resources",
          "Effect": "Allow",
          "Principal": {"AWS": [
            "arn:aws:iam::111122223333:user/KMSUser",
            "arn:aws:iam::111122223333:role/KMSRole",
            "arn:aws:iam::444455556666:root"
          ]},
          "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
          ],
          "Resource": "*",
          "Condition": {"Bool": {"kms:GrantIsForAWSResource": "true"}}
        }
      ]
    }

The key is to remove the Condition for "kms:GrantIsForAWSResource" as below.

    {
      "Version": "2012-10-17",
      "Id": "key-consolepolicy-2",
      "Statement": [
        {
          "Sid": "Enable IAM User Permissions",
          "Effect": "Allow",
          "Principal": {"AWS": "arn:aws:iam::111122223333:root"},
          "Action": "kms:*",
          "Resource": "*"
        },
        {
          "Sid": "Allow access for Key Administrators",
          "Effect": "Allow",
          "Principal": {"AWS": [
            "arn:aws:iam::111122223333:user/KMSAdminUser",
            "arn:aws:iam::111122223333:role/KMSAdminRole"
          ]},
          "Action": [
            "kms:Create*",
            "kms:Describe*",
            "kms:Enable*",
            "kms:List*",
            "kms:Put*",
            "kms:Update*",
            "kms:Revoke*",
            "kms:Disable*",
            "kms:Get*",
            "kms:Delete*",
            "kms:TagResource",
            "kms:UntagResource",
            "kms:ScheduleKeyDeletion",
            "kms:CancelKeyDeletion"
          ],
          "Resource": "*"
        },
        {
          "Sid": "Allow use of the key",
          "Effect": "Allow",
          "Principal": {"AWS": [
            "arn:aws:iam::111122223333:user/KMSUser",
            "arn:aws:iam::111122223333:role/KMSRole",
            "arn:aws:iam::444455556666:root"
          ]},
          "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
          ],
          "Resource": "*"
        },
        {
          "Sid": "Allow attachment of persistent resources",
          "Effect": "Allow",
          "Principal": {"AWS": [
            "arn:aws:iam::111122223333:user/KMSUser",
            "arn:aws:iam::111122223333:role/KMSRole",
            "arn:aws:iam::444455556666:root"
          ]},
          "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
          ],
          "Resource": "*"
        }
      ]
    }

Theatrical

I would upvote your remarks multiple times. I also struggled quite a bit. I'm still unsure about the RevokeGrant permissions though. I left them in as well. One very minor thing that might help others. I faced the issue that my shared encrypted snapshots were re-encrypted usung the default aws/ebs keys until I read this link: docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIEncryption.html – October

Answer 2

After reading your helpful information I was able to solve it so I decided to post my findings her for others as well.

Here is exactly what I did to allow a Custom KMS key (CMK) from the "dev" account to be accessed and used by the "SharedAccountId".

For this example, lets say the "dev" account is in us-west-2 and the "SharedAccount" is in us-east-1.

Cloudformation to create a key:

NOTE: Launch this cloudformation stack in "Dev" Account, which in this example lives in us-west-2

{
"Description": "Creates a KMS key used to encrypt snapshots and allows sharing with another account.",
"Outputs": {
    "AMIKeyIdOutput": {
        "Description": "The KMS Key id used to encrypted snapshots.",
        "Export": {
            "Name": {
                "Fn::Sub": "${AWS::StackName}-kmskeyid"
            }
        },
        "Value": {
            "Ref": "AMIKmsKey"
        }
    },
    "AMIKmsAliasOutput": {
        "Description": "The KMS key alias used to encrypted snapshots.",
        "Export": {
            "Name": {
                "Fn::Sub": "${AWS::StackName}-kmsalias"
            }
        },
        "Value": {
            "Ref": "AMIKmsAlias"
        }
    }
},
"Parameters": {
    "SharedAccountId": {
        "AllowedPattern": "^(?!\\s*$).+",
        "ConstraintDescription": "You must supply a account id you want to share with.",
        "Description": "The account id you want to share this key with.",
        "Type": "String"
    }
},
"Resources": {
    "AMIKmsAlias": {
        "Properties": {
            "AliasName": {
                "Fn::Sub": "alias/amiencryptionkey"
            },
            "TargetKeyId": {
                "Ref": "AMIKmsKey"
            }
        },
        "Type": "AWS::KMS::Alias"
    },
    "AMIKmsKey": {
        "Properties": {
            "Description": "AMI encryption key.",
            "EnableKeyRotation": "true",
            "Enabled": "true",
            "KeyPolicy": {
                "Statement": [
                    {
                        "Action": [
                            "kms:*"
                        ],
                        "Effect": "Allow",
                        "Principal": {
                            "AWS": {
                                "Fn::Sub": "arn:aws:iam::${AWS::AccountId}:root"
                            }
                        },
                        "Resource": [
                            "*"
                        ],
                        "Sid": "Allow access for Key Administrators"
                    },
                    {
                        "Action": [
                            "kms:Decrypt",
                            "kms:Encrypt",
                            "kms:DescribeKey",
                            "kms:ReEncrypt*",
                            "kms:GenerateDataKey*"
                        ],
                        "Effect": "Allow",
                        "Principal": {
                            "AWS": [
                                {
                                    "Fn::Join": [
                                        "",
                                        [
                                            "arn:aws:iam::",
                                            {"Ref":"SharedAccountId"},
                                            ":root"
                                        ]
                                    ]
                                },
                                {
                                    "Fn::Join": [
                                        "",
                                        [
                                            "arn:aws:iam::",
                                            {"Ref":"SharedAccountId"},
                                            ":role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
                                        ]
                                    ]
                                },
                                {
                                    "Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
                                }
                            ]
                        },
                        "Resource": [
                            "*"
                        ],
                        "Sid": "Allow use of the key"
                    },
                    {
                        "Action": [
                            "kms:CreateGrant",
                            "kms:ListGrants",
                            "kms:RevokeGrant"
                        ],
                        "Effect": "Allow",
                        "Principal": {
                            "AWS": [
                                {
                                    "Fn::Sub": "arn:aws:iam::${AWS::AccountId}:root"
                                },
                                {
                                    "Fn::Join": [
                                        ":",
                                        [
                                            "arn:aws:iam:",
                                            {"Ref":"SharedAccountId"},
                                            "root"
                                        ]
                                    ]
                                },
                                {
                                    "Fn::Join": [
                                        "",
                                        [
                                            "arn:aws:iam::",
                                            {"Ref":"SharedAccountId"},
                                            ":role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
                                        ]
                                    ]
                                },
                                {
                                    "Fn::Sub": "arn:aws:iam::${AWS::AccountId}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
                                }
                            ]
                        },
                        "Resource": [
                            "*"
                        ],
                        "Sid": "Allow attachment of persistent resources."
                    }
                ],
                "Version": "2012-10-17"
            }
        },
        "Type": "AWS::KMS::Key"
    }
}
}

Also it's important to note that some of the principals are not needed but it should be good enough to get you started. After setting up your kms key like the above logic then you have to run the following cli command:

NOTE: In this example
* SharedAccountId in us-east-1
* KMS Key lives in "Dev" account which is in us-west-2

aws kms create-grant \
--region us-east-1 \
--profile SharedAccountProfile \
--key-id arn:aws:kms:us-west-2:<DevAccountId>:key/<KMS_KEY_ID From above CF template> \
--grantee-principal arn:aws:iam::<SharedAccountId>:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling \
--operations "Encrypt" "Decrypt" "ReEncryptFrom" "ReEncryptTo" "GenerateDataKey" "GenerateDataKeyWithoutPlaintext" "DescribeKey" "CreateGrant"

That should do it. Now you can share encrypted AMI's between accounts and allow Autoscaling Groups to launch instances with them.

Integument

Thanks for sharing what you've done here! I'm going to edit my answer to be more explicit as well. – Theatrical

The AWS docs make me want to quit the tech industry on most days. Creating the grant is ultimately what solved the issue for me. docs.aws.amazon.com/autoscaling/ec2/userguide/… – Inweave