Terraform ELB S3 Permissions Issue

Question

I am having an issue using Terraform (v0.9.2) adding services to an ELB (I'm using: https://github.com/segmentio/stack/blob/master/s3-logs/main.tf).

When I run terraform apply I get this error:

* module.solr.module.elb.aws_elb.main: 1 error(s) occurred:

* aws_elb.main: Failure configuring ELB attributes: 
    InvalidConfigurationRequest: Access Denied for bucket: my-service-
    logs. Please check S3bucket permission
    status code: 409, request id: xxxxxxxxxx-xxxx-xxxx-xxxxxxxxx

My service looks like this:

module "solr" {
  source = "github.com/segmentio/stack/service"
  name = "${var.prefix}-${terraform.env}-solr"
  environment = "${terraform.env}"
  image = "123456789876.dkr.ecr.eu-west-2.amazonaws.com/my-docker-image"
  subnet_ids = "${element(split(",", module.vpc_subnets.private_subnets_id), 3)}"
  security_groups = "${module.security.apache_solr_group}"
  port = "8983"
  cluster = "${module.ecs-cluster.name}"
  log_bucket = "${module.s3_logs.id}"

  iam_role = "${aws_iam_instance_profile.ecs.id}"
  dns_name = ""
  zone_id = "${var.route53_zone_id}"
}

My s3-logs bucket looks like this:

module "s3_logs" {
  source = "github.com/segmentio/stack/s3-logs"
  name = "${var.prefix}"
  environment = "${terraform.env}"
  account_id = "123456789876"
}

I checked in S3 and the bucket policy looks like this:

{
  "Version": "2012-10-17",
  "Id": "log-bucket-policy",
  "Statement": [
  {
  "Sid": "log-bucket-policy",
  "Effect": "Allow",
  "Principal": {
  "AWS": "arn:aws:iam::123456789876:root"
  },
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::my-service-logs/*"
  }
  ]
}

As far as I can see ELB should have access to the S3 bucket to store the logs (it's running in the same AWS account).

The bucket and the ELB are all in eu-west-2.

Any ideas on what the problem could be would be much appreciated.

Banausic

Answer 1

The docs for ELB access logs say that you want to allow a specific Amazon account to be able to write to S3, not your account.

As such you want something like:

{
  "Id": "Policy1429136655940",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1429136633762",
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-loadbalancer-logs/my-app/AWSLogs/123456789012/*",
      "Principal": {
        "AWS": [
          "652711504416"
        ]
      }
    }
  ]
}

In Terraform you can use the aws_elb_service_account data source to automatically fetch the account ID used for writing logs as can be seen in the example in the docs:

data "aws_elb_service_account" "main" {}

resource "aws_s3_bucket" "elb_logs" {
  bucket = "my-elb-tf-test-bucket"
  acl    = "private"

  policy = <<POLICY
{
  "Id": "Policy",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-elb-tf-test-bucket/AWSLogs/*",
      "Principal": {
        "AWS": [
          "${data.aws_elb_service_account.main.arn}"
        ]
      }
    }
  ]
}
POLICY
}

resource "aws_elb" "bar" {
  name               = "my-foobar-terraform-elb"
  availability_zones = ["us-west-2a"]

  access_logs {
    bucket   = "${aws_s3_bucket.elb_logs.bucket}"
    interval = 5
  }

  listener {
    instance_port     = 8000
    instance_protocol = "http"
    lb_port           = 80
    lb_protocol       = "http"
  }
}

Courcy

Getting the Resource right in the policy was what fixed things for me. The documentation called for something like "arn:aws:s3:::my-loadbalancer-logs/my-app/AWSLogs/123456789012/*". Leaving off the account id at the end worked (e.g. "arn:aws:s3:::my-loadbalancer-logs/my-app/AWSLogs/*"`) – Daren

Helped me immensely! Thanks for sharing :) – Buchenwald

Unfortunately this answer is outdated as this doesn't work with the AWS application or network load balancer, the data.aws_elb_service_account.main.arn is solely part of the classic load balancer and has since been deprecated – Papilionaceous

still works today. modified "Resource": "arn:aws:s3:::my-elb-tf-test-bucket/AWSLogs/*", to "Resource": "arn:aws:s3:::my-elb-tf-test-bucket/*". Key is that data.aws_elb_service_account.main.arn is not account arn. its the service arn – Frow

Answer 2

In the bucket policy, the account number must be NOT yours. Instead it belongs to AWS, and for each region, the account numbers you should use in your bucket policy are listed at: https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-access-logs.html#attach-bucket-policy

For instance, for us-east-1 region the account number is 127311923021.

Although the question is about Terraform, I post CloudFormation snippet created a bucket for ELB's access logs its Bucket policy:

    MyAccessLogsBucket:
        Type: AWS::S3::Bucket
        DeletionPolicy: Retain


    MyAllowELBAccessBucketPolicy:
        Type: AWS::S3::BucketPolicy
        Properties: 
            Bucket: !Ref MyAccessLogsBucket
            PolicyDocument: 
                Version: "2012-10-17"
                Statement: 
                    - Effect: "Allow"
                      Principal: 
                          AWS: "arn:aws:iam::127311923021:root"
                      Action: 
                          - "s3:PutObject"
                      Resource: !Sub "arn:aws:s3:::${MyAccessLogsBucket}/AWSLogs/*"

In the principle, 127311923021 is used as this is AWS account number which should be used for account number in us-east-1.

Cymograph

Answer 3

Even when having everything by the docs, I kept getting the "Access Denied for bucket" error. Removing the encryption from the bucket worked for me.

Wideangle

Thanks, that fixed it for me. Did you find a way to modify the policy so that this isn't necessary? – Bilicki

No, I stopped trying as it wasn't critical for my case. Would love to hear if anyone found a way to make this work. – Wideangle

Also solved it for me. It's not mentioned in the docs that this can be a problem. Turning off encryption conflicts with CIS AWS Foundations Benchmark. – Jarad

Answer 4

Bucket permissions When you enable access logging, you must specify an S3 bucket for the access logs. The bucket must meet the following requirements. Requirements The bucket must be located in the same Region as the load balancer. Amazon S3-Managed Encryption Keys (SSE-S3) is required. No other encryption options are supported. The bucket must have a bucket policy that grants Elastic Load Balancing permission to write the access logs to your bucket. Bucket policies are a collection of JSON statements written in the access policy language to define access permissions for your bucket. Each statement includes information about a single permission and contains a series of elements.

Use one of the following options to prepare an S3 bucket for access logging.

Amazon S3-Managed Encryption Keys (SSE-S3) is required. No other encryption options are supported.

So AWS docu says KMS is not supported...

Parlormaid

Answer 5

This is a complete working example.

data "aws_caller_identity" "current" {}
data "aws_elb_service_account" "elb_account_id" {}

resource "aws_s3_bucket" "lb_logs" {
  bucket = "${local.name}-loadbalancer-logs"
}

resource "aws_s3_bucket_policy" "lb_logs" {
  bucket = aws_s3_bucket.lb_logs.id
  policy = data.aws_iam_policy_document.allow_lb.json
}

data "aws_iam_policy_document" "allow_lb" {
  statement {
    effect = "Allow"
    resources = [
      "arn:aws:s3:::${aws_s3_bucket.lb_logs.bucket}/AWSLogs/${data.aws_caller_identity.current.account_id}/*",
    ]
    actions = ["s3:PutObject"]
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_elb_service_account.elb_account_id.id}:root"]
    }
  }

  statement {
    effect = "Allow"
    resources = [
      "arn:aws:s3:::${aws_s3_bucket.lb_logs.bucket}/AWSLogs/${data.aws_caller_identity.current.account_id}/*",
    ]
    actions = ["s3:PutObject"]
    principals {
      type        = "Service"
      identifiers = ["delivery.logs.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"
      values   = ["bucket-owner-full-control"]
    }
  }

  statement {
    effect = "Allow"
    resources = [
      "arn:aws:s3:::${aws_s3_bucket.lb_logs.bucket}",
    ]
    actions = ["s3:GetBucketAcl"]
    principals {
      type        = "Service"
      identifiers = ["delivery.logs.amazonaws.com"]
    }
  }
}

resource "aws_alb" "lb" {
  name               = "${local.name}-lb"
  load_balancer_type = "application"
  subnets            = var.subnet_ids
  security_groups    = [
    aws_security_group.lb.id
  ]

  access_logs {
    bucket  = aws_s3_bucket.lb_logs.id
    enabled = true
  }

  lifecycle {
    create_before_destroy = true
  }

  tags = {
    Name = "${local.name}-lb"
  }
}

Eiger

Answer 6

In my case, it was the request_payer option set to Requester. Need to set to BucketOwner to work.

Photofinishing