Terraform error refreshing state access denied

Question

I'm using gitbucket for both my repository and for pipelines. I have a terraform config file with a remote state configured which runs fine on my local machine however it fails when running in gitbucket. I keep getting access denied error. Here's the main.tf:

terraform {
backend "s3" {
    bucket = "zego-terraform-test"
    key    = "test/terraform.tfstate"
    region = "eu-west-1"
  }
}

data "terraform_remote_state" "remote_state" {
  backend = "s3"

  config {
    bucket = "zego-terraform-test"
    key    = "test/terraform.tfstate"
    region = "eu-west-1"
  }
}

variable "region" {}

provider "aws" {
  region     = "${var.region}"
  access_key = {}
  secret_key = {}
  token      = {}
}

module "vpc" {
  source = "./modules/vpc"
}

Here's my gitbucket-pipelines.yml:

image: python:3.5.1
pipelines:
  default:
    - step:
        caches:
          - pip
        script: # Modify the commands below to build your repository.
          - apt-get update
          - apt-get install unzip
          - wget https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip
          - unzip terraform_0.11.7_linux_amd64.zip
          - rm terraform_0.11.7_linux_amd64.zip
          - export PATH="$PATH:${BITBUCKET_CLONE_DIR}"
          - terraform init
            -backend-config "access_key=$AWS_ACCESS_KEY"
            -backend-config "secret_key=$AWS_SECRET_KEY"
            -backend-config "token=$TOKEN"

When I run the .tf file in this pipeline I get this error:

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: AccessDenied: Access Denied
    status code: 403

When I remove remote state config it runs fine. Why am I getting the access denied error even though I'm using the same creds on my local machine and in gitbucket environment?

Record

Are you sure your credentials are being read in properly? What happens if you have your pipeline run the aws sts get-caller-identity command from the AWS CLI? – Exiguous

For me setting the AWS_PROFILE correctly solved the issue. – Affectional

For future readers, In my case, the region of the bucket was not correct. – Chalcography

Answer 1

Was getting the same error. For our use case, we have to manually remove the terraform.tfstate file under .terraform/ directory and run init again.

Mcquoid

I'm using windows. I accidentally deleted contents inside terraform.tfstate file. After which when i attempt to delete terraform.tfstate file. Its automatically being recreated. i deleted the resource manually in cloud. Now i can't to get rid of this .tfstate file in windows local. – Trimorphism

Answer 2

In case a solution has not been found for this issue, you can use either "profile=" or "role_arn=" in the config section of your terraform_remote_state stanza. The same is true for the AWS Provider and the backend configuration.

I chased this issue all day today not realizing that role_arn was available for terraform_backend_state data source.

Juice

Answer 3

Same error happened to me when I was using several aws accounts using profile names.
In my case, I was missing profile property in the backend configuration. Added profile and removed .terraform directory, and ran terraform init - it worked.

terraform {
  required_version = ">= 1.0.0"
  backend "s3" {
    // here!
    profile = "crazyoptimist"
    bucket = "devops-terraform-crazyoptimist"
    key = "terraform.tfstate"
    region = "us-west-2"
    encrypt = true
  }
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

provider "aws" {
  region = "us-west-2"
  profile = "crazyoptimist"
}

Interweave

Answer 4

This error might occur when switching between terraform backends. To solve it you can run terraform init -reconfigure to configure the backend to the new one.
It will achieve the same result as removing the terraform.tfstate file under .terraform and run terraform init.

Defiance

Answer 5

At first glance it seems reasonable. Have you tried having the terraform init and -backend-config's all on one line? I wonder if the - at the beginning is messing with the yml format?

Zounds

Answer 6

received same error while running terraform init command in codebuild, simply gave s3 bucket access to role created for codebuild and error was resolved

Herve

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Behaviorism

Answer 7

In my case the backend file of one of the data blocks of data.tf had permission issues, I just recreated that file and did terraform plan again, the problem sorted. Took ages to figure this out.

data "terraform_remote_state" "gateway" {
  backend = "s3"

  config = {
    bucket = "xxx-terraform-remote"
    key    = "xxx/terraform.tfstate"
    region = "eu-west-1"
  }
}

Ieyasu

Answer 8

In my case, there was an issue with the order in which AWS client looks for credentials.

I stored AWS credentials used by terreform in ~/.aws/credentials, but I've also had different AWS credentials set in environment varaibles.

I had to remove AWS credentials from my env variables and it worked.

Acclimatize

Answer 9

Terraform will need the following AWS IAM permissions on the target backend bucket:

s3:ListBucket on arn:aws:s3:::mybucket

s3:GetObject on arn:aws:s3:::mybucket/path/to/my/key

s3:PutObject on arn:aws:s3:::mybucket/path/to/my/key

s3:DeleteObject on arn:aws:s3:::mybucket/path/to/my/key

enter image description here

Semmes

It would be easier to understand the answer if you could expand on what you mean by path/to/my/key. Instead of inserting a screenshot of a documentation, please add a link to the documentation directly as it might contain other context that could help. Also if you use monospace for actions/ARNs the answer is easier to read, for example s3:ListBucket or arn:aws:s3:::mybucket. You can change a text to monospace by wrapping it in ` (backtick) characters. – Nimocks

Answer 10

Throwing out my 2 cents here, this solved the issue, in my case and no-one here mentioned this possible issue.

The bucket name I was specifying Terraform to create in an s3_bucket.tf file was "terraform-state-bucket". I than had a backend.tf file that pointed the state file to be saved there.

BUT! Of course S3 bucket name must be GLOBALLY unique, across all AWS accounts worldwide.

So I was receiving 403 error, because I was trying to access someone else's bucket!

After giving it a unique name (I first created the bucket manually via AWS console just to make sure) it worked like charm.

Giveaway

Answer 11

It looks like the profile credential is valid however there is not enough permissions

Try:

1. Navigate IAM -> Users -> wb-cli -> permissions tab -> Add permission

2. Add AdministratorAccess or select the specific permissions you wish to grant the user.

Altimeter