AWS Glue Access denied for crawler with administrator policy attached

Question

I am trying to run a crawler across an s3 datastore in my account which contains two csv files. However, when I try to run the crawler, no tables are loaded, and I see the following errors in cloudwatch for the each of the files:

• Error Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;
• Tables created did not infer schemas from this file.

This is especially odd as the IAM role has the AdministratorAccess policy attached, so there should not be any access denied issue.

Any help would be appreciated.

Sleepyhead

Answer 1

I made sure that I wasn't missing something offered in the other suggestions, but I wasn't. It turns out there was another level of restrictions on reading the bucket imposed by my organization, though i'm not sure what it was.

Sleepyhead

Answer 2

Check to see if the files you are crawling are encrypted. If they are, then your Glue role probably doesn't have a policy that allows it to decrypt.

If so, it might need something like this:

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "kms:Decrypt"
    ],
    "Resource": [
      "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
      "arn:aws:kms:us-west-2:111122223333:key/0987dcba-09fe-87dc-65ba-ab0987654321"
    ]
  }
}

Antonelli

Answer 3

Make sure the policies attached to you IAM role have these :

1. AmazonS3FullAccess
2. AwsGlueConsoleFullAccess
3. AwsGlueServicerole.

Buhler

And if files are encrypted with KMS key then it requires to have Decode permission for KMS – Selfannihilation

can you explain why does awsglueconsolefullaccess necessary – Anadiplosis

AmazonS3FullAccess is a very loose policy. You can tighten this up by including a customer managed policy that allows s3:ListBucket and s3:GetObject for the bucket that contains your data. See the note under step 5 of: docs.aws.amazon.com/glue/latest/dg/create-an-iam-role.html – Goebbels

Answer 4

We had a similar issue with an S3 crawler. According to AWS, S3 crawlers, unlike JDBC crawlers, do not create an ENI in your VPC. This means your bucket policy must allow access from outside the VPC.

Check that your bucket policy does not have an explicit deny somewhere on S3:*. If there is one, make sure to add a conditional on the statement and add the role id in the conditional as aws:userId in the statement. Keep in mind the role id and role arn is not the same thing.

To get the role id:

aws iam get-role --role-name Test-Role

Output:

{
  "Role": {
      "AssumeRolePolicyDocument": "<URL-encoded-JSON>",
      "RoleId": "AIDIODR4TAW7CSEXAMPLE",
      "CreateDate": "2013-04-18T05:01:58Z",
      "RoleName": "Test-Role",
      "Path": "/",
      "Arn": "arn:aws:iam::123456789012:role/Test-Role"
  }
}  

You might also need to add a state that allows s3:putObject* and s3:getObject* with the aws principal the assumed role. The assumed role will look something like:

arn:aws:sts::123456789012:assumed-role/Test-Role/AWS-Crawler

Hope this helps.

Deepfreeze

Answer 5

In my case the issue was: the crawler was configured in different region than S3 bucket it meant to crawl. After configuring new crawler in the same region as my S3 bucket the problem was resolved.

Generalization

Answer 6

This is an S3 bucket policy issue. I made my tables public (bad policy I know) and it worked.

Epigenous

Answer 7

IAM Roles Here are the complete roles you need to give in order for Glue Crawler to work properly.

IAM Roles

Jamboree