What permission am I missing for AWS Glue and Development Endpoint?

Question

I'm getting the following error when I try to create a development endpoint for AWS Glue.

{ "service":"AWSGlue",
"statusCode":400,
"errorCode":"ValidationException",
"requestId":"<here goes an UUID>",
"errorMessage":"Role arn:aws:iam::<IAM ID>:role/AWSGlueServiceRole-DefaultRole 
      should be given assume role permissions for Glue Service.\n",
"type":"AwsServiceError" }

And my role has the following permissions.

• AmazonS3FullAccess
• AWSGlueServiceNotebookRole
• AmazonAthenaFullAccess
• AWSGlueServiceRole
• CloudWatchLogsReadOnlyAccess
• AWSGlueConsoleFullAccess
• AWSCloudFormationReadOnlyAccess

Any clues on what am I missing?

Soudan

Answer 1

I was tripped up by this as well; the problem is that when you use the console to create a default glue service role it ends up creating the IAM role like this:

arn:aws:iam:::role/service-role/AWSGlueServiceRole-DefaultRole

Make note of the "service-role" in the path.

But then when choosing that role as the role you want to use in the console wizard for setting up a new dev endpoint it doesn't include the "service-role" in the path and looks for a role named like this:

arn:aws:iam:::role/AWSGlueServiceRole-DefaultRole

I think this is just a bug in the console wizard for creating dev endpoints. I got around it by creating a new role that doesn't have "service-role" in the path and then chose that role in the console wizard and was able to successfully create a dev endpoint.

Secant

If you follow the data lake QuickStart, don't reuse the role it creates. It creates the "/service-role" version of the role and fails when used to create a development endpoint. – Disclose

This is the exact problem I encounter. – Contrasty

Answer 2

In your trust relationship, the trust should be established with glue.amazonaws.com. Your role (AWSGlueServiceRole-DefaultRole) may not have this. To confirm, go to the IAM roles console, select the IAM role: AWSGlueServiceRole-DefaultRole and click on the Trust Relationship tab.

The json for this should look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Example screenshot for the Trust relationship:

Angilaangina

Have you looked at this AWS document: docs.aws.amazon.com/glue/latest/dg/…. The user (using which you have logged in to the AWS console) should have iam:PassRole permission; not just the IAM role (AWSGlueServiceRole-DefaultRole). – Angilaangina

I'll try to add that. But says here docs.aws.amazon.com/glue/latest/dg/attach-policy-iam-user.html that I can skip this if my role has full access – Soudan

That statement indicates that - the user you are logging into the AWS console should have that policy attached. Because AWSGlueConsoleFullAccess has iam:PassRole. – Angilaangina

Added AWSGlueConsoleFullAccess to the user logged in and the iam:PassRole as inline. Still does not work – Soudan

Do I need to have one role for the job and another role for development endpoint? – Soudan

Answer 3

The problem was somehow related to an old Role that I already messed up with. Created a brand new role just for development following this link and this link, worked like a charm.

Soudan

Answer 4

My problem was a variation on the one described by @kinzleb (or perhaps the same).

Somehow, the IAM Role attached to my Glue job got set to a customer-managed role† ARN, like this:

arn:aws:iam::000011110000:role/AWSGlueService

Whereas my other (working) jobs had a service role ARN, like this:

arn:aws:iam::000011110000:role/service-role/AWSGlueServiceRole

I wasn't able to discover the difference in the AWS Console because the UI doesn't make it possible to differentiate between a customer-managed and a service role (you can't see the ARN), but I compared a examples of working and non-working jobs via the AWS CLI like so:

$ aws glue --region my-aws-region get-job --job-name my_working_job | jq .Job.Role
"arn:aws:iam::000011110000:role/service-role/AWSGlueServiceRole"

$ aws glue --region my-aws-region get-job --job-name my_failing_job | jq .Job.Role
"arn:aws:iam::000011110000:role/role/AWSGlueServiceRole"

I confirmed via the IAM console that I don't even have a customer-managed role with that ARN. I only have the service-managed role. The error message is highly-misleading to the point of obfuscation.

I solved the problem by using $ aws glue --region my-aws-region update-job --job-name my-failing-job --job-update ... from the command line to assign the service role to the job.

If this issue is introduced by a console bug, shame on AWS for having it still be there many years after @kinzleb's answer! And AWS, please update your stuff so I can differentiate between customer-managed and service roles within the Glue console website!

---

*: I anonymized the AWS account ID, hence the fake-looking 000011110000.

†: "Customer-managed role" is fancy AWS-speak for "a role managed by me, the customer, who owns my AWS account and can create roles in it". Used in distinction to "service role" which is when the service creates the role in my account on my behalf and keeps it updated (these ones have /service-role/ in the ARN path) and also in distinction to service-linked roles which are another concept altogether and have /aws-service-role/ in the ARN path... Phew.

Mithridatism