How to use the AWS Python SDK while connecting via SSO credentials

Question

I am attempting to create a python script to connect to and interact with my AWS account. I was reading up on it here https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html

and I see that it reads your credentials from ~/.aws/credentials (on a Linux machine). I however and not connecting with an IAM user but SSO user. Thus, the profile connection data I use is located at ~/.aws/sso/cache directory.

Inside that directory, I see two json files. One has the following keys:

• startUrl
• region
• accessToken
• expiresAt

the second has the following keys:

• clientId
• clientSecret
• expiresAt

I don't see anywhere in the docs about how to tell it to use my SSO user.

Thus, when I try to run my script, I get error such as

botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the DescribeSecurityGroups operation: AWS was not able to validate the provided access credentials

even though I can run the same command fine from the command prompt.

Gherardi

Can you clarify your setup. What's in the sso file? You can specify new location of the credential file using AWS_SHARED_CREDENTIALS_FILE env variable if this is what you want to do. – Glossography

@Glossography I'm sorry if I didn't clarify. sso is not a file but a directory. I"m updating the ticket. – Gherardi

Are those clientId and clientSecret and accessToken temporary AWS credentials? If yes, than you have to load it in boto3 manually I think, and create new boto3 session with the credentials. – Glossography

@Glossography Yes, they are. They expire every so often and have to be refreshed. Can you give some more details about how I might use a session to achieve this? – Gherardi

aws cli has get-role-credentials for working with sso. Maybe it needs to be used to refresh credentials. I'm not sure how it works exactly. – Glossography

Answer 1

This was fixed in boto3 1.14.

So given you have a profile like this in your ~/.aws/config:

[profile sso_profile]
sso_start_url = <sso-url>
sso_region = <sso-region>
sso_account_id = <account-id>
sso_role_name = <role>
region = <default region>
output = <default output (json or text)>

And then login with $ aws sso login --profile sso_profile

You will be able to create a session:

import boto3
boto3.setup_default_session(profile_name='sso_profile')
client = boto3.client('<whatever service you want>')

Presumptive

Answer 2

UPDATED for latest boto3 on 2023.10.23 (hat tip commenter @Adam Smith whose invisible hand guided us to updating extracting the role credentials in newer versions of boto3):

So here's the long and hairy answer tested on boto3==1.28.69:

It's an eight-step process where:

1. register the client using sso-oidc.register_client
2. start the device authorization flow using sso-oidc.start_device_authorization
3. redirect the user to the sso login page using webbrowser.open
4. poll sso-oidc.create_token until the user completes the signin
5. list and present the account roles to the user using sso.list_account_roles
6. get role credentials using sso.get_role_credentials
7. create a new boto3 session with the session credentials from (6)
8. eat a cookie

Step 8 is really key and should not be overlooked as part of any successful authorization flow.

In the sample below the account_id should be the account id of the account you are trying to get credentials for. And the start_url should be the url that aws generates for you to start the sso flow (in the AWS SSO management console, under Settings).

from time import time, sleep
import webbrowser
from boto3.session import Session

# if your sso is setup in a different region, you will
# want to include region_name=sso_region in the 
# session constructor below
session = Session()
account_id = '1234567890'
start_url = 'https://d-0987654321.awsapps.com/start'
region = 'us-east-1' 
sso_oidc = session.client('sso-oidc')
client_creds = sso_oidc.register_client(
    clientName='myapp',
    clientType='public',
)
device_authorization = sso_oidc.start_device_authorization(
    clientId=client_creds['clientId'],
    clientSecret=client_creds['clientSecret'],
    startUrl=start_url,
)
url = device_authorization['verificationUriComplete']
device_code = device_authorization['deviceCode']
expires_in = device_authorization['expiresIn']
interval = device_authorization['interval']
webbrowser.open(url, autoraise=True)
for n in range(1, expires_in // interval + 1):
    sleep(interval)
    try:
        token = sso_oidc.create_token(
            grantType='urn:ietf:params:oauth:grant-type:device_code',
            deviceCode=device_code,
            clientId=client_creds['clientId'],
            clientSecret=client_creds['clientSecret'],
        )
        break
    except sso_oidc.exceptions.AuthorizationPendingException:
        pass
 
access_token = token['accessToken']
sso = session.client('sso')
account_roles = sso.list_account_roles(
    accessToken=access_token,
    accountId=account_id,
)
roles = account_roles['roleList']
# simplifying here for illustrative purposes
role = roles[0]

# earlier versions of the sso api returned the 
# role credentials directly, but now they appear
# to be in a subkey called `roleCredentials`
role_creds = sso.get_role_credentials(
    roleName=role['roleName'],
    accountId=account_id,
    accessToken=access_token,
)['roleCredentials']
session = Session(
    region_name=region,
    aws_access_key_id=role_creds['accessKeyId'],
    aws_secret_access_key=role_creds['secretAccessKey'],
    aws_session_token=role_creds['sessionToken'],
)

Blacktail

You're suggesting that the end-user fires off secret credentials to https://d-0987654321.awsapps.com/start - that's a bit alarming. What is this endpoint supposed to represent? Otherwise (if this works) - cool solution :) – Lest

Apologies, that was intended to be a representation of the default sso url that aws generates for you when you set up the AWS SSO idp hive in the management console. I thought the descending from 0 pattern would make it obvious, but you're right that I should clarify above. – Blacktail

This answer is so good I wish I could hit the upvote twice. Thanks a ton. – Carolus

Change the sso_oidc session to the below to be able to login to a profile that does not have a Role assigned (default profile): config = botocore.config.Config(region_name=region, signature_version=botocore.UNSIGNED) sso_oidc = session.client('sso-oidc', config=config) The UNSIGNED argument above can be removed to see the difference. – Orthopedic

I believe role_creds should be sso.get_role_credentials(...)['roleCredentials'] – Rutabaga

I've noticed that the token returned from create_token does not have a refresh token field, unlike the tokens the AWS CLI gets (I can see them in the .aws/sso/cache directory, and as a result they expire after 8 hours and I need to reauthorize manually (which is really bad for my process) – Windhover

Answer 3

Your current .aws/sso/cache folder structure looks like this:

$ ls
botocore-client-XXXXXXXX.json       cXXXXXXXXXXXXXXXXXXX.json

The 2 json files contain 3 different parameters that are useful.

botocore-client-XXXXXXXX.json -> clientId and clientSecret
cXXXXXXXXXXXXXXXXXXX.json -> accessToken

Using the access token in cXXXXXXXXXXXXXXXXXXX.json you can call get-role-credentials. The output from this command can be used to create a new session.

Your Python file should look something like this:

import json
import os
import boto3

dir = os.path.expanduser('~/.aws/sso/cache')

json_files = [pos_json for pos_json in os.listdir(dir) if pos_json.endswith('.json')]

for json_file in json_files :
    path = dir + '/' + json_file
    with open(path) as file :
        data = json.load(file)
        if 'accessToken' in data:
            accessToken = data['accessToken']

client = boto3.client('sso',region_name='us-east-1')
response = client.get_role_credentials(
    roleName='string',
    accountId='string',
    accessToken=accessToken
)

session = boto3.Session(aws_access_key_id=response['roleCredentials']['accessKeyId'], aws_secret_access_key=response['roleCredentials']['secretAccessKey'], aws_session_token=response['roleCredentials']['sessionToken'], region_name='us-east-1')

Peba

This might work...for awhile, in your current environment. But it would be more stable to just specify desired profile name (from env-vars and/or command-line parameters.) This is preferred, because region and tokens are inherited from profile. – Slippage

Answer 4

What works for me is the following:

import boto 3


session = boto3.Session(profile_name="sso_profile_name")
session.resource("whatever")

using boto3==1.20.18.

This would work if you had previously configured SSO for aws ie. aws configure sso.

Interestingly enough, I don't have to go through this if I use ipython, I just aws sso login beforehand and then call boto3.Session(). I am trying to figure out whether there is something wrong with my approach - I fully agree with what was said above with respect to transparency and although it is a working solution, I am not in love with it.

---

EDIT: there was something wrong and here is how I fixed it:

1. run aws configure sso (as above);
2. install aws-vault - it basically replaces aws sso login --profile <profile-name>;
3. run aws-vault exec <profile-name> to create a sub-shell with AWS credentials exported to environment variables.

Doing so, it is possible to run any boto3 command both interactively (eg. iPython) and from a script, as in my case. Therefore, the snippet above simply becomes:

import boto 3


session = boto3.Session()
session.resource("whatever")

Here for further details on AWS vault.

Hartzell

Answer 5

A well-formed boto3-based script should transparently authenticate based on profile name. It is not recommended to handle the cached files or keys or tokens yourself, since the official code methods might change in the future. To see the state of your profile(s), run aws configure list --examples:

$ aws configure list --profile=sso

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      sso           manual    --profile

The SSO session associated with this profile has expired or is otherwise invalid.
To refresh this SSO session run aws sso login with the corresponding profile.

$ aws configure list --profile=old

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                      old           manual    --profile
access_key     ****************3DSx shared-credentials-file    
secret_key     ****************sX64 shared-credentials-file    
    region                us-west-1              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

Slippage

Answer 6

If you have ever executed aws sso configure command in the terminal before and followed the prompts, then you should have something like the following in your ~/.aws/config file.

[profile <profile_name>]
sso_session = <sso_session_name>
sso_account_id = <account_id>
sso_role_name = <role_name>
region = <region>
output = text
[sso-session <sso_session_name>]
sso_start_url = <sso_start_url>
sso_region = <region>
sso_registration_scopes = sso:account:access

After logging in the terminal using aws sso login --profile <profile_name> where <profile_name> must match one of the profiles in the ~/.aws/config file above, you need to set AWS_PROFILE environment variable to the same profile name above. If you want to set the environment variable only for the current terminal (where you run your Python script), run export AWS_PROFILE=<profile_name> in the terminal. Or if it is a profile that you often use, add the same export statement above to something like ~/.bashrc file. The good thing about setting the environment variable is that you do not need to edit your code.

Hollins