Gitlab runner unable to run aws commands

Question

I am trying to run GitLab's job using their shared Runners,
I've created a .gitlab-ci.yml and kept it at my project's root,
Configured AWS creds as the environment variables -

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION

under Settings -> CI / CD -> Variables

Double checked the branch I've trying to build from is protected

Following is my .gitlab-ci.yml -

stages:
    - build

build:
    image: python:latest
    stage: build
    script:
            - apt-get update
            - apt-get install -y zip unzip
            - pip install awscli
            - mkdir ~/.aws/
            - touch ~/.aws/credentials
            - pip install awscli
            - printf "[eb-cli]\naws_access_key_id = %s\naws_secret_access_key = %s\nregion = %s\n" "$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" "$AWS_DEFAULT_REGION" >> ~/.aws/credentials
            - bash cicdScript.sh

CICD script has the aws command -

$(aws s3 ls)

But I still get the following error -

Unable to locate credentials. You can configure credentials by running "aws configure".

Reference -
https://medium.com/faun/continuous-static-upload-to-aws-s3-using-gitlab-runners-17f0260a5af2

Dabber

It's as the error says, it can't find the aws command. Which image are you using for this? Can you post your .gitlab-ci.yml ? – Vesuvius

seems like you just don't have AWS command line tools installed on your GitLab host – Inessive

@Vesuvius Added the .gitlab-ci.yml and env variable setup – Dabber

Uncertain why you're using [eb-cli] in the credentials file, maybe try using [default] instead? Although it's unsecure, what happens when you try cat ~/.aws/credentials after the printf? (do revoke your keys after this though if it does work...) – Vesuvius

Answer 1

You can use

- pip install awscli
- aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
- aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
- aws configure set region $AWS_DEFAULT_REGION

instead of

- mkdir ~/.aws/
- touch ~/.aws/credentials
- pip install awscli
- printf "[eb-cli]\naws_access_key_id = %s\naws_secret_access_key = %s\nregion = %s\n" "$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" "$AWS_DEFAULT_REGION" >> ~/.aws/credentials

Bioastronautics

Answer 2

In your code, the profile is set to 'eb-cli' for your credentials

printf "[eb-cli]\naws_access_key_id = %s\naws_secret_access_key = %s\nregion = %s\n" "$AWS_ACCESS_KEY_ID" "$AWS_SECRET_ACCESS_KEY" "$AWS_DEFAULT_REGION" >> ~/.aws/credentials

The command you should use is

aws s3 ls --profile eb-cli

If you dont pass the profile, [default] will be picked. Since default was not configured the issue is caused.

Actaeon

Answer 3

If you want to run aws cli commands in the pipelines, the easiest way is described in the documentation:

• https://docs.gitlab.com/ee/ci/cloud_deployment/

Steps:

1. Sign on to your AWS account.
2. Create an IAM user.
3. Select your user to access its details. Go to Security credentials > Create a new access key.
4. Note the Access key ID and Secret access key.
5. In your GitLab project, go to Settings > CI/CD. Set the following CI/CD variables:

Environment variable name   Value

- AWS_ACCESS_KEY_ID Your Access key ID.
- AWS_SECRET_ACCESS_KEY Your secret access key.
- AWS_DEFAULT_REGION    Your region code. You might want to confirm that the AWS service you intend to use is available in the chosen region.

6. Variables are protected by default. To use GitLab CI/CD with branches or tags that are not protected, clear the Protect variable checkbox.

Finally add in the .gitlab-ci.yml the following:

deploy:
  stage: deploy
  image: registry.gitlab.com/gitlab-org/cloud-deploy/aws-base:latest
  script:
    - aws s3 ...
    - aws create-deployment ...
  environment: production

Mousse