aws s3 ls: An HTTP Client raised an unhandled exception: Invalid header value

Question

I'm trying to implement a pipeline that package and copy Python code to S3 using Gitlab CI. Here is the job that is causing the problem:

package:
  stage: package
  image: python:3.8
  script:
    - apt-get update && apt-get install -y zip unzip jq
    - pip3 install awscli 
    - aws s3 ls
    - ./cicd/scripts/copy_zip_to_s3.sh
  only:
    refs:
      - developer

I want to mention that in the section before_script in .gitlab-ci.yml, I've already exported the AWS credentials (AWS SECRET ACCESS KEY, AWS_ACCESS_KEY_ID, etc) from Gitlab environment variables. I've checked thousands of times my credentials and they are totally correct. I want also to mention that the same script works perfectly for another project under the same group in Gitlab. Here is the error:

$ aws s3 ls
An HTTP Client raised an unhandled exception: Invalid header value b'AWS4-HMAC-SHA256 Credential=AKIAZXXXXXXXXXX\n/2020XX2/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=ab53XX6eb72XXXXXX2152e4XXXX93b104XXXXXXX363b1da6f9XXXXX'
ERROR: Job failed: exit code 1

./cicd/scripts/copy_zip_to_s3.sh do the package and the copy, same error occurs when executing it, that's why I've added a simple aws command aws s3 ls to show that even a simple 'ls' is not working.

Any solutions, please? Thank you all in advance.

Disobey

Looks like somehow you included a newline in your access key variable. – Daddy

Copy + paste issues like that happen sometimes. Hard to detect because most displays will not actually show the erroneous whitespace characters. – Daddy

Answer 1

This was because of an additional line added to AWS ACCESS KEY variable.

Disobey

Something similar happens with me because I created an environment file in WINDOWS generating the end of the line with '\r\n' and, when trying to run the AWS CLI on Linux, this reported problem appeared. – Backswept

that was my problem too. thanks! – Jalousie

Answer 2

I had similar issue when running a bash script on Cygwin in Windows. The fix was removing the \r\n from the end of the values I was putting into environment variables.

Here's my whole script if anyone is interested. It assumes a new AWS role, sets those creds into environment variables, then opens a new bash shell which will respect those set variables.

#!/bin/bash

hash aws 2>/dev/null
if [ $? -ne 0 ]; then
    echo >&2 "'aws' command line tool required, but not installed. Aborting.";
    exit 1;
fi;

unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_SESSION_TOKEN
ROLEID=123918273981723
TARGETARN="arn:aws:iam::${ROLEID}:role/OrganizationAccountAccessRole"

COMMAND="aws sts assume-role --role-arn $TARGETARN --role-session-name damien_was_here"

RESULT=$($COMMAND) 

#the calls to tr -d \r\n are the important part with regards to this question.

AccessKeyId=$(echo -n "$RESULT" | jq -r '.Credentials.AccessKeyId' | tr -d '\r\n')
SecretAcessKey=$(echo -n "$RESULT" | jq -r '.Credentials.SecretAccessKey' | tr -d '\r\n')
SessionToken=$(echo -n "$RESULT" | jq -r '.Credentials.SessionToken' | tr -d '\r\n')

export AWS_ACCESS_KEY_ID=$AccessKeyId
export AWS_SECRET_ACCESS_KEY=$SecretAcessKey
export AWS_SESSION_TOKEN=$SessionToken

echo Running a new bash shell with the environment variable set. 
bash

Counterproductive

Answer 3

In my case it was an additional character '\n' created by the "echo" command during Base64 encoding of the AWS_ACCESS_KEY_ID.

In order to fix this use '-n' argument with the 'echo' command, e.g.

$ echo -n AKIAJGN2PSI2F5OHJSQQ | base64
QUtJQUpHTjJQU0kyRjVPSEpTUVE=

Bechler