Setting credentials for https git clone in AWS CodeBuild

Question

I am running a CodeBuild on a project that has private requirements stored in CodeCommit.

I need to add a command in buildspec.yml that loads the https git credentials so git clone works when CodeBuild runs pip install.

The build fails with fatal: could not read Username for 'https://git-codecommit.us-west-2.amazonaws.com': No such device or address

Ceram

Answer 1

Since the CodeBuild environment uses an IAM role for credentials (not a username and password), you will need to configure the CodeCommit credential helper in your buildspec:

phases:
  install:
    commands:
      - git config --global credential.helper '!aws codecommit credential-helper $@'
      - git config --global credential.UseHttpPath true

Guardroom

This worked for us with cloning a secondary repository into CodeBuild from GitHub – Global

Answer 2

CodeBuild now provides an easier dial for this buildspec by setting the "git-credential-helper" to yes. Documentation @ https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax

Jinja

This is a helpful answer, but FWIW I find that documentation useless. It doesn’t give any context, guidance, etc. I have no idea how to actually use the feature. – Waistband

this doesnt seem to work with github, I still get permission errors – Beefburger

Answer 3

• List item

So I was looking for a method to do this as well. I figured out two ways to clone the repo without the use of assume roles. What I am gathering from your post you want to clone the repo via the buildspec.yml

The first option as mentioned in an earlier post is to use the native function that is offered by Codebuild. The caveat is that it's limited to the AWS account you are in (at least that's where my research has led me without using Codepipeline). I am providing a sample for you to review as well.

I will also assume that most of you have worked with AWS Codecommit before and know how to set up users to connect to repos. If you haven't please visit this page and get familiar with Codecommit configurations. Links are listed below that can help you with this.

Using One AWS Account to clone Codecommit repository inside that AWS Account:

version: 0.2
env:

  git-credential-helper: yes
    
phases:
  install:
    commands:
      - echo "STARTING PYTHON INSTALLATION"
      - "curl -s -qL -o python.tgz https://www.python.org/ftp/python/${PY_VERSION}/Python-${PY_VERSION}.tgz"
      - "tar xf python.tgz -C /usr/bin/"
      - "python --version"
      - python -m pip install -U pip
      - pip install git-remote-codecommit

  pre_build:
    commands:
      - aws --version
      - git --version

      # Clone directories
      - echo CLONE DIRECTORIES
      - mkdir /usr/bin/repo
      - cd /usr/bin/repo

      #Leveraging git remote clone for codecommit
      - git clone codecommit://your-repo1-name new-repo1-name 

  build:
    commands: 
      - cd /usr/bin/new-repo1-name 
      - do your git commands from here

The key for this to work is to make sure certain settings are enabled.

• git-credential-helper: yes
• Python
• pip install git-remote-codecommit
• git clone codecommit://your-repo1-name. (The command must be exactly like this)

Again I reiterate I have only made this work within one AWS Account. To date, I am not able to make this work cross-account without leveraging other AWS Services. To avoid leveraging other services I was able to put this together which creates an AWS Codecommit user that can use SSH. For this example, I stored my ssh private key in the parameter store as well as the ssh key ID. There are other methods I will share that can leverage S3 buckets which I will attach but the example below is to build the RSA and the config on the fly.

The SSH method to connect to a different AWS account Codecommit repository

version: 0.2
env:

  parameter-store:
    ssh_key: variable_ssh_key
    cc_user: variable_codecommit_user

  git-credential-helper: yes
    
phases:
  install:
    commands:
      - echo UPDATING SSH CLIENT
      - "which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )"
      - echo "STARTING PYTHON INSTALLATION"
      - "curl -s -qL -o python.tgz https://www.python.org/ftp/python/${PY_VERSION}/Python-${PY_VERSION}.tgz"
      - "tar xf python.tgz -C /usr/bin/"
      - "python --version"
      - python -m pip install -U pip

  pre_build:
    commands:
      - aws --version
      - git --version
      
      # Adds a private SSH key to allow us to clone or npm install Git repositories
      - eval $(ssh-agent -s)
      - mkdir -p ~/.ssh

      # Configure SSH Key
      #- ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa <<< y. #generate a new ssh key on demand
      - echo "$ssh_key" > ~/.ssh/id_rsa
      - cd ~/.ssh/
      - cat id_rsa
      - |
        echo "Multiline command"
        cat > ~/.ssh/config <<EOL
        Host host-unique-name
            Hostname git-codecommit.us-east-1.amazonaws.com
            User ${cc_user}
            IdentityFile ~/.ssh/id_rsa
        EOL
      - cat ~/.ssh/config
      
      # Configure SSH Permissions
      - chmod 700 ~/.ssh
      - chmod 600 ~/.ssh/config
      - chmod 600 ~/.ssh/id_rsa
      - ssh-keyscan -t rsa1,rsa,dsa git-codecommit.us-east-1.amazonaws.com >> ~/.ssh/known_hosts

      # Clone directories
      - echo CLONE DIRECTORIES
      - mkdir /usr/bin/repo
      - cd /usr/bin/repo

      #leveraging typical git clone
      - git clone ssh://host-unique-name/v1/repos/your-repo1-name

  build:
    commands: 
      - cd /usr/bin/new-repo1-name 
      - do your git commands from here

As you can see this will create the ssh key and allow Codebuild to clone the repositories locally. Please note I am adding a link for a similar example that uses S3 to download the RSA_ID.

Sample SSH clone S3 Bucket: https://gist.github.com/gemmadlou/36deec54dea3defbdd8cbd6574e0261d

The key for this to work is to make sure certain settings are enabled.

Env Phase

• git-credential-helper: yes

Install Phase

• "which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )"
• Python
• pip install git-remote-codecommit

pre_build:

• Create .ssh directory
• add rsa private key
• Create AWS Config file
• configure permissions for ssh key and config file
• Create known_host file (critically important)

Here are additional links to reference for items that helped me put this together:

• Setup steps for HTTPS Connections to AWS Codecommit
• AWS Cross account access to Codecommit repo with roles only
• Fetch AWS Secret keys from Bash script
• CWN connecting to codecommit repos using AWS Pipeline (codecommit, codebuild, codepipeline)
• Codecommit setup ssh key

Bonn

I provided in this comment (https://mcmap.net/q/536369/-how-to-get-aws-codebuild-to-perform-full-clone) a solution for cross-account codeCommit git clone using STS to assume a role in the CodeCommit account. – Triplett

Answer 4

You can use de credentils helper from git, you can put the following commands into builspec pipeline

phases:
  install:
    commands:    
    - echo "https://username:[email protected]" > ~/.git-credentials
    - git config credential.helper 'store'

Git documentation https://git-scm.com/docs/git-credential-store

Obstreperous

This solution exposes your Bitbucket's username and password – Neutralization