How do you specify GitHub access token with CodeBuild from CloudFormation

Question

I have been wasting a lot of time trying to get a GitHub web hook setup with CloudFormation. The docs for this process are beyond useless, for example: https://docs.aws.amazon.com/codebuild/latest/userguide/sample-access-tokens.html

$ aws codebuild import-source-credentials --generate-cli-skeleton
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument operation: Invalid choice, valid choices are:

batch-delete-builds                      | batch-get-builds
batch-get-projects                       | create-project
create-webhook                           | delete-project
delete-webhook                           | invalidate-project-cache
list-builds                              | list-builds-for-project
list-curated-environment-images          | list-projects
start-build                              | stop-build
update-project                           | help

My issue is that I cannot find a way to specify a GitHub Access Token for CodeBuild with CloudFormation. I am simply trying to setup a web hook for a github repository to run a simple test suite when a pull request is created, updated, etc. As previously mentioned, I have found a lot of half-baked documentation like https://docs.aws.amazon.com/codebuild/latest/userguide/sample-github-pull-request.html that outlines how to setup a web hook with github, however, when I try and follow these guides I get a no GitHub token error in CloudFormation. Other documents say you need to set the access token from the UI or from the CLI, but the CLI is clearly broken and why would I create the CodeBuild resource in the management console when I am trying to setup cloud formation? I see no where in the example CloudFormation template to include the personal access token from GitHub and the documentation for the Source > Auth element is defined in terms of itself. "The resource value that applies to the specified authorization type" tells me nothing about what this "resource" is. Is this the GitHub personal access token that I have spent the last 8 hours looking for in the docs? I have no idea. I did try sticking my personal access token in that field and I get the same result. "No Access token found, please visit AWS CodeBuild console to connect to GitHub"

The following is my CloudFormation template:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "AWS CodeBuild Template",
    "Parameters": {
    },
    "Resources": {
        "CodeBuildProject": {
            "Type": "AWS::CodeBuild::Project",
            "Properties": {
                "Name": "TestingCodeBuild",
                "Description": "A description about my project",
                "ServiceRole": { "Fn::GetAtt": [ "CodeBuildServiceRole", "Arn" ] },
                "Artifacts": {
                    "Type": "no_artifacts"
                },
                "Environment": {
                    "Type": "LINUX_CONTAINER",
                    "ComputeType": "BUILD_GENERAL1_SMALL",
                    "Image": "ubuntu:bionic",
                    "EnvironmentVariables": [
                      {
                        "Name": "varName",
                        "Value": "varValue"
                      }
                    ]
                },
                "Source": {
                    "Auth" : {
                        "Resource": "WTF IS THIS VALUE, Docs say a resource is a resource for use with the type."
                        "Type" : "OAUTH"
                    },
                    "BuildSpec" : "buildspec.yml",
                    "GitCloneDepth" : 1,
                    "ReportBuildStatus" : true,
                    "Location" : "https://github.com/user/repo.git",
                    "Type" : "GITHUB"
                },
                "Triggers": {
                    "FilterGroups": [
                        [
                            {
                                "Pattern" : "PULL_REQUEST_CREATED, PULL_REQUEST_UPDATED, PULL_REQUEST_REOPENED",
                                "Type" : "EVENT"
                            }
                        ]
                    ],
                    "Webhook" : true
                },
                "TimeoutInMinutes": 10,
                "Tags": [
                    {
                      "Key": "Key1",
                      "Value": "Value1"
                    },
                    {
                      "Key": "Key2",
                      "Value": "Value2"
                    }
                ]
            }
        },
        "CodeBuildServiceRole": {
            "Type": "AWS::IAM::Role",
            "Properties": {
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": [
                                    "codebuild.amazonaws.com"
                                ]
                            },
                            "Action": "sts:AssumeRole"
                        }
                    ]
                },
                "Path": "/",
                "Policies": [
                    {
                        "PolicyName": "CodeBuildAccessPolicies",
                        "PolicyDocument": {
                            "Version": "2012-10-17",
                            "Statement": [
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "codecommit:CancelUploadArchive",
                                        "codecommit:GetBranch",
                                        "codecommit:GetCommit",
                                        "codecommit:GetUploadArchiveStatus",
                                        "codecommit:UploadArchive"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "codedeploy:CreateDeployment",
                                        "codedeploy:GetApplicationRevision",
                                        "codedeploy:GetDeployment",
                                        "codedeploy:GetDeploymentConfig",
                                        "codedeploy:RegisterApplicationRevision"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "codebuild:BatchGetBuilds",
                                        "codebuild:StartBuild"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "devicefarm:ListProjects",
                                        "devicefarm:ListDevicePools",
                                        "devicefarm:GetRun",
                                        "devicefarm:GetUpload",
                                        "devicefarm:CreateUpload",
                                        "devicefarm:ScheduleRun"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "iam:PassRole"
                                    ],
                                    "Resource": "*"
                                },
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "elasticbeanstalk:*",
                                        "ec2:*",
                                        "elasticloadbalancing:*",
                                        "autoscaling:*",
                                        "cloudwatch:*",
                                        "s3:*",
                                        "sns:*",
                                        "cloudformation:*",
                                        "rds:*",
                                        "sqs:*",
                                        "ecs:*"
                                    ],
                                    "Resource": "*"
                                }
                            ]
                        }
                    }
                ]
            }
        }
    }
}

Update So I managed to get it to connect to GitHub by manually creating an unrelated CodeDeploy project called "TempProj" and connecting that to GitHub. Now it can automagically connect to GitHub when you create stacks in Cloud Formation. You can even delete the instance and it will just continue to work.

Widely

Answer 1

You can use AWS Secrets Manager to securely store your GitHub OAuth Token, then you can use a dynamic reference in your CloudFormation template which will resolve to the stored value.

Here's a link to the docs: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager

When creating a secret with Secrets Manager, there are 3 parts to one secret:

• Secret Name (the label for this secret e.g. GitHubToken)
• Secret Key e.g. OAuthToken
• Secret Value (the actual thing you want to store)

The above example would be referenced in your CloudFormation template as:

'{{resolve:secretsmanager:GitHubToken:SecretString:OAuthToken}}'

A fuller snippet from a CloudFormation template for a CodePipeline that will trigger the pipeline to run after every git push to the specified branch of your repo is shown below:

...
MyPipeline:
  Type: AWS::CodePipeline::Pipeline
  Properties:
    Stages:
      -
        Name: GetSource
        Actions:
          -
            Name: GetMyGithubRepoSourceOnPush
            ActionTypeId:
              Category: Source
              Owner: ThirdParty
              Version: 1
              Provider: GitHub
            OutputArtifacts:
              - Name: NameOfArtifactForNextStages
            Configuration:
              Owner: MyGithubUsername
              Repo: MyGithubRepoName
              Branch: MyRepoBranchName
              OAuthToken: '{{resolve:secretsmanager:NameOfSecret:SecretString:KeyOfSecret}}'

I hope that helps.

Bringhurst

Answer 2

AWS::CodeBuild::SourceCredential is a new AWS resource, appeared in CloudFormation Resource Specification v5.1.0, that lets you connect CodeBuild with Github using Github's Personal Access Token (if you do not know how to create it, check out this quick guide).

Quick Example:

AWSTemplateFormatVersion: "2010-09-09"

Resources:
  # This resource allows to connect CodeBuild with Github using Personal Access Token.
  CodeBuildSourceCredential:
    Type: AWS::CodeBuild::SourceCredential
    Properties:
      AuthType: PERSONAL_ACCESS_TOKEN
      ServerType: GITHUB
      Token: "<YOUR-PERSONAL-GITHUB-ACCESS-TOKEN>"

  # CodeBuild resource.
  CodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Source:
        Auth:
          Resource: !Ref CodeBuildSourceCredential
          Type: OAUTH

Tip: Store personal access token in AWS Secrets Manager and fetch it Using Dynamic References to Specify Template Values.

Tightrope

This does work, however what if I am working with a team of developers and I need to deploy CodeBuildSourceCredential for many stacks? I have tried separating this out into a prerequisite stack and referencing it into a SSM parameter but unfortunately SSM only supports the sharing of strings not objects. – Commutation

Your "tip" is actually the preferred solution now, and personally I find it simpler to understand and implement. From the very top of the SourceCredential doc you linked to: > We strongly recommend that you use AWS Secrets Manager to store your credentials. If you use Secrets Manager, you must have secrets in your secrets manager. For more information, see Using Dynamic References to Specify Template Values. – Afoul

Thanks a lot! For some reason, for me it only works if I do create the SourceCredential but if I do not reference in Project.Source.Auth.Resource. If I do not reference the SourceCredential, it does not work. – Imaginative

Answer 3

I've just been through a journey where the original answers here helped greatly.

Since the original question is a couple of years old, things may have evolved, and I'd like to add my findings in 2024.

• Use AWS::CodeBuild::SourceCredential sparingly - see it as a once-off/default access method. It should not be deployed in more than a single stack as it will cause headaches down the road. Deploying more than once will result in errors like:

Failed to call ImportSourceCredentials, reason: Access token with server type GITHUB already exists.

• Instead, create a SecretsManager secret with the correct JSON keys and
• configure CodeBuildProject/Source/Auth accordingly for each project/stack.

SecretsManager

SecretsManager is interesting. Inexperience urged me to figure out more than I've learned from answers here.

• Setting up a secret with a single/custom JSON key and the Personal Access Token (PAT) works and will set the default github/token to use (as coverd in other answers here).
• If you potentially have to support multiple GitHub PATs, having a SecretsManager Secret per PAT will allow you to set up the projects with more flexibility. The truest way seems to be to create the Secret with the correct JSON keys, ServerType, AuthType and Token.

For example, we're setting up a new secret with the name /github/some-org/personal-access-token:

{
    "ServerType":"GITHUB",
    "AuthType":"PERSONAL_ACCESS_TOKEN",
    "Token":"{TokenValue}"
}

The secret will have an ARN like: arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:/github/some-org/personal-access-token-ab12ef

This entire secret is used by CodeBuild when retrieving the source. For example, when ServerType is left out, CodeBuild will fail with an error like: Retrieving Secrets Manager Token Failed with Error: ServerType is required

Elsewhere, this value may be resolved using:

• {{resolve:secretsmanager:/github-some-org/personal-access-token:SecretString}} if the entire Secret is needed
• {{resolve:secretsmanager:/github-some-org/personal-access-token:SecretString:Token}} if only the JSON key called Token is needed.

CodeBuild CloudFormation

Having done that, the Secret's ARN may be used as the Resource value in /Source/Auth/Resource along with /Source/Auth/Type of SECRETS_MANAGER.

...
  CodeBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      ...
      Source:
        Auth:
          Resource: arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:/github/some-org/personal-access-token-ab12ef
          Type: SECRETS_MANAGER
    ...

Note: At the time of writing, the SAM JSON Schema for AWS::CodeBuild::Project SourceAuth stated that the only allowed value for Type is OAUTH. CloudFormation documentation is up to date.

Following this pattern, projects may have different PATs and doesn't depend on the singular github oauth token that's possibly been set up through ClickOps via the console.

Just be mindful of pricing on SecretManager. In the end it all adds up ;-)

Hundley