CloudFormation doesn't deploy to API gateway stages on update

Question

When I run CloudFormation deploy using a template with API Gateway resources, the first time I run it, it creates and deploys to stages. The subsequent times I run it, it updates the resources but doesn't deploy to stages.

Is that behaviour as intended? If yes, how'd I get it to deploy to stages whenever it updates?

(Terraform mentions a similar issue: https://github.com/hashicorp/terraform/issues/6613)

Gum

answered in https://mcmap.net/q/297688/-how-to-automate-api-gateway-deployment-via-sam-deploy – Gynous

Answer 1

Seems like there is no way to easily create a new Deployment whenever one of your Cloudformation Resources changes.

One way to work around that would be to use a Lambda-backed Custom Resource (see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html).

The Lambda should create the new Deployment, only if one of your Resources has been updated. To determine if one of your Resources has been updated,
you will probably have to implement custom logic around this API call: http://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_DescribeStackEvents.html

In order to trigger updates on your Custom Resource, I suggest you supply a Cloudformation Parameter that will be used to force an update of your Custom Resource (e.g. the current time, or a version number).

Note that you will have to add a DependsOn clause to your Custom Resource that will include all Resources relevant to your API. Otherwise, your deployment might be created before all your API Resources are updated.

Hope this helps.

Barbarous

Certainly helps. :) Amazon suggested another way - can share if interested. – Gum

@Gum certainly would like to know! – Barbarous

@Gum Running into this issue as well, what is it that you found from Amazon? – Elaterin

AWS support suggested using Serverless, which is excellent for this. – Gum

Any source or link that that we can check? I'm also interested on this. – Teeth

@Gum I'm confused... Serverless is just Amazon's offering of Lambda, API Gateway, S3, and DynamoDB as an alternative to server-based approaches (like EC2). I don't quite see how that is "another way." – Talmudist

Just a guess on Amazon meant when they suggested Serverless - Serverless framework. it works when you add/update respources/methods. serverless.com – Slayton

@siraj-pathan has a lovely option with the AWS CLI, which was perfect for me what that's what kicks off my Cloudformation changes...https://mcmap.net/q/295946/-cloudformation-doesn-39-t-deploy-to-api-gateway-stages-on-update – Expediency

Answer 2

CloudFormation in Amazon's words is:

AWS CloudFormation takes care of provisioning and configuring those resources for you http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html

Redeployment of APIs is not a provisioning task... It is a promotion activity which is part of a stage in your software release process.

AWS CodePipeline is a continuous delivery service you can use to model, visualize, and automate the steps required to release your software. http://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html

CodePipeline also supports execution of Lambda functions from Actions in the pipeline. So, as advised before, create a Lambda function to deploy your API but call it from Codepipeline instead of CloudFormation.

Consult this page for details: http://docs.aws.amazon.com/codepipeline/latest/userguide/actions-invoke-lambda-function.html

Selway

Answer 3

When your template specifies a deployment, CloudFormation will create that deployment only if it doesn't already exist. When you attempt to run it again, it observes that the deployment still exists so it won't recreate it, thus no deployment. You need a new resource id for the deployment so that it will create a new deployment. Read this for more information: https://currentlyunnamed-theclassic.blogspot.com/2018/12/mastering-cloudformation-for-api.html

Tedtedd

As @dontpanic42 suggested "keep in mind that if you aren't generating your templates with something that can insert a valid timestamp in place of $TIMESTAMP$, you must update that manually with a time stamp or otherwise unique ID" – Tedtedd

I had used Python to replace this with current timestamp (without symbols) but will be soon be switching to CDK to generate my template. – Tedtedd

Answer 4

I was using above approach but it looks to complicated to me just to deploy API gateway. If we are changing name of the resources then it takes time to delete and recreate the resources which increases down time for you application.

I'm following below approach to deploy API gateway to the stage using AWS CLI and it is not affecting the deployment with Cloudformation stack.

What I'm doing is, running below AWS CLI command after deployment is completed for API Gateway. It will update the existing stage with latest updates.

aws apigateway create-deployment --rest-api-id tztstixfwj --stage-name stg --description 'Deployed from CLI'

Preoccupation

I switched out the API ID and stage name, added this line to my build commands I'm using in code pipeline, and BAM it started working! 4 hours of debugging and fiddling resolved in 2 minutes. 10/10 – Weinhardt

Thanks! I also went through a similar situation and this approach came to my mind. – Preoccupation

Perfect, I'm running cloudformation, but I deploy it from CI via the AWS CLI, so any other CLI command to force an API gateway redeploy when I've updated something important (ApiKeyRequired on a ::Method anybody ?) is perfect - I was on this track but thanks for saving me some time! FYI aws apigateway update-deployment... does not in fact (currently) work. – Expediency

Nice workaround! And, when deploying it from CI/CD via AWS CLI, you can actually retrieve the --rest-api-id value starting from the API Gateway name: aws apigateway get-rest-apis --no-paginate --query "items[?name == 'my-rest-api-name'].id | [0] – Benelux

Answer 5

The answer here is to use the AutoDeploy property of the Stage:

  Stage:
    Type: AWS::ApiGatewayV2::Stage
    Properties:
      StageName: v1
      Description: 'API Version 1'
      ApiId: !Ref: myApi
      AutoDeploy: true

Note that the 'DeploymentId' property must be unspecified when using 'AutoDeploy'.

See documentation, here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigatewayv2-stage.html

Schear

Is it possible to do this with V1 of Api Gateway? I have not found anything so far. – Heathenize

This is only applicable to V2 API's (so doesn't include Private REST APIs for example) – Canto

Private REST APIs are V1. This solution will be applicable only for V2 APIs such as HTTP and Websocket APIs. – Mohamed

This is a solution but not the solution. ApiGatewayV2 does not support all of the features that a V1 REST API does. Examples: custom gateway responses, API keys, usage plans, canary releases, mocks, edge optimisation, x-ray, and more. See docs.aws.amazon.com/apigateway/latest/developerguide/… – Theorize

Answer 6

From the blogspot post linked by TheClassic (best answer so far!), you have to keep in mind that if you aren't generating your templates with something that can insert a valid timestamp in place of $TIMESTAMP$, you must update that manually with a time stamp or otherwise unique ID. Here is my functional example, it successfully deletes the existing deployment and creates a new one, but I will have to update those unique values manually when I want to create another change set:

    rDeployment05012019355:
        Type: AWS::ApiGateway::Deployment
        DependsOn: rApiGetMethod
        Properties:
            RestApiId:
                Fn::ImportValue: 
                    !Sub '${pApiCoreStackName}-RestApi'
            StageName: !Ref pStageName

    rCustomDomainPath:
        Type: AWS::ApiGateway::BasePathMapping
        DependsOn: [rDeployment05012019355]
        Properties:
            BasePath: !Ref pPathPart
            Stage: !Ref pStageName
            DomainName:
                Fn::ImportValue: 
                    !Sub '${pApiCoreStackName}-CustomDomainName'
            RestApiId:
                Fn::ImportValue: 
                    !Sub '${pApiCoreStackName}-RestApi'

Selfeffacing

https://mcmap.net/q/297689/-set-the-ecs-cloudformation-update-stack-timeout This solution has been a huge help for managing ECS Task timeouts - I've imported it here to solve the APIGW deployment issue by running a transform on my Cloudformation template which will automatically update the resource name with a parameter change, signaling the transformation by updating the string parameter with the current epoch time. – Superabundant

Good callout. When managing test and prod environments, you may even want to manage separate deployments for each stage. That way you can deploy first to test or staging, run some tests, and deploy to production in a separate stack update when everything passes. – Triazine

Answer 7

I may be late, but here are the options which which you do a redeployment if a API resources changes, may be helpful to people who still looking for options -

1. Try AutoDeploy to true. If you are using V2 version of deployment. Note that you need to have APIGW created through V2. V1 and V2 are not compatible to each other. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigatewayv2-stage.html#cfn-apigatewayv2-stage-autodeploy

2. Lambda backed custom resource, Lambda inturn call createDeployment API - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html

3. CodePipeline that has an action that calls a Lambda Function much like the Custom Resource would - https://docs.aws.amazon.com/codepipeline/latest/userguide/actions-invoke-lambda-function.html

4. SAM(Serverless Application Model) follows a similar syntax to CloudFormation which simplifies the resource creation into abstractions and uses those to build and deploy a normal CloudFormation template. https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html

5. If you are using any abstraction layer to cloudformation like Sceptre, you can have a hook to call createDeployment after any-update to the resource https://sceptre.cloudreach.com/2.3.0/docs/hooks.html

I gone with third option since I kept using Sceptre for Cloudformation deployment. Implementing hooks in sceptre is easy as well.

Chazan

Does SAM take care of deploying API Gateway after each change? – Cobaltic

@Cobaltic no, it does not. – Hanselka

Answer 8

Reading through this article, I did not come to a conclusion right away, as the information here is stretched through multiple sources. I try to sum up all the findings from here (and linked source) as my personal testing to help others avoid the hunt.

Important to know is that each API always has a dedicated URL. The associated stages only get a separate suffix. Updating the deployment does not change the URL, recreating the API does.

API
├─ RestAPI (incl. Resource, Methods etc)
├─ Deployment
    ├─ Stage - v1 https://6s...com/v1
    ├─ Stage - v2 https://6s...com/v2

Relation stage and deployment:

To deploy AWS API Gateway through CloudFormation (Cfn) you need a RestApi-Cfn-Resource and a Deployment-Cfn-Resource. If you give the Deployment-Resource a stage name, the deployment automatically creates a deployment on top of the "normal" creation. If you leave this out, the API is created without any stage. Either way, if you have a deployment, you can add n-stages to a deployment by linking the two, but a stage and its API always has only one deployment.

Updating simple API:

Now if you want to update this "simple API" just consisting of a RestAPI plus a deployment you face the issue, that if the deployment has a stage name - it can not be updated as it already "exists". To detect that the deployment has to be updated in the first place, you have to either add a timestamp or hash to the deployment resource name in CloudFormation else there is even no update triggered.

Solving the deployment update:

To now enable updating the deployment, you have to split deployment and stage up into separate Cfn-Resources. Meaning, you remove the stage name from the Deployment-Cfn-Resource and create a new Stage-Cfn-Resource which references the deployment resource. This way you can update the deployment. Still, the stage - the part you reference via URL - is not automatically updated.

Propagating the update from the deployment to your stages:

Now that we can update the deployment - aka the blueprint of the API - we can propagate the change to its respective stage. This step AS OF MY KNOWLEDGE is not possible using CloudFormation. Therefore, to trigger the update you either need to add a "custom resource" our you do it manually. Other "none" CloudFormation ways are summed up by @Athi's answer above, but no solution for me as I want to limit the used tooling.

If anybody has an example for the Lambda update, please feel free to ping me - then I would add it here. The links I found so far only reference a plain template.

I hope this helped others understanding the context a bit better.

Sources:

• Problem description with Cfn-template, 2
• Adding timestamp to deployment resource, 2
• Using CodePipeline as a solution
• Related question and CLI update answer
• Related terraform issue
• Related AWS forum thread

Settlement

Answer 9

If you have something to do the $TIMESTAMP$ replacement, I'd probably go with that as it's cleaner and you don't have to do any manual API Gateway management.

I have found that the other solutions posted here mostly do the job with one major caveat - you can't manage your Stage and Deployment separately in CloudFormation because whenever you deploy your API Gateway, you have some sort of downtime between when you deploy the API and when the secondary process (custom resource / lambda, code pipeline, what have you) creates your new deployment. This downtime is because CloudFormation only ever has the initial deployment tied to the Stage. So when you make a change to the Stage and deploy, it reverts back to the initial deployment until your secondary process creates your new deployment.

*** Note that if you are specifying a StageName on your Deployment resource, and not explicitly managing a Stage resource, the other solutions will work.

In my case, I don't have that $TIMESTAMP$ replacement piece, and I needed to manage my Stage separately so I could do things like enable caching, so I had to find another way. So the workflow and relevant CF pieces are as follows

• Before triggering the CF update, see if the stack you're about to update already exists. Set stack_exists: true|false

• Pass that stack_exists variable in to your CF template(s), all the way down to the stack that creates the Deployment and Stage

• The following condition:

Conditions:
  StackExists: !Equals [!Ref StackAlreadyExists, "True"]

• The following Deployment and Stage:

  # Only used for initial creation, secondary process re-creates this
  Deployment:
    DeletionPolicy: Retain
    Type: AWS::ApiGateway::Deployment
    Properties:
      Description: "Initial deployment"
      RestApiId: ...

  Stage:
    Type: AWS::ApiGateway::Stage
    Properties:
      DeploymentId: !If
        - StackExists
        - !Ref AWS::NoValue
        - !Ref Deployment
      RestApiId: ...
      StageName: ...

• Secondary process that does the following:

# looks up `apiId` and `stageName` and sets variables

CURRENT_DEPLOYMENT_ID=$(aws apigateway get-stage --rest-api-id <apiId> --stage-name <stageName> --query 'deploymentId' --output text)
aws apigateway create-deployment --rest-api-id <apiId> --stage-name <stageName>
aws apigateway delete-deployment --rest-api-id <apiId> --deployment-id ${CURRENT_DEPLOYMENT_ID}

Diphase

Answer 10

This worked for me :

cfn.yml

  APIGatewayStage:
    Type: 'AWS::ApiGateway::Stage'
    Properties:
      StageName: !Ref Environment
      DeploymentId: !Ref APIGatewayDeployment$TIMESTAMP$
      RestApiId: !Ref APIGatewayRestAPI
      Variables:
        lambdaAlias: !Ref Environment
      MethodSettings:
      - ResourcePath: '/*'
        DataTraceEnabled: true
        HttpMethod: "*"
        LoggingLevel: INFO      
        MetricsEnabled: true
    DependsOn: 
      - liveLocationsAPIGatewayMethod
      - testJTAPIGatewayMethod


  APIGatewayDeployment$TIMESTAMP$:
    Type: 'AWS::ApiGateway::Deployment'
    Properties:
      RestApiId: !Ref APIGatewayRestAPI
    DependsOn: 
      - liveLocationsAPIGatewayMethod
      - testJTAPIGatewayMethod

bitbucket-pipelines.yml

script: - python3 deploy_api.py

deploy_api.py

import time

file_name = 'infra/cfn.yml'
ts = str(time.time()).split(".")[0]
print(ts)

with open(file_name, 'r') as file :
  filedata = file.read()

filedata = filedata.replace('$TIMESTAMP$', ts)

with open(file_name, 'w') as file:
  file.write(filedata)

========================================================================

Read this for more information: https://currentlyunnamed-theclassic.blogspot.com/2018/12/mastering-cloudformation-for-api.html

Stratovision

Answer 11

Use SAM

AWS::Serverless::Api

This does the deployment for you when it does the Transformation

Aromatic

Saying "use this other tool" is never particularly helpful. – Coxcombry