How can I quickly and effectively debug CloudFormation templates?

Question

CloudFormation is a powerful AWS offering that allows the programmatic creation of AWS resource stacks, such as the web tier of an application, a high performance computing cluster, or an entire application stack, with a single API call. It is immensely powerful. Using it is surely considered a good AWS practice, especially when it's combined with Chef, Puppet, or cloud-init. Debugging it drives me to vice.

Take a production example: The stock mongodb cluster templates won't work for me. I don't particularly know why. I'm sure it's something simple as it almost always is. My problem isn't that I can't figure out what's wrong. It's that it takes the stack between 20 and 30 minutes to fail, and then another three or four minutes to delete, assuming it deletes the resources properly at all.

What am I missing? I know about the --disable-rollback flag and use it like oxygen. I learned long ago to wrap exit messages with cfn-signal and to throw them like ballast off a sinking ship. How can I make the template debugging process faster, or am I stuck forever noticing my mistakes half an hour after I make them?

Delvecchio

Even typos in the JSON sometimes don't get caught until 10+ minutes into the build process as it doesn't notice that a particular resource type doesn't have a particular attribute name until it tries to start that resource. – Hefner

My favorites in no particular order: when you switch resource contexts and encounter almost-but-not-quite-identical attributes, forgetting to add an escaped quotation mark inside a user script declaration, and anything to do with VolumeAttachments, since they occasionally fail for no reason at all. – Delvecchio

+1 for mentioning "disabling auto-rollback"--this was awesome for getting better error messages for substacks – Sclerosis

I've found that mistakes in policy segments will cause things like BucketPolicy to get stuck in CREATE stage forever - if something is stuck in CREATE start there. – Adanadana

@Delvecchio could you please clarify your question by editing it to briefly explain the benefit of using --disable-rollback and cfn-signal? – Dominate

Answer 1

Use the aws cloudformation validate-template command in the AWS CLI tool.

Connors

This is also available in the unified cli docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/… – Caprifig

By the way this is aws cloudformation validate-template in the new AWS CLI tools. – Delvecchio

For anyone coming across this question in 2017+, this is now aws cloudformation validate-template, and it only validates whether your template is valid JSON or YAML, not whether your keys and values are correct (for example doesn't check for typos in keys). – Edge

For some reason it seems to ignore the line-length, e.g. Property validation failure: [Length of value {XYZ} for property {/RepositoryDescription} is greater than maximum allowed length {100}]. According the validate-template command this was not an issue, but the UI returns this error. – Hildegaard

If this tool does only file format validation isn't; jsonlint or yamllint sufficient enough? As well this tool has the limitation of file size of 51,200 bytes. – Rasbora

Didn't realise you needed valid credentials to use this command... Any alternatives? – Arenas

If using this dont forget to format your file path correctly as ` --template-body file://`. – Beestings

As of February 2024, it is no longer true that this command "only checks for valid JSON" - it does actual verification. – Holsinger

Answer 2

Another option, a year later, is to abstract these templates to a 3rd party library, such as troposphere. That library constructs the JSON payload for you, and does a lot of validation along the way. This also solves the "Wow managing a 1000-line JSON file sure is sad" problem.

Delvecchio

"Wow managing a 1000-line JSON file sure is sad" - surely this is why nested stacks was created? ;-) Although I do concur - Troposphere is aws-some! – Little

Answer 3

How can I make the template debugging process faster, or am I stuck forever noticing my mistakes half an hour after I make them?

Here are a few best-practice suggestions, focusing specifically on improving the iteration speed of complex CloudFormation-template development:

Use CloudFormation tools to validate templates and stack updates

AWS has already outlined these in its own Best Practices document, so I won't repeat them:

• Validate Templates Before Using Them
• Create Change Sets Before Updating Your Stacks

The point of this step is to catch obvious syntax or logical errors before actually performing a Stack creation/update.

Test Resources in isolation

Before using any individual CloudFormation Resource in a complex Stack, make sure you thoroughly understand the full extent of that Resource's creation/update/delete behavior, including any limits on usage and typical startup/teardown times, by testing their behavior in smaller, standalone Stacks first.

• If you are developing or using any third-party Custom Resources, write unit tests using appropriate libraries for the language platform, to make sure the application logic behaves as expected across all use-cases.
• Be aware that the amount of time for an individual Resource to create/update/delete can vary widely between Resource Types, depending on the behavior of the underlying API calls. For example, a complex AWS::CloudFront::Distribution resource can sometimes take 30-60 minutes to create/update/delete, while an AWS::EC2::SecurityGroup updates in seconds.
• Individual Resources may have bugs/issues/limitations in their implementation, which are much easier to debug and develop workarounds for when tested in isolation, rather than within a much larger Stack. Keep in mind limitations such as AWS Service Limits depending on your individual AWS Account settings, or Region Availability of services depending on the Region within which you create your Stack.

Build complicated stacks in small increments

When performing a Stack creation/update, a failure in any single Resource will cause the Stack to rollback the entire set of Resource changes, which can unnecessarily destroy other successfully-created Resources and take a very long time when building a complicated stack with a long dependency-graph of associated Resources.

The solution to this is to build your Stack incrementally in smaller Update batches, adding Resources one (or a few) at a time. This way, if/when a failure occurs in a resource creation/update, the rollback doesn't cause your entire Stack's resources to be destroyed, just the set of Resources changed in the latest Update.

Monitor the progress of stack updates

Be sure to Monitor the Progress of your Stack Update by viewing the stack's events while a creation/update is performed. This will be the starting-point for debugging further issues with individual resources.

Publicly

About CloudFront and how long it takes, are you aware of anyway to have the stack be in completed state, without waiting, i want to get the outputs, but dont need to wait for the distribution to be deployed in my case – Jolynjolynn

Answer 4

Have you looked at the AWS CloudFormation Template Editor that is included in the AWS Toolkit for Eclipse? It has syntax highlighting, statement completion, and deployment to AWS CloudFormation.

Off

"AWS Toolkit for Visual Studio" has been indispensable for me. – Liddle

Answer 5

AWS CloudFormation linter provides additional static analysis beyond aws cloudformation validate-template

It informs which resource types and instance types are unavailable in certain regions, validates property values against allowed values, catches circular resource dependencies, syntax errors, template limits, and much more

Beyond the CLI, a popular mechanism to run the linter is installing an editor plugin like the Visual Studio Code extension which runs every file save

Other mechanisms like pre-commit Git hooks are described here

Propraetor

Answer 6

Late to the party but I might also add that it is worthwhile spending a bit of time configuring and learning your editor. I know that sounds laughably basic as an answer but try it.

In my case, with vim, I performed much better once I took some time installing json syntax plugins, and also (finally) understood folding techniques to navigate large CF files easily. Mine now suggests typos (commas where they shouldn't be etc) and the color highlighting saves a lot of time giving clear visual clues.

This might help mitigate syntax errors, but in-template logical errors are better fixed by other tools. Hopefully one day there will be a "preview" mode on CF.

Tew

Not a laughable suggestion at all. I'm not sure I could code without syntax highlighting. – Delvecchio

there is a preview for CFN that shows all resources you're about to create, and it will also tell you how much your stack will cost. I'm using the Java API, so I'm not sure that it's available on the CLI, but try this: link – Recluse

Answer 7

For JetBrains IDEs (IntelliJ IDEA PhpStorm WebStorm PyCharm RubyMine AppCode CLion Gogland DataGrip Rider Android Studio ), there is at AWS CloudFormation plugin that supports deep checking of JSON and YAML CFN templates

Vanessa

Answer 8

If you are dealing with EC2 machines, then I would recommend you to login to the EC2 machine and tail the boot.log file (/var/log/boot.log in RHEL6/Centos). This file gets updated with all your shell activities (activities like: installation, downloading files, copying files etc.).

Also, use editors like http://www.jsoneditoronline.org/ to get a TREE representation of your JSON. This helps you to check the order of JSON elements.

And when you update files always use tools like http://www.git-tower.com/blog/diff-tools-mac/ or an actual version control system to ensure that you did not accidentally change something which might break your script.

Pass

Answer 9

In addition to the AWS CLI aws cloudformation validate-template command there is a node-based cfn-check tool that does deeper validation.

Vanessa

Answer 10

A recent new feature added to Cloudformation this past December was the addition of additional Parameter Types. These new Types allow your templates to perform stronger data checking, and also can "fail-fast" when creating resources and nested Cloudformation stacks. You also have the ability to provide nicer human-readable custom error messages when invalid values are passed in using the new ConstraintDescription attribute.

The new types are especially helpful when dealing with various VPC resources. You can ensure that Parameters for your templates are the correct type, and are explicit about expecting a single value vs. a List.

For example:

"Parameters" : {
  "SingleGroup": { "Type": "AWS::EC2::SecurityGroup::Id", ...},
  "GroupList": {"Type": "List<AWS::EC2::SecurityGroup::Id>", ...}
}

Whoops

Answer 11

Please checkout my cloudformation validator at https://pypi.org/project/cloudformation-validator/

This will validate the schema and then validate again a list of rules, and allow for custom rules. I also allows for easy integration with deployment tools.

Hairless

Answer 12

You can also make use of the CloudFormation Designer available from amazon here: https://console.aws.amazon.com/cloudformation/designer/home?region=us-east-1

Simply paste your template (JSON) on the "Template" pane and then click on the tick symbol to validate your template. Any errors will show up in the "Error" pane.

Hope this helps.

Decrepit

Answer 13

If you want a quick and easy way to validate multiple templates at once within a given path. Just use this python script I wrote :).

Example use: python cf-validator.py --templates_path=../../environment-stack/

import boto3
import botocore
import argparse
import sys
import glob, os

def cli_parms_setup(arguments):
    """Setups inputs for parms"""
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('--templates_path', required=True, dest='templates_path')
    return parser.parse_args(arguments)


def get_template_file_names(templates_path):
    """Finds all yaml template within given path"""
    template_file_names = []
    for file in glob.glob(templates_path + "/**/*.yaml", recursive=True):
        template_file_names.append(file)
    print(template_file_names)
    return template_file_names

def cfn_validator(path, templates_to_vaildate):
    """Takes in list of templates to validate"""
    client = boto3.client('cloudformation')
    for yaml_template in templates_to_vaildate:
        cwd = os.getcwd()
        cf_template = open(cwd + path + yaml_template).read()
        try:
            client.validate_template(
                TemplateBody=cf_template
            )
        except botocore.exceptions.ClientError as e:
            print("botocore.exceptions.ClientError has occurred with template " + yaml_template)
            raise e

if __name__ == '__main__':
    cli_parms = cli_parms_setup(sys.argv[1:])
    template_file_names = get_template_file_names(cli_parms.templates_path)
    cfn_validator(cli_parms.templates_path,template_file_names)

Kristiekristien

Answer 14

The most quick and effective method to debug CloudFormation template is AWS CloudFormation Linter i.e. cfn-lint followed by AWS CLI. Both are officially provided by AWS.

AWS CloudFormation Linter (cfn-lint)

cfn-lint helps lint and validate CloudFormation templates (JSON or YAML) against the CloudFormation Resource Specification along with more checks. It also checks for valid values for resource properties and CloudFormation best practices. You can run cfn-lint <file-name>.

Installation guide: https://github.com/aws-cloudformation/cfn-lint?tab=readme-ov-file#install

cfn-lint is also available for multiple IDEs, thanks to the AWS team:

• IntelliJ / PyCharm / Rider / WebStorm: https://plugins.jetbrains.com/plugin/10973-cfn-lint
• Visual Studio Code: https://marketplace.visualstudio.com/items?itemName=kddejong.vscode-cfn-lint

AWS CLI

You can use aws cloudformation validate-template <file-name> to validate your CloudFormation template file. However, it's a lesser linting tool than AWS CloudFormation Linter considering it only checks for formatting errors or issues related to JSON or YAML syntax and specification. On the contrary, AWS CloudFormation Linter lints against a lot more checks and potential issues.

Installation guide: https://aws.amazon.com/cli/

Bonus Tip: You can include both AWS CloudFormation Linter (cfn-lint) and AWS CLI in your build pipeline to check and lint CloudFormation templates while you or your peer developers write and push CloudFormation code. This helps minimise accidental pushes having poor code.

Sells