How to enable CloudWatch logging and X-ray for stepfunction in Terraform?

Question

In AWS console, we can easily enable cloudwatch logging and X-ray for a step function statemachine, but I want my resource fully managed by Terraform, from this page:https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sfn_state_machine

It seems like Terraform doesn't support this at the moment (also see: https://github.com/hashicorp/terraform-provider-aws/issues/12192)

Does anyone know if there is any workaround to achieve this? I'd really like to be able to enable both cloudwatch logs & X-ray from Terraform. I can't find much info on this. Might someone be able to help please? Many thanks.

Guanine

Answer 1

UPDATE : This is feature is recently released 3.27.0 (February 05, 2021)

Corresponding documentation link : sfn_state_machine#logging

You can wrap the command for enabling the logging inside terraform null_resource as it showin the in the linked issueEnabling Step Function Logging To CloudWatch #12192, something like below:

Prerequisite :

aws-cli/2.1.1

Before:

    {
    "stateMachineArn": "arn:aws:states:us-east-1:1234567890:stateMachine:mystatemachine",
    "name": "my-state-machine",
    "status": "ACTIVE",
    "definition": "{\n  \"Comment\": \"A Hello World example of the Amazon States Language using an AWS Lambda Function\",\n  \"StartAt\": \"HelloWorld\",\n  \"States\": {\n    \"HelloWorld\": {\n      \"Type\": \"Pass\",\n      \"End\": true\n    }\n  }\n}\n",
    "roleArn": "arn:aws:iam::1234567890:role/service-role/StepFunctions-MyStateMachine-role-a6146d54",
    "type": "STANDARD",
    "creationDate": 1611682259.919,
    "loggingConfiguration": {
        "level": "OFF",
        "includeExecutionData": false
    }
}

resource "aws_sfn_state_machine" "sfn_state_machine" {
  name     = "mystatemachine"
  role_arn = "arn:aws:iam::1234567890:role/service-role/StepFunctions-MyStateMachine-role-a6146d54"

  definition = <<EOF
{
  "Comment": "A Hello World example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Pass",
      "End": true
    }
  }
}
EOF
}

resource "aws_cloudwatch_log_group" "yada" {
  name = "/aws/vendedlogs/states/myloggroup"
}

resource "null_resource" "enable_step_function_logging" {
      triggers = {
    state_machine_arn  = aws_sfn_state_machine.sfn_state_machine.arn
    logs_params=<<PARAMS
    {
        "level":"ALL",
        "includeExecutionData":true,
        "destinations":[
            {
                "cloudWatchLogsLogGroup":{
                    "logGroupArn":"${aws_cloudwatch_log_group.yada.arn}:*"
                    }
                }
            ]
            }
    PARAMS
    }
  provisioner "local-exec" {
    command = <<EOT
set -euo pipefail

aws stepfunctions update-state-machine --state-machine-arn ${self.triggers.state_machine_arn}  --tracing-configuration enabled=true --logging-configuration='${self.triggers.logs_params}'

EOT
    # interpreter = ["bash"]
  }
}

After:

{
    "stateMachineArn": "arn:aws:states:us-east-1:1234567890:stateMachine:mystatemachine",
    "name": "mystatemachine",
    "status": "ACTIVE",
    "definition": "{\n  \"Comment\": \"A Hello World example of the Amazon States Language using an AWS Lambda Function\",\n  \"StartAt\": \"HelloWorld\",\n  \"States\": {\n    \"HelloWorld\": {\n      \"Type\": \"Pass\",\n      \"End\": true\n    }\n  }\n}\n",
    "roleArn": "arn:aws:iam::1234567890:role/service-role/StepFunctions-MyStateMachine-role-a6146d54",
    "type": "STANDARD",
    "creationDate": 1611687676.151,
    "loggingConfiguration": {
        "level": "ALL",
        "includeExecutionData": true,
        "destinations": [
            {
                "cloudWatchLogsLogGroup": {
                    "logGroupArn": "arn:aws:logs:us-east-1:1234567890:log-group:/aws/vendedlogs/states/myloggroup:*"
                }
            }
        ]
    }
}

Sphygmograph

Hey thanks @Sphygmograph , what about the enabling for X-ray? Do we have a command for it or can we use CloudFormation? – Guanine

@Cecilia as per the aws cli update-state-machin you pass various kinds of options including the --tracing-configuration with x-ray enabling. My answer is to nudge you into how can you solve it using terrafrom null_resource – Sphygmograph

Thanks, just a follow-up question regarding the ${state-machine-arn}, if I specified the state machine name in locals, should I use ${local.statemachine-name.arn}? and why we need set -euo pipefail? – Guanine

I got error Error: Invalid reference on xxx file in resource "null_resource" --tracing-configuration "${TRACING_PARAMS}" \ A reference to a resource type must be followed by at least one attribute access, specifying the resource name., it seems like I need a triggers parameter for this null_resource, please see: registry.terraform.io/providers/hashicorp/null/latest/docs/… – Guanine

Thanks @Sphygmograph , I got error Error: Error running command 'set -euo pipefail aws stepfunctions update-state-machine --state-machine-arn arn:aws:xxxxxxxxxxx': exit status 1. Output: Environment variable -euo pipefail not defined, I don't understand why we need set -euo pipefail here and it causes this error. – Guanine

I removed set -euo pipefail and terraform applied successfully, but if I navigate to stepfunction console, it shows me that the logging & X-ray are still disabled.... (the log groups exists) I include athe terraform apply console logs in the original question. – Guanine

set -euo pipefail making sure if there are any errors running the shell code inside the null_resource it errors out. It applied successfully because you removed the error checking. The aboce code I tested on my machine Mac OSX before posting it here. – Sphygmograph

@Cecilia you might wanna use this and get rid of null_resource in the terraform code. – Sphygmograph

Hi @Sphygmograph this is really great news!! Although I didn't find the related info in the release notes, but it's in the documentation now. – Guanine

@Cecilia here is the PR for change Enhancement: Step Functions for Express Workflows #12249 – Sphygmograph

Awesome! Hopefully x-ray will be supported soon. Thank you! – Guanine

@Cecilia I think the troubling part for you is resolved now. – Sphygmograph

Answer 2

Currently, it's still an ongoing feature request on Terraform, you can track the status on this github issue.

Charr

Answer 3

Since this has been released in version 3.39.0 of the Terraform AWS provider. We can check the Terraform documentation to make this works:

To enable logging and tracing we can pass the logging_configuration and tracing_configuration arguments to the state machine resource:

resource "aws_cloudwatch_log_group" "sfn_log_group" {
  name = "/aws/vendedlogs/states/myloggroup"
}

resource "aws_sfn_state_machine" "sfn_state_machine" {
  name     = "my-state-machine"
  role_arn = aws_iam_role.iam_for_sfn.arn

  logging_configuration { # enable logging
    log_destination        = "${aws_cloudwatch_log_group.sfn_log_group.arn}:*"
    include_execution_data = true
    level                  = "ALL"
  }
  
  tracing_configuration { # enable tracing
    mode = "Active"
  }
}

Make sure the State Machine has the correct IAM policies for logging and tracing. For more info you can check here: Logging and monitoring in AWS Step Functions.

Crystallo