Exporting AWS Data Pipeline as CloudFormation template to use it in Terraform

Question

I'm trying to export existing AWS Data Pipeline task to Terraform infrastructure somehow.

Accordingly, to this issue, there is no direct support for Data Pipelines, but it still seems achievable using CloudFormation templates (terraform resource).

The problem is that I cannot find a way to export existing pipeline into CloudFormation template.

Exporting the pipeline with its specific definition syntax won't work as I've not found a way to include this definition into CloudFormation. CloudFormer does not support exporting pipelines either.

Does anybody know how to export a pipeline to CloudFormation or any other way to get AWS Data Pipeline automated with Terraform?

Thank you for your help!

Haddington

Feb 2019 update: a PR exists to solve this issue and is awaiting merge. You can use that fork if you don't want to wait: github.com/terraform-providers/terraform-provider-aws/issues/… – Duque

Answer 1

UPD [Jul. 2019]: Some progress has been made in the terraform repository. aws_datapipeline_pipeline resource has been implemented, but it is not yet clear how to use it. Merged pull request

Original answer:

As a solution to this problem, I've come up with a node.js script, which covers my use case. In addition, I've created a Terraform module to be used in Terraform configuration.

Here is the link to the gist with the code

Will copy usage examples here.

Command Line:

node converter-cli.js ./template.json "Data Pipeline Cool Name" "Data Pipeline Cool Description" "true" >> cloudformation.json

Terraform:

module "some_cool_pipeline" {
  source = "./pipeline"
  name = "cool-pipeline"
  description = "The best pipeline!"
  activate = true
  template = "${file("./cool-pipeline-template.json")}"

  values = {
    myDatabase = "some_database",
    myUsername = "${var.db_user}",
    myPassword = "${var.db_password}",
    myTableName = "some_table",
  }
}

Haddington

The new aws_datapipeline_pipeline resource seems to only track the pipeline with terraform. I'm missing the parameter pointing to the json pipeline config... Might go for your solution as well! Thanks – Heloise