Terraform is throwing InvalidArgumentException Duplicate ProcessorParameter passed to ProcessingConfiguration on Kinesis Firehose dynamic partitioning

Question

I'm trying to create a Kinesis Firehose using terraform with dynamic partitioning using two partition queries from the JSON I'm recieving, my processing configuration looks like this

processing_configuration {
  enabled = true
  processors {
    type = "RecordDeAggregation"
    parameters {
      parameter_name  = "SubRecordType"
      parameter_value = "JSON"
    }
  }
  processors {
    type = "MetadataExtraction"
    parameters {
      parameter_name  = "JsonParsingEngine"
      parameter_value = "JQ-1.6"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{transaction_id:.transaction_id}"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{stage:.stage}"
    }
  }
}

But when I execute this part of the code it returns a duplication error for the processing configuration.

I also tried to create an appart processor for the new ExtractionQuery, it looks like this

processing_configuration {
  enabled = true
  processors {
    type = "RecordDeAggregation"
    parameters {
      parameter_name  = "SubRecordType"
      parameter_value = "JSON"
    }
  }
  processors {
    type = "MetadataExtraction"
    parameters {
      parameter_name  = "JsonParsingEngine"
      parameter_value = "JQ-1.6"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{transaction_id:.transaction_id}"
    }
  }
  processors {
    type = "MetadataExtraction"
    parameters {
      parameter_name  = "JsonParsingEngine"
      parameter_value = "JQ-1.6"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{stage:.stage}"
    }
  }
}

But it fails with an error that says only one MetadataExtraction processor is allowed.

Celt

Answer 1

Solved by merging both queries in one using JQ format, that way firehose would separate them, tried it using this snippet and worked.

processing_configuration {
  enabled = true
  processors {
    type = "RecordDeAggregation"
    parameters {
      parameter_name  = "SubRecordType"
      parameter_value = "JSON"
    }
  }
  processors {
    type = "MetadataExtraction"
    parameters {
      parameter_name  = "JsonParsingEngine"
      parameter_value = "JQ-1.6"
    }
    parameters {
      parameter_name  = "MetadataExtractionQuery"
      parameter_value = "{transaction_id:.transaction_id,stage:.stage}"
    }
  }
}

Celt

I had exactly this problem, and tried exactly the things you tried, before stumbling upon this exact answer to my problem. Nice solution! TF docs could be a bit more helpful here. – Miscarriage