AWS Step cannot correctly invoke AWS Batch job with complex parameters

Question

I have an existing AWS Steps orchestration that is executing a AWS Batch job via lambdas. However AWS have recently added the ability to directly invoke other services like AWS Batch from a step. I am keen to use this new functionality but cannot get it working.

https://docs.aws.amazon.com/step-functions/latest/dg/connectors-batch.html

So my new step operation that I want to use to invoke Batch.

"File Copy": {
  "Type": "Task",
  "Resource": "arn:aws:states:::batch:submitJob.sync",
  "Parameters": {
     "JobName": "MyBatchJob",
     "JobQueue": "MySecondaryQueue",
     "ContainerOverrides.$": "$.lts_job_container_overrides",
     "JobDefinition.$": "$.lts_job_job_definition",
  },
  "Next": "Upload Start"
}

Note that I am trying to use the $. JSONpath syntax in order to dynamically have parameters passed through the steps.

When given the following inputs

"lts_job_container_overrides": {
  "environment": [
    {
      "name": "MY_ENV_VARIABLE",
      "value": "XYZ"
    },
  ],
  "command": [
    "/app/file_copy.py"
  ]
},
"lts_job_job_definition": "MyBatchJobDefinition"

I was expected that the environment and command values would be passed through to the corresponding parameter (ContainerOverrides) in AWS Batch. Instead, it appears that AWS Steps is trying to promote them up as top level parameters - and then complaining that they are not valid.

{
 "error": "States.Runtime",
 "cause": "An error occurred while executing the state 'File Copy' 
 (entered at the event id #29). The Parameters 
 '{\"ContainerOverrides\":{\"environment\": 
 [{\"name\":\"MY_ENV_VARIALBE\",\"value\":\"XYZ\"}],\"command\": 
 [\"/app/file_copy.py\"]},\"JobDefinition\":\"MyBatchJobDefinition\"}' 
 could not be used to start the Task: [The field 'environment' is not 
 supported by Step Functions, The field 'command' is not supported by 
 Step Functions]"
}

How can I stop AWS Steps from attempting to interpret the values I am trying to pass through to AWS Batch?

---

I have tried taking JSON path out of the mix and just specifying the ContainerProperties statically (even though this long term won't be a solution). But even then I encounter issues.

"ContainerOverrides": {
   "environment": [
    {
      "name": "RUN_ID",
      "value": "xyz"
    }
   ],
    "command": "/app/file_copy.py"
   }

In this case steps itself rejects the definition file on load.

Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The field 
'environment' is not supported by Step Functions at /States/File 
Copy/Parameters, SCHEMA_VALIDATION_FAILED: The field 'command' is not  
supported by Step Functions at /States/File Copy/Parameters'

So it just appears that ContainerOverrides is problematic fullstop? Have I misunderstood how it is intended to be used in this scenario?

---

The above issue has been resolved (as per the answer below) in the AWS Batch documentation - the following note has been added by AWS:

Note

Parameters in Step Functions are expressed in CamelCase, even when the native service API is pascalCase.

Honeywell

Answer 1

This should work, I've tested it seems to be working fine for me. Both Environment and its object keys and Command should be first letter capital.

{
   "StartAt": "AWS Batch: Manage a job",
   "States": {
      "AWS Batch: Manage a job": {
         "Type": "Task",
         "Resource": "arn:aws:states:::batch:submitJob.sync",
         "Parameters": {
            "JobName": "test",
            "JobDefinition": "jobdef",
            "JobQueue": "testq",
            "ContainerOverrides": {
               "Command": [
                  "/app/file_copy.py"
               ],
               "Environment": [
                  {
                     "Name": "MY_ENV_VARIABLE",
                     "Value": "XYZ"
                  }
               ]            
            }
         },
         "End": true
      }
   }
}

Rattray

How is this different from my first example? I notice the ContainerOverrides is missing the trailing .$ - but understood that was needed in order to enable JSONpath – Honeywell

I understand you've already tried without JSONpath but I think this error is because of three missing required fields? – Rattray

No - sorry, yes - I did have those fields in when I actually tried. I cut them out in making this stackoverflow post (they were just adding noise to the examples) - but as they are mandatory I will put them back in the example. Thanks. (to be clear though the problem was not that they weren't there in the actual testing) – Honeywell

I've updated my original answer. Both Environment and Command needs to be first letter capital. – Rattray

Thanks Athar - that has indeed fixed it.. all pass through values to AWS Batch must be capitalised, including Command, Environment, Name and Value - whether or not they are capitalised in the batch documentation or not. – Honeywell

In fact, the AWS documentation that has RetryStrategy with "attempts": 5 is misleading.. it will not work. It must be "Attempts": 5 – Honeywell

This is a AWS documentation error - I will submit something to AWS to clarify/correct the docs. Thanks Athar – Honeywell

You are right Andrew AWS documentation has error in the given example. But I am glad issue has been resolved. :) – Rattray