How do I retrieve AWS Batch job parameters?

Question

How do I retrieve parameters from an AWS Batch job request? Suppose I have a job submitter app that sends a job request with the following code (in C#):

    SubmitJobRequest submitJobRequest = new SubmitJobRequest()
    {
        JobName = "MyJobName",
        JobQueue = "MyJobQueue",
        JobDefinition = "MyJobDefinition:1",
        Parameters = new Dictionary<string, string>() { {"Foo", "Bar" } },
    };

    SubmitJobResponse submitJobResponse = AWSBatchClient.SubmitJob(submitJobRequest);

What I want to be able to do now is retrieve what's in the Parameters field in submitJobRequest in my docker app that gets launched. How do I do that? It's not passed in as program args, as I've tested that (the only args I see are those were statically defined for 'Command' my job definition). I know that I can set environment variables via container overrides and then retrieve them via Environment.GetEnvironmentVariable (in C#). But I don't know how to get the parameters. Thanks.

Vice

Answer 1

Here is an example using yaml cloudformation.(or you can use json for same properties).You can declare the parameters using a Ref in the command section. I am using user_name but you can add more. We might have a limit of 30KB payload

ContainerProperties:
   Command:
       - "python"
       - "Your command here"
       - "--user_name"
       - "Ref::user_name"

Now you can submit your job to the queue like this. I am using python and boto3 client:

  # Submit the job
    job1 = client.submit_job(
        jobName=jobName,
        jobQueue=jobQueue,
        jobDefinition=jobDefinition,
        parameters={
            'user_name':user_name
        }
    )

To retrieve the parameters use this(I am using argparse):

  parser = argparse.ArgumentParser(description='AWS Driver Batch Job Runner')
  parser.add_argument('--user_name', dest='user_name', required=True)
  args = parser.parse_args()
  print(args.user_name)

Responsum

Where do you exactly configure the first block of lines, starting with ContainerProperties? I do not get where I should place it to configure the access to the parameters. I am starting a AWS batch process from a AWS lambda function and I am sending the parameter as you show in the second block. However, I am not receiving it on the batch process since I print sys.argv and I only see the name of the file and a echo message of hello world – Earring

@LucasCaminoYou can use this for AWS batch in cloudformation docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/… – Responsum

Thank you for replying, @sheetal. I actually found out the issue I was having with retrieving "parameters" in AWS Batch Jobs was related to how AWS defines concepts in a way I wasn't expecting. All this confusion motivated me to write an answer so I could share what I had learned when I finally solved my problem. I invite you and whoever may be in my situation to check this post. Maybe what worked for me, works for someone else! – Earring

Answer 2

Found the answer I was looking for. I just had to add a ref to the parameter for Command in the job definition. In the question example, I would've needed to specify Ref::Foo for Command in the job definition and then "Bar" would've gotten passed as program args to my container app.

To expand on my example, in my specific case, my program uses the CommandLineParser package for passing parameters. Suppose one of the CommandLine options is called Foo. If I were running the program from a command line, I'd set a value for Foo with something like "--Foo Bar". To effectively do the same for my batch job, in my job definition, for Command, I would specify "--Foo Ref::Foo" (without quotes). Then for the Parameters field in my SubmitJobRequest object, I would set Foo exactly as per my original example and then my batch program would see "Bar" for the Foo CommandLine option (just like as if it was run with "--Foo Bar"). Hope that helps.

Vice

can you post an example? Also it's kind of annoying that aws batch docs don't clearly show this in an example... – Kinetics

@adam Edited my answer to provide more details. Hope that helps. And agree that Amazon doesn't provide good Batch examples. – Vice

@AdamHughes check my answer. – Responsum

Can you post example? – Systematology