AWS Batch Arrays - array size?

Question

I'm using AWS Batch and have started using Array Jobs. AWS_BATCH_JOB_ARRAY_INDEX is passed as an Environment Variable to the container.

Is the array size passed in some way? It is mandatory to know whether the index was related to 5 jobs or 1000 jobs. Currently I'm passing it as my own environment variable but thought that that info would be passed to the container in some way already.

Unpremeditated

Whatever you do, don't manually name it AWS_BATCH_JOB_ARRAY_SIZE. I did this, and then neither of the two env vars were then made available to the job. The name JOB_ARRAY_SIZE works fine instead. Alternatively, it can be passed as a command-line arg, etc. – Shushan

Answer 1

This is not possible at the moment. I've made a feature request for it, which you can upvote here: https://github.com/aws/containers-roadmap/issues/1631

In the meantime, I found a hacky workaround. The job ID for array workers appears to conform to $PARENT_JOB_ID:$AWS_BATCH_JOB_ARRAY_INDEX. So, to the extent that you can rely on this formatting of array worker IDs, you can describe the parent job and get the total array size from there. Here's an example using boto3:

import os
import boto3

worker_job_id = os.environ['AWS_BATCH_JOB_ID']
parent_job_id = worker_job_id.split(":")[0]

response = boto3.client('batch').describe_jobs(jobs=[parent_job_id])

parent_job = response['jobs'][0]
array_size = parent_job.get('arrayProperties', {}).get("size")

print("array_size =", array_size)

Emmet

Answer 2

if my understanding is correct, are you asking where should the array size to be passed in aws batch?

In Jobs section, click submit job - in environment select Array.

Refer: https://docs.aws.amazon.com/batch/latest/userguide/submit_job.html

Oleate

Yes, you provide the number of jobs you want at the time of the job submission HOWEVER that information never makes it to inside the docker from what I can tell. Currently I seem to be required to manually add this to may environment variable but it would have seemed to be a natural value for the job to pass iit in already. – Unpremeditated