Starting AWS Batch from SQS queue

Question

I have a long running batch job that am planning to move to AWS Batch. Since it takes couple of hours to run this job, I cant use AWS Lambda.

But, one of the requirement is to trigger this job from my AWS SQS queue.

I couldn't find any reference documentation on this. Is it even feasible?

Parris

Do you need to just trigger the batch job or do you need to monitor and notify once the job is complete? – Iced

I just need to trigger the job when a new item is added to the SQS queue. Monitoring and notification is not a priority now, as we have other systems that can do the same. – Parris

In that case, why not have a Lambda Trigger configured for the SQS Queue that'd just initiate the Batch job using a boto3 Batch client.submit_job() and exit out with a successful or failed job-start status or the actual submit_job return status whichever you need? – Iced

+1 to what @Iced suggested. If you are familiar with Lambda already you can use it (and the SQS integration) to invoke a batch job rather than running the logic itself. – Formative

Answer 1

You can implement below method on listening of SQS message :

public void onMessage( Message message ) throws JMSException {

    SQSTextMessage sqsTextMessage = ( SQSTextMessage ) message;
    String text = sqsTextMessage.getText();
  
    AWSBatch awsBatch = AWSBatchClientBuilder.standard()
       .withCredentials(new DefaultAWSCredentialsProviderChain())
       .withRegion("region")
       .build();

    SubmitJobRequest submitRequest = new SubmitJobRequest()
       .withJobDefinition("job-definition-name")
       .withJobName("job-name")
       .withJobQueue("job-queue");
  
    SubmitJobResult submitResult = awsBatch.submitJob(submitRequest);
  }

Carleycarli

Answer 2

Unfortunately, while SQS can natively trigger AWS Lambda functions, it does not support this same native triggering with Batch jobs. As a workaround, you'll have to just trigger a Lambda that implements a call to the Batch SubmitJob API using REST or your favorite AWS SDK (e.g. Python Client.submit_job).

NOTE: Depending on your architecture, you could leverage EventBridge, which allows you to configure API destinations, which are HTTP endpoints that you can invoke as the target of an EventBridge rule. In this case, EventBridge Rule could directly trigger Batch, allowing you to potentially remove SNS/SQS/Lambda from your architecture.

Oscar