How can I use docker compose on AWS Batch?

Question

I have a multi-container (docker compose) application. I would like to scale it offline on AWS Batch for processing large volumes of data on S3. My .yml file for docker compose looks something like this:

version: '2'

services:
    container1:
      container_name:
      image:
      ports:

    container2:
      container_name:
      image:
      depends_on: container1
      ports:

I, unfortunately, cannot find any examples or tutorials online dealing with such a case. Can anyone help me understand how should I approach this problem?

Durnan

Answer 1

I'm not sure what the capabilities were when this was posted but I recently had a project for this issue.

{
  "jobDefinitionName": "batch-job-definition",
  "containerProperties": {
    "image": "public.ecr.aws/docker/library/docker:20-dind",
    "command": ["docker-compose","-f", "/mnt/efs/docker-compose.yml","up"],
    "volumes": [
      {
        "efsVolumeConfiguration": {
          "fileSystemId": "fs-XXXXXXXXXXXXXXXXX",
          "rootDirectory": "/",
          "transitEncryption": "ENABLED",
          "authorizationConfig": {
            "accessPointId": "fsap-XXXXXXXXXXXXXXXXX",
            "iam": "ENABLED"
          }
        }
      }
    ],
    "mountPoints": [
      {
        "containerPath": "/mnt/efs",
        "readOnly": false,
        "sourceVolume": "efs-volume"
      }
    ],
    "user": "root",
    "networkConfiguration": {
      "assignPublicIp": "ENABLED"
    }
  },
  "platformCapabilities": [
    "EC2"
  ],
  "containerOrchestrationType": "ECS"
}

Nekton

Answer 2

Date: 11-06-2018:

Currently, it is not possible, if it is a multi-container docker compose, you can't. There's no option for a docker-compose up instruction, just a single docker image is allowed. The definition would be like next:

"Properties" : {
    "Type" : "container",
    "Parameters" : {
      "net": "host"
    },
    "ContainerProperties" : {
      "Command" : [ "command1" ],
      "Memory" : 2000,
      "Environment": [
          {
            "Name" : "NAME",
            "Value" : "true"
          }
      ],
      "Vcpus" : 2,
      "Image" : "redis:latest"
    },
    "JobDefinitionName" : "job-1",
    "RetryStrategy" : {
        "Attempts" : 1
    }
}

In fact in the official docs is specified container as the only valid value for type:

• https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#type

Type type When you register a job definition, you specify the type of job. At this time, only container jobs are supported.

Type: String

Valid values: container

Required: Yes

---

For individual containers there's a cli tool to transform docker-compose to a job definition but actually, that's something you can do manually:

• https://github.com/turnerlabs/compose-to-batch

Archaize

Thanks for this answer. Very unfortunate that batch doesn't support docker-compose up... They force us to use docker for this service, but then we have to pass env variables and into the job anyway which IMO eliminates one of the better features of docker (bundling .env file and calling docker-compose) – Insubstantial

Does they support docker compose now? Have u solved this problem? What was your solution – Ketonuria