Mounting an elastic file system to AWS Batch Computer Enviroment

Question

I'm trying to get my elastic file system (EFS) to be mounted in my docker container so it can be used with AWS batch. Here is what I did:

1. Create a new AMI that is optimized for Elastic Container Services (ECS). I followed this guide here to make sure it had ECS on it. I also put the mount into /etc/fstab file and verified that my EFS was being mounted (/mnt/efs) after reboot.

2. Tested an EC2 instance with my new AMI and verified I could pull the docker container and pass it my mount point via

docker run --volume /mnt/efs:/home/efs -it mycontainer:latest

Interactively running the docker image shows me my data inside efs

3. Set up a new compute enviorment with my new AMI that mounts EFS on boot.

4. Create a JOB definition File:

   {
   "jobDefinitionName": "MyJobDEF",
   "jobDefinitionArn": "arn:aws:batch:us-west-2:#######:job-definition/Submit:8",
   "revision": 8,
   "status": "ACTIVE",
   "type": "container",
   "parameters": {},
   "retryStrategy": {
       "attempts": 1
   },
   "containerProperties": {
       "image": "########.ecr.us-west-2.amazonaws.com/mycontainer",
       "vcpus": 1,
       "memory": 100,
       "command": [
           "ls",
           "/home/efs",
       ],
       "volumes": [
           {
               "host": {
                   "sourcePath": "/mnt/efs"
               },
               "name": "EFS"
           }
       ],
       "environment": [],
       "mountPoints": [
           {
               "containerPath": "/home/efs",
               "readOnly": false,
               "sourceVolume": "EFS"
           }
       ],
       "ulimits": []
   }
   

   }

5. Run Job, view log

Anyway, while it does not say "no file /home/efs found" it does not list anything in my EFS which is populated, which I'm inerpreting as the container mounting an empty efs. What am I doing wrong? Is my AMI not mounting the EFS in the compute environment?

Colver

Answer 1

I covered this in a recent blog post

https://medium.com/arupcitymodelling/lab-note-002-efs-as-a-persistence-layer-for-aws-batch-fcc3d3aabe90

You need to set up a launch template for your batch instances, and you need to make sure that your subnets/security groups are configured properly.

Gena

Welcome, Sean, and thanks for deciding to contribute. Would it be possible to flesh this answer out a little, so that if the third-party link rots away, what remains stands as its own answer? – Carma