EFS mount failing with mount.nfs4: access denied by server

Question

Trying to mount an EFS file system. The only thing I changed was removing the default SG created with the EFS group and replacing it with a custom SG that my EC2 instances are already in.

AWS provides the necessary command for mounting the NFS share and it SHOULD work verbatim. Often it does. But sometimes you get this:

mount.nfs4: access denied by server while mounting fs-xxxxxxxx.efs.us-west-2.amazonaws.com:/

Sadly, the troubleshooting documentation says under the heading "Action to Take":

If you are attempting to mount the file system using IAM...

... and has absolutely zero recommendation for what to do it your are NOT attempting to mount the FS using IAM.

Now in the first place, I am quite sure I am not doing something wrong because I have playbooks that I've used dozens of times to mount EFS (NFS) shares to EC2 instances are they are quite polished by now. So why would it sometimes fail?

Harmonic

Answer 1

I faced a similar issue and followed StartupGuy's steps. That didn't particularly fix my issue, so I traced the cloud trail events and realised that the access policy needs to have mount access perms as well.

This is the default action for fs-policy:

             ...
             "Action": [
                "elasticfilesystem:ClientRootAccess",
                "elasticfilesystem:ClientWrite"
            ]
            ...

You need to add "elasticfilesystem:ClientMount" as well to the fs policy.

Kheda

where do you configure this policy? IAM? I don't get it – Driven

@Driven The File System Policy can be added in the console UI under the tab with the same name after you click on your file system. It can also be set with the API and Terraform. – Chrysoprase

This solved it for me. The AWS docs make it seem like ClientWrite includes an implied ClientMount permission, but it doesn't. – Orange

Answer 2

Well as it turns out, AWS is not always as slick as it usually feels and sometimes things get botched on the back-end. In this case, it was possible that replacing the SG actually might have appeared to work in the UI, but on the back-end did not take effect. I'm just guessing.

What I did was just delete the existing EFS and create the exact same EFS. This time the only thing I did different was set my custom EFS SG on creation of the EFS, instead of replacing it after creation and viola, my playbooks were working again.

There is absolutely no intuitive (or documented) reason that I can think of, why starting with non-default SG should be any different than replacing it, when its the exact same SG. In either case you are setting up the EFS to use your selected SG and EFS is not objecting. Besides, I've done that before with no problems. So I am chalking this up as a EFS/SG back-end screw up that wasted a lot of time to troubleshoot.

In summary, if a new EFS share is giving you the mount.nfs4: access denied by server error when trying a standard mount (and you know you are doing everything else correctly) - just delete it and just re-create it. Don't necessarily assume you are doing something wrong and that AWS services can't screw up now and then.

Harmonic

same solution for me. i simply re-created my efs and it worked. you should mark this as answered – Agrigento

Answer 3

I got this error message when the directory on the EFS did not exist while trying to access through an access point.

Hectoliter

From the AWS docs: "When you mount a file system with an access point, the root directory for the access point is created if the directory doesn't already exist." docs.aws.amazon.com/efs/latest/ug/efs-access-points.html – Luckett

From the same document: "Amazon EFS creates a root directories only if you have provided the OwnUid, OwnGID, and permissions for the directory. If you do not provide this information, Amazon EFS does not create the root directory. If the root directory does not exist, attempts to mount using the access point will fail. " – Hectoliter

Answer 4

If you have a File System Policy set and you are using 'sudo mount -t nfs' or 'nfs4,' then you get this error. You can either:

• A.) Remove the file system policy from the file system
• B.) Use the following command to mount: sudo mount -t efs -o tls,iam ...

Source: https://docs.aws.amazon.com/efs/latest/ug/mounting-IAM-option.html

Chrysoprase

Answer 5

For me, the problem was that I had a policy requiring encryption in transit for the drive, and the instance creation wizard creates a bad /etc/fstab entry.

1. The policy requires that the drive be mounted with tls. Instructions for this are given here: https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-helper-ec2-linux.html, IF you use the mount helper, and specify -o tls.
2. The /etc/fstab created by the instance creation wizard does not perform the proper mount. In fact, the "Using the NFS client" option on that same page is equivalent to the bad entry which is created.

Here is what a proper /etc/fstab entry looks like for encryption in transit:

fs-0123456789abcdef0:/ /mnt/fs-1 efs tls,_netdev 0 0

Broder

Answer 6

Agree with Michael M's solution. We had identical issue after creating volumes with default SG and then moving the mount targets to a specific EFS SG. After two days of troubleshooting I finally dropped the volumes and recreated them, using the EFS SG at the time of creation. Worked without issue after that.

Defiant

Please don't add "thank you" as an answer. Once you have sufficient reputation, you will be able to vote up questions and answers that you found helpful. - From Review – Furlough

Answer 7

I was getting this error mounting an EFS access point into a container (FARGATE).

Permissions-wise, I do not have the permissions mentioned in other answers. All I have is the task execution pole, that is required and is bound an aws predefined policy AmazonECSTaskExecutionRolePolicy, and a task policy that I have defined to write on log stream, execute commands in the container, kms, etc. No EFS related permissions.

What I was doing wrong is that I was not defining Root directory creation permissions. This permission is used to create the Root directory path within EFS. So the access denied in my case meant that the directory was not yet there and I was trying to mount it in the container.

These permissions are not necessary if someone is going to mount filesystems root directory. Actually, you probably shouldn't be using an access point to do that, but other than the root directory, you need it.

In fact, as a test, if one doesn't define Root directory creation permissions (therefor gets the access denied error), can mount the filesystem's root directory and manually create the necessary path, and then try to mount it, it will work just fine.

Objection