How do I have an S3 bucket return 404 (instead of 403) for a key that does not exist in the bucket/

Question

I am using S3 to store some business critical documents. I want the bucket to return a 404 status code when trying to access an object that does not exist in the bucket.

However, I am finding that it keeps on returning me "403

here is an example of a session using the S3 website url.

> GET /foobar.txt HTTP/1.1
> User-Agent: curl/7.21.6 (x86_64-pc-linux-gnu) libcurl/7.21.6 OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
> Host: <bucketname>.s3-website-us-east-1.amazonaws.com
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
< Last-Modified: Mon, 09 Sep 2013 19:10:28 GMT
< ETag: "14e13b81b3ce5b129d1f206b3e514885"
< x-amz-error-code: AccessDenied
< x-amz-error-message: Access Denied
< x-amz-request-id: <snip>
< x-amz-id-2: <snip>
< Content-Type: text/html
< Content-Length: 11
< Date: Thu, 26 Sep 2013 20:01:45 GMT
< Server: AmazonS3
< 
Not found!

Note, the "Not Found!" string is coming from the error document set on the bucket properties when enabling S3 website hosting.

I have also tried accessing using the bucket url directly

http://.s3.amazonaws.com/

and that returns the same, except that instead of the error document, I get a XML document

How do I solve this problem?

Terryn

Answer 1

S3 returns a 403 instead of a 404 when the user doesn't have permission to list the bucket contents.

If you query for an object and receive a 404, then you know that object doesn't exist. This is information you shouldn't know if you don't have permission to list the bucket contents so instead of telling you it doesn't exist, S3 just tells you that you're trying to do something you're not allowed to do. When you get a 403 instead of a 404 you have no way of knowing that the object you requested doesn't exist. It might not exist or it might exist and you just don't have permission to access it. There's no way for you to know for sure and so no security is bypassed.

I believe anyone with access to list the bucket contents will get a 404 instead of a 403.

Proprietor

See my post below for additional details about the format of the ARN. – Forethought

Here is an answer that explains how to add the List permission using the AWS console: https://mcmap.net/q/98244/-s3-static-website-hosting-route-all-paths-to-index-html – Phila

Returning a 404 implies no such thing. It may well be forbidden, not missing. The 403 spec: 10.4.4 403 Forbidden The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity. If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead. – Astound

That depends on the server. Note the word "can" is not "should" in the 403 spec. If you know the server behaves this way, as S3 does, a 404 does imply the object does not exist. – Proprietor

Answer 2

Not Sure if you're looking for this. Making your objects public to everyone solves the 404 issue. However, I do not believe that it is the ideal way to go through with it.

AWS CloudFront provides a feature called Origin Access Identity (OAI). How it works is given in detail here.

Basically in a nutshell, Associate an OAI with your Origin in CloudFront and update the bucket policy to allow the OAI with GetObject and ListBucket as shown

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "AllowOAIRead",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity your_OAI_ID"
        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your_bucket_name/*",
        "arn:aws:s3:::your_bucket_name"
      ]
    }
  ]
}

Dyspepsia

You can see the documentation over here. docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html in the permission section. – Twitter

This is the correct answer. The others make it vulnerable. – Cheshvan

That answer should be marked as the best one – Tryout

Isn't it at risk to allow the CloudFront distribution to list all bucket objets? Allowing listing everything, only to have a proper 404 error, don't seems to follow the "least privilege" security principle – Gathard

Note also that origin access control (OAC) is deprecated in favor of origin access identity (OAI): docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/… – Gathard

Answer 3

The exact requirement seems to be that your user has ListBucket permission for your particular bucket AND the ARN is exactly of the form arn:aws:s3:::your_bucket_name.

I also needed to add a completely new statement to my policy because other permissions like GetObject still require that the ARN ends with /* or some other suitable wildcard.

{
  "Action": [
    "s3:ListBucket"
  ],
  "Sid": "StmtNNNNNNNNNNNNNNNwholebucket",
  "Resource": [
    "arn:aws:s3:::your_bucket_name"
  ],
  "Effect": "Allow"
}

To summarize, the important bit for me was that if the ARN is NOT of the form arn:aws:s3:::your_bucket_name/* for ListBucket or you will still get 403 instead of 404.

Forethought

<pre>"Principal": "*"</pre> missing – Jonme

Yes, the Principal is probably needed if you add the policy directly on the bucket. I added it on IAM user in which case it is not needed. – Forethought

DO NOT DO: Principal *. You would authorize anyone to access the bucket, security nightmare – Helles

Answer 4

I needed to extend the policy like this:

 "Action": [
   "s3:Get*",
   "s3:List*"
 ],
 "Resource": [
   "arn:aws:s3:::bucket_name",
   "arn:aws:s3::: bucket_name/*"
 ],

bucket_name is needed, because without it you don't get 404 for missing objects, but 403 always, bucket_name/* is needed to actually access stuff in the bucket.

Pike

These action wildcards give much broader permissions than most will need. Other answers mention adding a separate statement for s3:ListBucket specifically which is a better security practice. – Conchoidal

Answer 5

Make sure in your permissions Everyone has View Permissions.

You may want to add a bucket policy too:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your_bucket_name/*"
        }
    ]
}

Rooky

Using the bucket policy means you don't need the everyone permissions. – Kittenish