CloudFront + S3 Website: "The specified key does not exist" when an implicit index document should be displayed

Question

I've just deployed a static website to Amazon S3, which can currently be viewed here: http://www.rdegges.com.s3-website-us-east-1.amazonaws.com/

If you click any of the article links, you'll notice the following error:

S3 is complaining the file doesn't exist. Now, here's what's weird about this -- I'm using CloudFront on my domain. So when you click that article link, it's sending the request to CloudFront which then tries to fetch the file back from the S3 bucket.

However, if you visit that same URL from S3 directly, eg: http://www.rdegges.com.s3-website-us-east-1.amazonaws.com/2015/building-a-heroku-addon-planning/ the page will load just fine.

It appears that something is getting lost in translation here.

Anyone got a suggestion of what I can do to fix my settings?

Canna

Answer 1

I'll go out on a limb and say that the specified key doesn't technically exist, so the error message is technically accurate but doesn't tell the whole story. This should be an easy fix.

S3 buckets have two¹ endpoints, "REST" and "website." They have two different feature sets. The web site endpoint provides magical resolution of index documents (e.g. index.html, which appears to be what is actually supposed to be returned to the browser in the example you provided) while the REST endpoints don't.

When you configure CloudFront in front of a bucket used for web site hosting, you usually don't want to configure the origin as an "S3" origin by selecting the bucket name from the drop-down list; instead, you want to configure it as a "Custom" origin, and use the web site endpoint hostname as provided in the S3 console (e.g. example-bucket.s3-website-us-east-1...) because otherwise, CloudFront assumes you want it to use the REST endpoint for the bucket (which allows authentication and private content, which the web site endpoint doesn't).

Important

Do not select the name of your bucket from the list, for example, example.com.s3.amazonaws.com.

http://docs.aws.amazon.com/gettingstarted/latest/swh/getting-started-create-cfdist.html

The documentation was refactored since this question was originally answered, so the message shown above now appears one page later, and has been reworded, but the gist is the same. The "name of the bucket" seems to refer to the choices shown in the drop-down, which is not what you want.

Note

Be sure to specify the static website hosting endpoint, not the name of the bucket.

http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-cloudfront-walkthrough.html

The hint that you're using the REST endpoint for the bucket is that the error message is returned in XML. If you were using the web site endpoint, then the web site endpoint would return error messages in HTML.

Create a new origin for the CloudFront distribution, as described, then change the behavior to send requests to the new origin, then send a CloudFront cache invalidation request for /* and you should be set.

See also:

http://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteEndpoints.html#WebsiteRestEndpointDiff

---

¹ two endpoints. Technically, there are more than two, since all buckets have at least two possible REST endpoint hostnames... but there are two types of endpoints. Buckets also have an optional transfer acceleration endpoint that uses the AWS edge network (the same infrastructure that powers CloudFront) for faster/optimized transfers, particularly from geographic locations more distant from the region where the bucket is provisioned, but without using the CloudFront cache. This endpoint looks like https://example-bucket.s3-accelerate.amazonaws.com if you activate it, and carries an additional usage charge for most requests since you are using more of the AWS network and less of the public Internet... but, that is a difference in the behind-the-scenes deployment of the endpoint, not the behavior of the endpoint. The transfer acceleration endpoint is still a REST endpoint, so just like the other REST endpoints, it does not have the web site hosting features. CloudFront won't let you use an acceleration endpoint for an origin domain name, because that wouldn't make sense -- if such a configuration were allowed, requests and responses would loop through the AWS Edge Network twice and increase both latency and costs without providing any benefit.

Henze

Thanks! This is exactly what my issue was. It's funny because I've deployed numerous sites this way before, and this was never an issue -- I guess it was all relatively luck up until now, since my sites were mainly single pages. Heh. – Canna

Makes sense. Single page sites will work fine if you override the root object in the CloudFront configuration, but for index.html to work elsewhere, or custom error pages, or redirect rules, you need the web site endpoint. – Henze

Yeah! Thanks so much. Super helpful. I hope someone else who makes this dumb mistake can find your answer useful as well =) – Canna

@Canna dumb mistake maker checking in. With 45mins for each CloudFront tweak, this claimed a day of my life. Talk about subtle... – Surely

@Surely you'll find that CloudFront tweaks typically start working within a couple of minutes. You only need to wait for the distribution to transition back to Deployed if there's some doubt as to whether the change is not working as expected. If you know you got it wrong, you can push additional changes without ill effects. It also helps speed up troubleshooting, if you're getting errors, to zero-out the Error Caching Minimum TTLs – error responses are cached using different timers. – Henze

Thanks so much for this cogent and well articulated answer! – Insinuation

Step by step instructions from AWS: aws.amazon.com/premiumsupport/knowledge-center/… – Entellus

Hint: if the AWS Console doesn't show the website endpoint, you can enter it manually by just copying it from the bucket static website hosting property. Its usually the same but with the s3 component replaced with s3-website – Sunburst

Answer 2

I encountered the same issue and this is how I resolved it.

In CloudFront Origin Settings set Origin Domain Name to <website bucket>.s3-website-us-west-2.amazonaws.com

In CloudFront General Settings make sure to have index.html as Default Root Object.

In S3 make sure to have Use this bucket to host a website selected and set index.html as Index document.

Lemos

The solution for me was that someone had set the default root object up as /index.html and not index.html. – Tracheotomy

Thanks! This is exactly the solution for my case. – Peyter

You my man, are a legend, you have no idea how long I looked for this answer! I used a similar setup for image hosting and running a image resizing script via lambda and API Gateway. – Amarillas

Awesome. This was the fix! – Allieallied

This worked with an additional change for me -- my S3 bucket contained a folder which held my assets. In addition to the changes noted above I had to modify the web endpoint to add my folder after the URL like: website bucket>.s3-website-us-west-2.amazonaws.com/MYFOLDER – Dragone

I also fixed it this way, but now the problem is that when I manually type in a route other than the root route (using Angular, S3 and Cloudfront) then I get the same error, e.g. example.com works, but example.com/products yields the same error. How would I fix this? – Inexpiable

My solution was to add index.html as Default Root Object in Cloudfront – Foredate

Answer 3

Are you hosting a React Site on S3? Possibly with React Router?

NOTE: I have my S3 Bucket closed away from public access, and I did not have any desire to make it public. Also, while the problem I faced is the same as what is asked above, the solution in my case is different.

In case you, like me, do NOT want to make your S3 endpoint public, and are facing this issue with React Router where direct URLs with routes are returning a 403 Forbidden on CloudFront, read on:

This is happening because CloudFront expects the S3 bucket to have the file that your URL supposedly points to.

For instance, if you have a static site "example.com" and a route "/stack", CloudFront will try to look for the 'stack' file in your S3 bucket. This obviously does not exist, and CloudFront will return an error.

Best way to resolve this is to set up a custom Error in CloudFront for HTTP error code 403. The custom error response can point to the "index.html" page with a "200 OK" status. This will solve the error, point the user to the right file, and at the same time, retain the original URL (example.com/stack) which will then be picked up by React Router.

Cello

Answer 4

Access Cloudfront -> Your Distribution

Goto Error Pages and Click Add Error Page . Enter details like the below

Straightway

Answer 5

I was also having a similar problem I have followed these steps which have solved this issue

1. Go to CloudFront Distributions
2. Click the ID
3. After Clicking ID you will find different categories like General, Origins and Origin Groups.
4. Click the Origins and Origin Groups
5. Click the checkbox of your s3 bucket and click edit
6. Under grant read Permissions on Bucket click "Yes, Update bucket policy"

This step has solved my issue.

Algol

Answer 6

See AWS docs on how to use CloudFront to serve a static website hosted on S3 over https.

Transcribing content below for convenience (or in case the link ever goes bad).

---

1. Use the Amazon S3 console to create a bucket and to enable static website hosting on the bucket.

2. From the Static website hosting dialog box, copy the Endpoint of your bucket without the leading http://. The format is similar to bucketname.s3-website-region.amazonaws.com. You need the endpoint in this format for a later step.

3. Add a bucket policy that allows public read access to the bucket that you created.

4. Create a CloudFront web distribution. Be sure to configure the following:

   • For Origin Domain Name, enter the endpoint that you copied in step 2.
   • For Allowed HTTP Methods, select GET, HEAD, OPTIONS.
   • For Alternate Domain Names (CNAMEs), enter the CNAME you want to use for your website.

5. If you don't want to use SSL (HTTPS) for your website, proceed to the next step. If you want to use SSL for your website, you can choose Request or Import a Certificate with ACM to request a certificate. For more information, see Using Alternate Domain Names and HTTPS.

6. Choose Create Distribution.

7. Update the DNS records for your domain to point your website's CNAME to your CloudFront distribution's domain name. You can find your distribution's domain name in the CloudFront console in a format that is similar to d1234abcd.cloudfront.net.

8. Wait for your DNS changes to propagate and for the previous DNS entries to expire.

Entellus

Answer 7

If you are using React Js, this steps should work for you. The answer provided above is working. Posted the screenshot here for better reference.

Creak

Answer 8

I was facing the same issue, how I solved it, below are the steps

• Go to Amazon S3 and then go to your bucket directory

• Enable S3 Static Website Hosting

• Go to Permissions tab -> Block all public access -> uncheck Block all public access and save it

• Go to Permissions tab -> Bucket Policy -> Edit and Save

  {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
  }
  

Note: change this line "Resource": "arn:aws:s3:::your-bucket-name/*"

Ex: "Resource": "arn:aws:s3:::ng-ci-cd-aws-demo/*"

reference: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html

Doscher

Answer 9

There are a few requirements to get s3 websites working with Cloudfront:

• Objects in the bucket must be publicly accessible.
• Objects in the bucket can't be encrypted by AWS Key Management Service (AWS KMS).
• The bucket policy must allow access to s3:GetObject.
• If the bucket policy grants public read access, then the AWS account that owns the bucket must also own the object.
• The requested objects must exist in the bucket.
• Amazon S3 Block Public Access must be disabled on the bucket.
• If Requester Pays is enabled, then the request must include the request-payer parameter.
• If you're using a Referer header to restrict access from CloudFront to your S3 origin, then review the custom header.

https://aws.amazon.com/premiumsupport/knowledge-center/s3-website-cloudfront-error-403/

The main issue I had was, there are two types of s3 buckets urls:

REST endpoints format: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro

https://bucket-name.s3.Region.amazonaws.com

Website endpoints format: https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteEndpoints.html

http://bucket-name.s3-website-Region.amazonaws.com
http://bucket-name.s3-website.Region.amazonaws.com

In my case I needed to use the website url not the REST url:

• REST: http://project-name.s3.us-east-1.amazonaws.com
• Website: http://project-name.s3-website-us-east-1.amazonaws.com

Using the website url will allow you to use the Website hosting features of S3 such as the index/error documents and redirect rules!

Schizophrenia

Answer 10

You should follow the steps below.

To update your bucket policy using the CloudFront console, follow these steps:

1. Open the CloudFront console and choose your distribution.
2. Choose the Origins and Origin Groups tab.
3. Select the S3 origin and choose Edit.
4. For Restrict Bucket Access, choose Yes.
5. For Origin Access Identity, choose the existing identity or create a new one.
6. For Grant Read Permissions on Bucket, choose Yes, Update Bucket Policy.
7. Choose Yes, Edit.

If you configured an OAI, the OAI must be included in the S3 bucket policy If you added an OAI to your CloudFront distribution, you must also include an allow statement for the OAI in the S3 bucket policy.

To verify if your bucket policy allows the OAI, open your S3 bucket in the Amazon S3 console. Then, choose the Permissions tab and review the bucket policy. The following is an example allow statement for an OAI:

{
  "Sid": "1",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EAF5XXXXXXXXX"
  },
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
}

Reference: https://aws.amazon.com/tr/premiumsupport/knowledge-center/s3-rest-api-cloudfront-error-403/

Semen

Answer 11

Easy solution, create and publish a cloudfront function and associate it with your distribution. This function will add /index.html to any requests that don't have a file name.

The function should look something like this:

    function handler(event) {
    var request = event.request;
    var uri = request.uri;

    // Check whether the URI is missing a file name.
    if (uri.endsWith('/')) {
        request.uri += 'index.html';
    }
    // Check whether the URI is missing a file extension.
    else if (!uri.includes('.')) {
        request.uri += '/index.html';
    }

    return request;
}

Reference: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/example-function-add-index.html

Charley

Answer 12

I had this problem when I was trying to include the bucket name in my route53 redirect for a customer, to make it easier e.g:

https://0832234.signin.aws.amazon.com/console/s3/?bucket=clientbucket.com

Clicking on "All buckets" and going back into the clients bucket / or removing the bucket from the url did the trick, I can now download and open the files.

Obnoxious