S3, Signed-URLs and Caching

Question

I am generating signed urls on my webapp (nodejs) using the knox nodejs-library. However the issue arises, that for every request, I need to regenerate an unique GET signed url for the current user, leaving browser's cache-control out of the game.

I've searched the web without success as browsers seem to use the full url as caching key so I am really curious how I can, under the given circumstances (nodejs, knox library) get the issue solved and use caching control while still being able to generated signed urls for each and every request as I need to verify the user's access rights.

I cannot believe there's no solution to that though.

Oceanic

One suggestion: leave the signing part out of the URL, and add custom http headers for authentication. – Proscription

I am curious wether this is possible at all with the knox library though – Oceanic

have the same conundrum, Did you find a way to deal with this/ – Demogorgon

Answer 1

I am working with Java AmazonS3 client, but the process should be the same.

There is a strategy that can be used to handle this situation.

You could use a fixed date time as an expiration date. I set this date to tomorrow at 12 pm.

Now every time you generate a url, it will be the same throughout that day until 00:00. That way browser caching can be used to some extent.

Markley

Nice way. Full explanation: bennadel.com/blog/… – Chevet

I use this function s3.getSignedUrl("getObject", {Bucket: "mybucket", Key:'myImage", Expires:18000} ) which specifies the url to expire in 180000seconds or 5 hours. How to set fixed date time instead of a amount of time as expiration? – Kailyard

@AdarshMadrecha answer is the way. I tested his answer and it works. Read the link he provided: advancedweb.hu/cacheable-s3-signed-urls. It has detailed explanation of the approach. – Kailyard

Answer 2

Expanding @semir-deljić Answer.

Every time we call getSignedUrl function, it will generate new URLs. This will result in images not being cached even if Cache Control header is present.

Thus, we are using timekeeper library to freeze time. Now when the function is called, it thinks that the time has not passed, and it returns same URL.

const moment = require('moment');
const tk = require("timekeeper");

function url4download(awsPath, awsKey) {

  function getFrozenDate() {
    return moment().startOf('week').toDate();
  }

  // Paramters for getSignedUrl function
  const params = {
    // Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
    // Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
    Bucket: awsBucket,
    Key: `${awsPath}/${awsKey}`,
    // 604800 == 7 days
    ResponseCacheControl: `public, max-age=604800, immutable`,
    Expires: 604800, // 7 days is max
  };

  const url = tk.withFreeze(getFrozenDate(), () => {
    return S3.getSignedUrl('getObject', params);
  });
  return url;
}

Note: Using moment().toDate(), as the timekeeper requires a Native Date Object.

Even tough the question is for using knox library, my answer uses aws official library.

// This is how the AWS & S3 is initiliased.
const AWS = require('aws-sdk');

const S3 = new AWS.S3({
  accessKeyId: awsAccessId,
  secretAccessKey: awsSecretKey,
  region: 'ap-south-1',
  apiVersion: '2006-03-01',
  signatureVersion: 'v4',
});

Inspiration: https://advancedweb.hu/cacheable-s3-signed-urls/

Chevet

This IS THE ANSWER. The ONE key concept of how this works: trick AWS S3's getSignedUrl to think its in a "preset" time in the past. E.g. of preset time is every 10 minutes interval in the day, e.g. 1pm, 1:10pm, 1:20pm, etc. Practical example: when a user requests an image at 1:02pm, server sets to nearest preset time (use timekeeper) which is 1pm and run s3.getSignedUrl(); this is as if, server run s3.getSignedUrl() at 1pm; a request at 1.04pm will run s3.getSignedUrl() at 1pm and so on. Therefore, making all URLs generated from 1pm to 1:10pm the same! I tested it, it works! – Kailyard

I must say the phrase "freeze time" is misleading. timekeeper's role here is more like a time traveller that can travel to a specified time. The code tk.withFreeze(time, callback) sets the Date object to specified time and then runs callback. So, what is useful with timekeeper is, you can be in any point in time, you can travel back or forward to a specified point in time and run your callback – Kailyard

This is wonderful. – Susansusana

Answer 3

If you use CloudFront with S3, you can use a Custom Policy, if you restrict each url to the user's IP and a reasonably long timeout, it means that when they request the same content again, they will get the same URL and hence their browser can cache the content but the URL will not work for someone else (on a different IP).

(see: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-creating-signed-url-custom-policy.html)

Inflection

This wouldn't work for me as users may change their IP quite frequently :( – Oceanic

@Oceanic did you find a solution. I am having same problem in android. – Demodulator

Answer 4

You can create your own browser cache logic by using window.caches

See an example in this other stackoverflow question

Kimberleekimberley

Answer 5

When calculating signed URL you can set 'signingDate' to a fixed moment in the past e.g. yesterday morning, then calculate expiration from that moment. Don't forget to use UTC and account for timezones.

import { S3Client, GetObjectCommand, GetObjectCommandInput } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

let signingDate = new Date();
signingDate.setUTCHours(0, 0, 0, 0);
signingDate.setUTCDate(signingDate.getUTCDate() - 1);

let params: GetObjectCommandInput = {
    Bucket: BUCKET_NAME,
    Key: filename
};
const command = new GetObjectCommand(params);
const url = await getSignedUrl(s3Client,
    command, {
        expiresIn: 3 * 3600 * 24, // 1 day until today + 1 expiration + 1 days for timezones
        signableHeaders: new Set < string > (),
        signingDate: signingDate
    });

Shrewmouse