Does Amazon S3 support symlinks?

Question

I have an object which I would like to address using different keys without actually copying the object itself, like a symlink in Linux. Does Amazon S3 provide such a thing?

Milinda

Answer 1

S3 does not support the notion of a symlink, where one object key is treated as an alias for a different object key. (You've probably heard this before: S3 is not a filesystem. It's an object store).

If you are using the static web site hosting feature, there is a partial emulation of this capability, with object-level redirects:

http://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html

This causes requests for "object-a" to be greeted with a 301 Moved Permanently response, with the URL for "object-b" in the Location: header, which serves a similar purpose, but is of course still quite different. It only works if the request arrives at the website endpoint (not the REST endpoint).

If you use a reverse proxy (haproxy, nginx, etc.) in EC2 to handle incoming requests and forward them to the bucket, then of course you have the option at the proxy layer of rewriting the request URL before forwarding to S3, so you could translate the incoming request path to whatever you needed to present to S3. How practical this is depends on your application and motivation, but this is one of the strategies I use to modify where, in a particular bucket, an object appears, compared to where it is actually stored, allowing me to rewrite paths based on other attributes in the request.

Honorable

Answer 2

I had a similar question and needed a solution, which I describe below. While S3 does not support symlinks, you can do this in a way with the following:

echo "https://s3.amazonaws.com/my.bucket.name/path/to/a/targetfile" > file
aws s3 cp file s3://my.bucket.name/file
wget $(curl https://s3.amazonaws.com/my.bucket.name/file)

What this is actually doing is getting the contents of the file, which is really just a pointer to the target file, then passing that to wget (curl can also be used to redirect to a file instead of wget).

This is really just a work around though as its not a true symlink but rather a creative solution to simulate symlinks.

Homophile

Can this be used for static sites served from S3 ? – Mertens

That's not a symbolic link. That's a straight up copy. There is a meaningful, consequential difference – Glassman

@Glassman Have a closer look. What's being stored at s3://my.bucket.name/file is the string literal https://s3.amazonaws.com/my.bucket.name/path/to/a/targetfile, not the contents of the file. The curl command retrieves that link and passes it to wget. So there is both indirection and no duplication of the file contents, as with a symbolic link. – Bouilli

and there are a lot of drawbacks now: (1) you have more files in S3, (2) some files now only contain PATHS to other files and nothing more, (3) every end-user knows everything about the file structure (paths to all pseudo-links and to actual files), and (4) now you run MORE HTTP-requests (because of HTTP-redirects). So it is not actual "symlink" , but (as @Homophile wrote) a workaround. Which actually can do the trick sometimes. – Sabaean

(5) This doesn't support symlinks pointing to symlinks. – Pamulapan

Answer 3

Symlinks no, but same object to multiple keys, maybe.

Please refer to Rodrigo's answer at Amazon S3 - Multiple keys to one object

If you're using the website serving on S3, you can do it via header x-amz-website-redirect-location

If you're not using the website serving, you can create your custom header (x-amz-meta-KeyAlias) and handle it manually.

Sophisticated

Too bad the metadata feature is not available for the bucket object itself. Someone named our bucket without considering DNS configuration and 10GB + data won't be easy (free) to copy. I guess it will need to be mv'd instead, but reluctant to change bucket name because third parties are using it... WTF Amazon? – Tar