Is there a way to touch() a file in Amazon S3?

Question

I'm currently working with Amazon S3 and I am writing a program that uses the modified dates. I'm looking for a way to edit the modified dates.

I could loop trough all the files and save them as they are, but this sounds like a bad solution.

In PHP there is this function touch().

Does anyone know a solution, or has the same problem?

Gypsophila

Answer 1

In response to @Daniel Golden's comment on @tkotisis answer. It looks like at least the AWS CLI tools do not let you copy an item on to itself. You can however 'force' a copy by updating the metadata.

$ aws s3 cp --metadata '{"touched":"now"}' s3://path/to/object s3://path/to/object

This recreates the object (downloads to the caller and reuploads it) replacing its content, owner and metadata. This will also trigger any attached Lambda events.

Appel

This results in an error: fatal error: An error occurred (404) when calling the HeadObject operation: Key "index.html" does not exist. – Portraiture

This won't work nicely if the S3 bucket versioning is enabled. It would create a duplicate copy... – Osbourne

This doesn't work if the goal is to CREATE the object (instead of just touching an existing object). – Autocephalous

Answer 2

You can achieve the same through a copy object request, specifying the CopySource to be same as the target key.

In essence, this will issue a PUT Object - COPY request to S3 with the corresponding source and target bucket/key.

Morphia

And do you know what exactly this does? Does it fire a GET and a PUT request for each file? – Gypsophila

Edited my answer to include this info. – Morphia

On running this command: s3cmd cp s3://path/to/file s3://path/to/file where the two paths are the same, I get this error: ERROR: S3 error: 400 (InvalidRequest): This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes. – Diphosgene

Quite different environment, but in luigi + python + spark you do that like this: fs.put_string("", path) Where the fs is an S3Client (is available through eg. S3Target). Basically you can put empty string to S3. – Apiece

Using aws client java iibrary, I got Daniel's error too. To workaround, I did a copy to a new key, delete old key etc, etc – Finespun

@DanielGolden Have you set the x-amz-metadata-directive request header to REPLACE, as noted in the docs? – Morphia

This worked - thanks. aws s3 cp s3://path/to/file s3://path/to/file --metadata-directive REPLACE – Milka

Answer 3

I find myself performing the copy trick pretty often when testing, to the extent that I've added a handy function to my .bashrc:

s3-touch() {
  aws s3 cp \
    --metadata 'touched=touched' \
    --recursive --exclude="*" \
    --include="$2" \
    "${@:3}" \
    "$1" "$1"
}

Example usage:

# will do a dryrun on a copy operation
s3-touch s3://bucket/prefix/ "20200311*" --dryrun

# the real thing, creating events for all objects
# in s3://bucket/prefix/ that start with 20200311
s3-touch s3://bucket/prefix/ "20200311*"

I'm doing this mainly for the S3-events that I want to trigger.

Spectacled

Answer 4

Here is another whay to upload a null (or o Byte) file to S3. I verified this works You can also use the S3 API to upload a file with no body, like so:

aws s3api put-object --bucket "myBucketName" --key "dir-1/my_null_file"

Normally you would specify a --body blob, but its option and will just add the key as expected. See more on S3 API put-object

The version of AWS CLI tested with is: aws-cli/2.0.4 Python/3.7.5 Windows/10 botocore/2.0.0dev8

Here's how I did it in PHP (even works in outdated 5.4, had to go way back):

// Init an S3Client
$awsConfig = $app->config('aws');
$aws       = Aws::factory($awsConfig);
$s3Bucket  = $app->config('S3_Bucket');
$s3Client  = $aws->get('s3');

// Set null/empty file.
$result = $s3Client->putObject([
    'Bucket' => $s3Bucket,
    'Key' => "dir-1/my_null_file",
    'Body' => '',
    'ServerSideEncryption' => 'AES256',
]);

Spool

Finally, an answer that worked for me! Thank you! – Hargrove

This doesn't actually answer the title question. touch should not change the contents of the file. It should only update the modification time if the file exists whereas this will truncate an existing file. – Octroi

Marked down as this doesn't answer the actual stated problem. – Fractocumulus

Answer 5

check out https://github.com/emdgroup/awscli-s3touch

It's a plugin to the AWS CLI that adds a touch command.

Usage:

aws s3 touch my-bucket --prefix myfolder/

It works by reading the events attached to the bucket and simulating them client side.

Lauranlaurance

Reading the source code, that plugin is just triggering the events that should be triggered once the PUT operation occurs, without actually touching the files. Misleading repo name – Culbreth

Answer 6

Following @g-io answer that simplified my day, here is another version of the same that makes it easy to touch a single file

s3-touch-single() {
  aws s3 cp \
    --metadata 'touched=touched' \
    "${@:3}" \
    "$1" "$1"
}

for example, looping an array of files we need to touch:

paths=("mydir/image.png" "mydir2/image2.png")
for i in "${paths[@]}"; do s3-touch-single "s3://my-bucket/$i"; done

Culbreth

Cool is there a way to do this with the java api ? – Cruikshank

Q: Should the 4th line read "${@:2}" or is the second parameter for something I've missed? – Afternoon

@Afternoon honestly i can't remember why and it does look redundant when looking at the script now – Culbreth

@Cruikshank you could use the java sdk in a similar way (just using java instead of bash). it's probably using the same APIs at the backend for both. – Culbreth

@BenYitzhaki Interesting I tried doing that that is changing a field in the metadata, then copying an object to itself with the new metadata. This does change the metadata but does NOT change the summary.getLastModified() value on subsequent access. – Cruikshank

Answer 7

I know this is old, but I just came across this and wasted some time before going to the docs

You can pipe stdin and use - for <source>

echo "" | aws s3 cp - s3://bucket-name/filename.txt

Source: https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-objects-copy

Republicanize

Answer 8

Add user-defined metadata to the bucket or folder or file inside. It will recursively touch all of the files to update the metadata and thus update the date.

example:

type = user defined
key = x-amz-meta-version
value = 1.0.1

Sacrifice