How to copy whole s3 folders using php aws sdk?

Question

I want copy a folder in aws s3 bucket. I am able to copy files but requirement is I should be able to copy whole s3 folder with some other name. example: if folder name is project1 I need to copy it as project1-copy in bucket called my_bucket. I have tried below code.

$s3->copyObject([
    'Bucket'     => my_bucket,
    'Key'        => "project-copy",
    'CopySource' => "my_bucket/project1",
]);

and also reverse as

$s3->copyObject([
    'Bucket'     => my_bucket,
    'Key'        => "project1",
    'CopySource' => "my_bucket/project-copy",
]);

In both case I am getting the error as

Error executing "CopyObject" on "https://s3.us-east-1.amazonaws.com/my_bucket/project1/"; AWS HTTP error: Client error: PUT https://s3.us-east-1.amazonaws.com/my_bucket/project1/ resulted in a 404 Not Found response: NoSuchKeyThe specified key does not exist.34/F992013F6C3125 (truncated...) NoSuchKey (client): The specified key does not exist. - NoSuchKeyThe specified key does not exist.34/F992013F6C312535YLrCcHhYJzimIKoR2knMvD3smhUbH75piDYTIx5SpKCPNkeWIifaYhKgezxnhpZzOHflzQngTnE=There was an error uploading

And I am sure of I am using right region and folder names also bucket name. Still getting this above error. How can I resolve this.?

Coworker

Answer 1

Folders do not exist in Amazon S3.

Instead, the Key (filename) of an object includes the full path, eg:

s3merahkee/project1/foo.txt

If you wish to copy an entire 'folder', then you will need to copy all files objects within that path.

Also, please note that you do not need to pre-create folders to store files. You could run a command like this:

aws s3 cp foo.txt s3://my-bucket/s3merahkee/project1/foo.txt

This command would 'create' the s3merahkee and project1 folders. Well, actually it doesn't create them at all because they don't exist, but the Amazon S3 management console and even the aws s3 ls command will make it 'appear' as though those folders exist. If you were to delete the file, those folders would immediately disappear (because they don't actually exist).

The easiest way to copy a whole folder is to use the AWS Command-Line Interface (CLI), like this:

aws s3 cp --recursive s3://my-bucket/folder1 s3://my-bucket/folder2

Eglanteen

But I would like to have code in php aws sdk not CLI – Coworker

Fine. Then you'll have to copy all the files in the 'folder'. There is no function for copying folders. – Eglanteen

Answer 2

Here is explaned how to copy multiple objects within Amazon S3, from one bucket to another or within the same bucket using PHP SDK: https://docs.aws.amazon.com/AmazonS3/latest/dev/CopyingObjectUsingPHP.html

If briefly, create a batch of operations form the list of files:

$batch[] = $s3->getCommand('CopyObject', [
    'Bucket'     => $targetBucket,
    'Key'        => "{targetKeyname}-{$i}",
    'CopySource' => "{$sourceBucket}/{$sourceKeyname}",
]);

and send that batch with \Aws\CommandPool::batch($s3, $batch);

Gannie

Answer 3

The accepted answer is correct in the sense that an object in S3 contains a full file path. There is however a way to do this using the SDK so you don't need to use the command line.

AWS suggests using the Transfer object to do this: https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/s3-transfer.html

Uncrown

The Transfer object doesn't allow for s3 to s3 actions. It's only for between client and s3 transactions. – Portion