Put file to Amazon S3 using multipart upload

Question

I'm trying to upload a file with the Amazon Java SDK, via multipart upload. The idea is to pass an upload-id to an applet, which puts the file parts into a readonly-bucket. Going this way, I avoid to store AWS credentials in the applet.

In my tests, I generate an upload-id with boto (python) and store a file into the bucket. That works well.

My Applet gets a "403 Access denied" from the S3, and I have no idea why.

Here's my code (which is partially taken from http://docs.amazonwebservices.com/AmazonS3/latest/dev/llJavaUploadFile.html):

AmazonS3 s3Client = new AmazonS3Client();
List<PartETag> partETags = new ArrayList<PartETag>();

long contentLength = file.length();
long partSize = Config.getInstance().getInt("part_size");
String bucketName = Config.getInstance().getString("bucket");
String keyName = "mykey";
String uploadId = getParameter("upload_id");

try {
    long filePosition = 0;
    for (int i = 1; filePosition < contentLength; i++) {

        partSize = Math.min(partSize, (contentLength - filePosition));

        // Create request to upload a part.
        UploadPartRequest uploadRequest = new UploadPartRequest()
            .withBucketName(bucket).withKey(keyName)
            .withUploadId(uploadId).withPartNumber(i)
            .withFileOffset(filePosition)
            .withFile(file)
            .withPartSize(partSize);

        // Upload part and add response to our list.
        partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());

        filePosition += partSize;
    }

    System.out.println("Completing upload");
    CompleteMultipartUploadRequest compRequest = new 
                CompleteMultipartUploadRequest(bucket, 
                                            keyName, 
                                            uploadId, 
                                            partETags);

    s3Client.completeMultipartUpload(compRequest);
} catch (Exception e) {
    s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(
            bucketName, keyName, uploadId));
}

In the applet debug log, I find this, then:

INFO: Sending Request: PUT https://mybucket.s3.amazonaws.com /mykey Parameters: (uploadId: V4hwobOLQ1rYof54zRW0pfk2EfhN7B0fpMJTOpHOcmaUl8k_ejSo_znPI540.lpO.ZO.bGjh.3cx8a12ZMODfA--, partNumber: 1, ) Headers: (Content-Length: 4288546, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 
24.01.2012 16:48:42 com.amazonaws.http.AmazonHttpClient handleErrorResponse
INFO: Received error response: Status Code: 403, AWS Service: null, AWS Request ID: DECF32CCFEE9EBF0, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: xtL1ixsGM2/vsxJ+cZRHpkPZ23SMfP8hZZjQCQnp8oWGwdS2/aGfYgomihyqaDCQ

Do you find any obvious failures in the code?

Thanks, Stefan

Hasseman

Answer 1

While your use case is sound and this is an obvious attempt indeed, I don't think the Multipart Upload API has been designed to allow this and you are actually violating a security barrier:

The upload ID is merely an identifier to assist the Multipart Upload API in assembling the parts together (i.e. more like a temporary object key) not a dedicated security mechanism (see below). Consequently you still require proper access credentials in place, but since you are calling AmazonS3Client(), which Constructs a new Amazon S3 client that will make anonymous requests to Amazon S3, your request yields a 403 Access denied accordingly.

What you are trying to achieve is possible via Uploading Objects Using Pre-Signed URLs, albeit only without the multipart functionality, unfortunately:

A pre-signed URL gives you access to the object identified in the URL, provided that the creator of the pre-signed URL has permissions to access that object. That is, if you receive a pre-signed URL to upload an object, you can upload the object only if the creator of the pre-signed URL has the necessary permissions to upload that object.

[...] The pre-signed URLs are useful if you want your user/customer to be able upload a specific object [...], but you don't require them to have AWS security credentials or permissions. When you create a pre-signed URL, you must provide your security credentials, specify a bucket name an object key, an HTTP method (PUT of uploading objects) and an expiration date and time. [...]

The lenghty quote illustrates, why a system like this likely needs a more complex security design than 'just' handing out an upload ID (as similar as both might appear at first sight).

Obviously one would like to be able to use both features together, but this doesn't appear to be available yet.

Clarisclarisa

Thanks a lot for your thoughts. Even since it's possible to perform anonymous requests to S3 if the bucket is publicly available, your assumption seems true, that multipart uploads are just not made for my case. And that pre-signed URLs are not available for multipart-uploads is too bad. However, I decided to use the AWS IAM mechanism to set a writeonly-policy for the bucket, and to store the credentials for a new, accordingly configurated user in the applet. From security perspective, it should be fine. – Hasseman

@schneck: Facilitating an IAM write-only policy is an excellent alternative indeed, I've focused too much on how you are trying to achieve your goal rather than the actual use case - you could take that even further by Making Requests Using IAM User Temporary Credentials to entirely avoid storing permanent credentials within your applet. – Clarisclarisa

Isn't the danger with a write-only, public bucket is that someone could spam your bucket using nothing more than curl? Or am I missing something? – Sholem