Attempting to convert some code to Go CDK when using KMS

Question

I have some code to upload and download files from Google Cloud Storage. Below is an abbreviated example:

import (
    "context"
    "io"
    "cloud.google.com/go/storage"
)

func upload(bucket, keyName, path string, reader io.Reader) error {
    ctx := context.Background()
    client, err := storage.NewClient(ctx)

    if err != nil {
        return err
    }
    defer client.Close()

    obj := client.Bucket(bucket).Object(path)
    writer := obj.NewWriter(ctx)

    defer writer.Close()

    writer.KMSKeyName = keyName

    if _, err = io.Copy(writer, reader); err != nil {
        return err
    }
    if err = writer.Close(); err != nil {
        return err
    }
    return nil
}

The tricky part is that I'm using Google KMS to manage the keys I'm using to encrypt files (Google's so-called "customer-managed encryption key" scheme). My understanding is that this encryption happens on Google's end.

The only solution I found using the Go CDK was to encrypt the files using Google KMS and then upload the encrypted blob. Is there no way to specify the encryption key in the same manner I was doing before with the Go CDK?

Thanks

Lippi

Answer 1

If you need to use provider-specific settings in the Go CDK, you can use the various As functions to get handles to the underlying provider API. In this case, you would want to use the blob.WriterOptions.BeforeWrite option. The benefit of doing it this way is that the BeforeWrite will no-op if you decide to switch bucket providers later (e.g. for unit testing).

import (
    "context"
    "io"

    "cloud.google.com/go/storage"
    "gocloud.dev/blob"
    _ "gocloud.dev/blob/gcsblob" // link in "gs://" URLs
)

func upload(ctx context.Context, bucket, keyName, path string, reader io.Reader) error {
    bucket, err := blob.OpenBucket(ctx, "gs://" + bucket)
    if err != nil {
        return err
    }
    defer bucket.Close()

    writeCtx, cancelWrite := context.WithCancel(ctx)
    defer cancelWrite()
    writer, err := bucket.NewWriter(writeCtx, path, &blob.WriterOptions{
        // Use BeforeWrite to set provider-specific properties.
        BeforeWrite: func(asFunc func(interface{}) bool) error {
            var gcsWriter *storage.Writer
            // asFunc returns true if the writer can be converted to the type
            // pointed to.
            if asFunc(&gcsWriter) {
                gcsWriter.KMSKeyName = keyName
            }
            return nil
        },
    })
    if err != nil {
        return err
    }
    if _, err := io.Copy(writer, reader); err != nil {
        cancelWrite()  // Abort the write to the bucket.
        writer.Close()
        return err
    }
    if err := writer.Close(); err != nil {
        return err
    }
    return nil
}

(While not directly related to your question, I added in code to abort the write on error to avoid partial objects being uploaded to your storage provider. We're adding docs that will demonstrate how to do common tasks with Go CDK APIs in the future, see #1576.)

Hussey

Answer 2

The code you posted is correct. Specifically, this line instructs the API to encrypt data using the provided Cloud KMS key:

writer.KMSKeyName = keyName

A few things that might be getting in the way:

• Make sure your Cloud KMS key and and Cloud Storage bucket are in the same region. For example, if you have a storage bucket in "US", your KMS key must also be in "US".

• Make sure you've granted your Cloud Storage service account access to use the KMS key. You can find the email of your Cloud Storage service account and then grant it access to use your KMS key:

  $ gcloud kms keys add-iam-policy-binding my-key \
      --project my-project \
      --keyring my-keyring \
      --location us \
      --member serviceAccount:[email protected] \
      --role roles/cloudkms.cryptoKeyEncrypterDecrypter 
  

• Make sure you're catching the error returned from upload upstream. All of these failure modes result in an error, so I'm wondering if that's getting lost somewhere.

Proof that it works:

// Completely unmodified version of your `upload` function
func main() {
    bucket := "<hidden>"
    kmsKey := "projects/<hidden>/locations/us/keyRings/kr/cryptoKeys/k"
    if err := upload(bucket, kmsKey, "foo", strings.NewReader("hello world")); err != nil {
        log.Fatal(err)
    }
}

When executed, this correctly creates an object in GCS backed by CMEK.

Jews