Google Cloud Storage Signed URLs with Google App Engine

Question

It's frustrating to deal with the regular Signed URLs (Query String Authentication) for Google Cloud Storage.

Google Cloud Storage Signed URLs Example -> Is this really the only code available in the whole internet for generating Signed URLs for Google Cloud Storage? Should I read it all and adapt it manually for Pure Python GAE if needed?

It's ridiculous when you compare it with AWS S3 getAuthenticatedURL(), already included in any SDK...

Am I missing something obvious or does everyone face the same problem? What's the deal?

Bates

Why do you need a signed URL in the first place? – Caralie

@AndreiVolgin I don't want to require my users to have google accounts. I just need temporary authenticated URLs. – Bates

@AndreiVolgin It's an interesting solution, but I'll have to pay for instance hours this way, instead of just serving the file directly from GCS. If my app wasn't hosted in GAE, I'd also have to pay for network transfer costs... – Bates

If you have many users, use a Compute Engine instance - it's many times cheaper. If you don't have many users yet, you may be within a free quota on GAE. – Caralie

Answer 1

Here's how to do it in Go:

func GenerateSignedURLs(c appengine.Context, host, resource string, expiry time.Time, httpVerb, contentMD5, contentType string) (string, error) {
    sa, err := appengine.ServiceAccount(c)
    if err != nil {
        return "", err
    }
    expUnix := expiry.Unix()
    expStr := strconv.FormatInt(expUnix, 10)
    sl := []string{
        httpVerb,
        contentMD5,
        contentType,
        expStr,
        resource,
    }
    unsigned := strings.Join(sl, "\n")
    _, b, err := appengine.SignBytes(c, []byte(unsigned))
    if err != nil {
        return "", err
    }
    sig := base64.StdEncoding.EncodeToString(b)
    p := url.Values{
        "GoogleAccessId": {sa},
        "Expires": {expStr},
        "Signature": {sig},
    }
    return fmt.Sprintf("%s%s?%s", host, resource, p.Encode()), err
}

Terbium

Answer 2

I have no idea why the docs are so bad. The only other comprehensive answer on SO is great but tedious.

Enter the generate_signed_url method. Crawling down the rabbit hole you will notice that the code path when using this method is the same as the solution in the above SO post when executed on GAE. This method however is less tedious, has support for other environments, and has better error messages.

In code:

def sign_url(obj, expires_after_seconds=60):

    client = storage.Client()
    default_bucket = '%s.appspot.com' % app_identity.get_application_id()
    bucket = client.get_bucket(default_bucket)
    blob = storage.Blob(obj, bucket)

    expiration_time = int(time.time() + expires_after_seconds)

    url = blob.generate_signed_url(expiration_time)

    return url

Maupin

Answer 3

I came across this problem recently as well and found a solution to do this in python within GAE using the built-in service account. Use the sign_blob() function in the google.appengine.api.app_identity package to sign the signature string and use get_service_account_name() in the same package to get the the value for GoogleAccessId.

Don't know why this is so poorly documented, even knowing now that this works I can't find any hint using Google search that it should be possible to use the built-in account for this purpose. Very nice that it works though!

Sullivan

Thank you very much. This works great, without pycrypto. Even on the SDK. – Leggy

And my code is here : #29848259 – Leggy

Answer 4

Check out https://github.com/GoogleCloudPlatform/gcloud-python/pull/56

In Python, this does...

import base64
import time
import urllib
from datetime import datetime, timedelta

from Crypto.Hash import SHA256
from Crypto.PublicKey import RSA
from Crypto.Signature import PKCS1_v1_5
from OpenSSL import crypto

method = 'GET'
resource = '/bucket-name/key-name'
content_md5, content_type = None, None

expiration = datetime.utcnow() + timedelta(hours=2)
expiration = int(time.mktime(expiration.timetuple()))

# Generate the string to sign.
signature_string = '\n'.join([
  method,
  content_md5 or '',
  content_type or '',
  str(expiration),
  resource])

# Take our PKCS12 (.p12) key and make it into a RSA key we can use...
private_key = open('/path/to/your-key.p12', 'rb').read()
pkcs12 = crypto.load_pkcs12(private_key, 'notasecret')
pem = crypto.dump_privatekey(crypto.FILETYPE_PEM, pkcs12.get_privatekey())
pem_key = RSA.importKey(pem)

# Sign the string with the RSA key.
signer = PKCS1_v1_5.new(pem_key)
signature_hash = SHA256.new(signature_string)
signature_bytes = signer.sign(signature_hash)
signature = base64.b64encode(signature_bytes)

# Set the right query parameters.
query_params = {'GoogleAccessId': '[email protected]',
                'Expires': str(expiration),
                'Signature': signature}

# Return the built URL.
return '{endpoint}{resource}?{querystring}'.format(
    endpoint=self.API_ACCESS_ENDPOINT, resource=resource,
    querystring=urllib.urlencode(query_params))

Funk

Is there something like this for google app engine python? gcloud seems to have too much overhead? – Runofthemill

I just had a little think about this - the error is "ImportError: No module named OpenSSL" - however you're only using crypto to convert the p12 to a pem key, so I'm just going to generate my pem key offline & upload that to app engine. So I should be able to remove those dependencies – Runofthemill

Answer 5

And if you don't want to write it by your own checkout this class on GitHub.

Really easy to use

GCSSignedUrlGenerator

Nations