How to set up a staging environment on Google App Engine

Question

Having properly configured a Development server and a Production server, I would like to set up a Staging environment on Google App Engine useful to test new developed versions live before deploying them to production.

I know two different approaches:

A. The first option is by modifying the app.yaml version parameter.

version: app-staging

What I don't like of this approach is that Production data is polluted with my staging tests because (correct me if I'm wrong):

1. Staging version and Production version share the same Datastore
2. Staging version and Production version share the same logs

_{Regarding the first point, I don't know if it could be "fixed" using the new namespaces python API.}

B. The second option is by modifying the app.yaml application parameter

application: foonamestaging

with this approach, I would create a second application totally independent from the Production version.
The only drawback I see is that I'm forced to configure a second application (administrators set up).
With a backup\restore tool like Gaebar this solution works well too.

What kind of approach are you using to set up a staging environment for your web application?
Also, do you have any automated script to change the yaml before deploying?

Narcolepsy

Note that option B may be a violation of the google app engine TOS. – Oeildeboeuf

@bdolan do you have any reference about it? – Narcolepsy

code.google.com/appengine/terms.html 4.4. You may not develop multiple Applications to simulate or act as a single Application or otherwise access the Service in a manner intended to avoid incurring fees. – Oeildeboeuf

@bdonlan, IANAL, but I don't think @system's use can be construed as "intended to avoid incurring fees" nor to "simulate or act as a single Application". @systempuntoout, my "staging" is actually done from an SDK (I don't want the staging open to the world at large), but your approach seems fine too (if you're not having problems with the "open to the world" part;-). – Penholder

Logs are already separated by version, so there's no "pollution" there. Using namespaces to avoid corrupting production data sounds like a great idea. Unless of course the bug in your staging version has to do with your namespacing being horribly broken. – Mcfarlin

@Mcfarlin You are definitely right, corrected thanks. There's a version combobox near the application combobox. – Narcolepsy

We go with option B, but use app-staging as the staging name. We've got Fabric deployment script that'll automatically adjust the app ID in app.yaml depending on our deployment targets. Though I'd like to investigate using option A plus the new datastore namespace functionality. That seems much cleaner. – Trackandfield

@Narcolepsy I wish I could upvote your question more than once. Without it I wouldn't ever have thought of using namespaces for staging data. Thanks! – Frady

@Frady Uhm, how do you deal with backups using the option A? Are you going to backup Staging and Production data together? – Narcolepsy

@Narcolepsy I think - pure speculation because I've yet to try option A - that you could easily backup data separately with the bulk loader --namespace=NAMESPACE flag. – Frady

Using multiple apps for staging/development environments is fine, and it's the solution most users end up going with, because of isolation issues with testing things using multiple versions. – Southwest

whats with the dates on this qu; edited Sep 27 '10 at 19:33 – Stanfordstang

i thought app engine was a recent thing – Stanfordstang

Answer 1

I chose the second option in my set-up, because it was the quickest solution, and I didn't make any script to change the application-parameter on deployment yet.

But the way I see it now, option A is a cleaner solution. You can with a couple of code lines switch the datastore namespace based on the version, which you can get dynamically from the environmental variable CURRENT_VERSION_ID as documented here: http://code.google.com/appengine/docs/python/runtime.html#The_Environment

Francesco

I also went with the 2nd option in the question, but I started before separate namespaces were available as well. – Lachrymator

@Narcolepsy I think it's possible : "The bulk loader supports a --namespace=NAMESPACE flag that allows you to specify the namespace to use. Each namespace is handled separately and, if you want to access all namespaces, you will need to iterate through them." from code.google.com/appengine/docs/python/multitenancy/… – Frady

@Frady sounds interesting, so option A seems feasible and clean – Narcolepsy

Other Uses for the Namespaces API : Creating Separate Datastore Instances for Testing and Production – Narcolepsy

The only problem with this is if you want to move from 'staging' to 'production', you have to redeploy your app, since they use different namespaces. Of course, you have to do that with separate apps, too. – Southwest

Answer 2

If separate datastore is required, option B looks cleaner solution for me because:

1. You can keep versions feature for real versioning of production applications.
2. You can keep versions feature for traffic splitting.
3. You can keep namespaces feature for multi-tenancy.
4. You can easily copy entities from one app to another. It's not so easy between namespaces.
5. Few APIs still don't support namespaces.
6. For teams with multiple developers, you can grant upload to production permission for a single person.

Atheistic

+1 for 6. e.g. I save auth for my personal credentials that can deploy to dev, I have to type the live account & password in every time. – Trygve

I also use option B. However, while trying to start using push-to-deploy, I find it hard to use this approach, as I need to keep one app.yaml per environment (before push-to-deploy, I just had a script that changed the application param before deploying, and then set it back), or use different branches, each with (potentially conflicting) app.yamls – Peraea

Is there an updated answer to this question as the application tag has been removed from app.yaml? – Ottinger

Answer 3

We went with the option B. And I think it is better in general as it isolates the projects completely. So for example playing around with some of the configurations on the staging server will not affect and wont compromise security or cause any other butterfly effect in your production environment.

As for the deployment script, you can have any application name you want in your app.yaml. Some dummy/dev name and when you deploy, just use an -A parameter:

appcfg.py -A your-app-name update .

That will simplify your deploy script quite much, no need to string replace or anything similar in your app.yaml

Credulity

Answer 4

We use option B.

In addition to Zygmantas suggestions about the benefits of separating dev from prod at application level, we also use our dev application to test performance.

Normally the dev instance runs without much available in the way of resources, this helps to see where the application "feels" slow. We can then also independently tweak the performance settings to see what makes a difference (e.g. front-end instance class).

Of course sometimes we need to bite the bullet and tweak & watch on live. But it's nice to have the other application to play with.

Still use namespaces and versions, just dev is dirty and experimental.

Trygve

Answer 5

Here is what the Google documentation says :

A general recommendation is to have one project per application per environment. For example, if you have two applications, "app1" and "app2", each with a development and production environment, you would have four projects: app1-dev, app1-prod, app2-dev, app2-prod. This isolates the environments from each other, so changes to the development project do not accidentally impact production, and gives you better access control, since you can (for example) grant all developers access to development projects but restrict production access to your CI/CD pipeline

With this in mind, add a dispatch.yaml file at the root directory, and in each directory or repository that represents a single service and contain that service, add a app.yaml file along with the associated source code, as explained here : Structuring web services in App Engine

Edit, check out the equivalent link in the python section if you're using python.

Reiterate

Answer 6

No need to create a separate project. You can use dispatch.yaml to route your staging URL to another service (staging) in the same project.

1. Create a custom domain staging.yourdomain.com

2. Create a separate app-staging.yaml, that specifies staging service.

   ... service: staging ...

3. Create distpatch.yaml that contains something like

   ...

   • url: "*staging.mydomain.com/" service: staging

   • url: "*mydomain.com/" service: default ...

4. gloud app deploy app-staging.yaml dispatch.yaml

Samaritan

While this may be a solution technically speaking, that's not what the Google documentation recommends – Reiterate

Answer 7

use of application in app.yaml has been shut down.

Instead Google recommends gcloud app deploy --project [YOUR_PROJECT_ID]

Please see https://cloud.google.com/appengine/docs/standard/python/config/appref

Kierkegaardian