Is it possible to use Go vendor libraries with Google App Engine?

Question

I'm trying to deploy a small test app to Google App Engine (standard). I get an error if I include any vendor libraries.

This is the error I get when trying to deploy

% gcloud app deploy
Services to deploy:

descriptor:      [/Users/matt/work/appenginetest1/src/hello/default/app.yaml]
source:          [/Users/matt/work/appenginetest1/src/hello/default]
target project:  REDACTED
target service:  [default]
target version:  [20170709t220721]
target url:      REDACTED


Do you want to continue (Y/n)?  y

Beginning deployment of service [default]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 0 files to Google Cloud Storage                ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Deployment contains files that cannot be compiled: Compile failed:
go-app-builder: build timing: 2×compile (210ms total), 0×link (0s total)
go-app-builder: failed running compile: exit status 2

main.go:6: can't find import: "github.com/julienschmidt/httprouter"

For some context, this is the tree of $GOPATH

% tree $GOPATH
/Users/matt/work/appenginetest1
└── src
    └── hello
        ├── default
        │   ├── app.yaml
        │   └── main.go
        ├── glide.lock
        ├── glide.yaml
        └── vendor
            └── github.com
                └── julienschmidt
                    └── httprouter
                        ├── path.go
                        ├── router.go
                        └── tree.go

Running the local server via dev_appserver.py works fine. It doesn't look like a case of app engine not finding the vendor directory as after running

% rm -rf ~/work/appenginetest1/src/hello/vendor/github.com

It errors before even pushing to the cloud

% gcloud app deploy
ERROR: (gcloud.app.deploy) Staging command [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/goroot-1.6/bin/go-app-stager /Users/matt/work/appenginetest1/src/hello/default/app.yaml /var/folders/nx/8w2_6q551cl50h3ff6lmy9s40000gn/T/tmp97Kiis/tmpe0MHQ0] failed with return code [1].

------------------------------------ STDOUT ------------------------------------
------------------------------------ STDERR ------------------------------------
2017/07/09 22:12:52 failed analyzing /Users/matt/work/appenginetest1/src/hello/default: cannot find package "github.com/julienschmidt/httprouter" in any of:
    /Users/matt/work/appenginetest1/src/hello/vendor/github.com/julienschmidt/httprouter (vendor tree)
    ($GOROOT not set)
    /Users/matt/work/appenginetest1/src/github.com/julienschmidt/httprouter (from $GOPATH)
GOPATH: /Users/matt/work/appenginetest1
--------------------------------------------------------------------------------

If I move the github.com directory from vendor to src, the deploy works without an issue.

app.yaml

service: default
runtime: go
api_version: go1


handlers:
- url: /.*
  script: _go_app

main.go

package hello

import (
  "fmt"
  "net/http"
  "github.com/julienschmidt/httprouter"
)

func init() {
  router := httprouter.New()
  router.GET("/hello/:name", Hello)
  http.Handle("/", router)
}

func Hello(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
  fmt.Fprintf(w, "Hello, %s!", ps.ByName("name"))
}

I can get this project to deploy if I move the github.com directory from vendor to src after running glide install.

Polycotyledon

Matt, did you ever get a sane solution to this problem? – Hibbs

Answer 1

You can address this by using your $GOPATH variable. Note that your $GOPATH environment variable can actually be a list of multiple places (see https://golang.org/cmd/go/#hdr-GOPATH_environment_variable for more):

The GOPATH environment variable lists places to look for Go code. On Unix, the value is a colon-separated string. On Windows, the value is a semicolon-separated string. On Plan 9, the value is a list.

Rather than moving the github.com directory from vendor to src, you could write a script which temporarily adds the vendor/github.com directory to your $GOPATH environment variable, performs the deployment, and then removes it from the environment variable.

Mcmasters

Answer 2

Following from the answer by rudolph1024 one method I use to push projects to app engine standard is to add a few lines in the make file. The most simple example being:

GOPATH=./vendor gcloud app deploy

This has the bonus of allowing you to use the go community standard repo layout from here instead of the google one here

app engine standard likes the app.yaml to be in the root directory and all go files under it will be used. So you can do something like

cp -r cmd/ ../../cmd  && cd ../../cmd && GOPATH=./vendor gcloud app deploy 

There are many ways to clean this up but this works and gives a general idea on the methodology.

Barksdale