How should I use vendor in Go 1.6?

Question

First I have read this answer: Vendoring in Go 1.6, then I use it as my example.

My gopath is GOPATH="/Users/thinkerou/xyz/", and the follow like:

thinkerou@MacBook-Pro-thinkerou:~/xyz/src/ou$ pwd
/Users/baidu/xyz/src/ou
thinkerou@MacBook-Pro-thinkerou:~/xyz/src/ou$ ls
main.go vendor

Now, I use go get, then becomes this:

thinkerou@MacBook-Pro-thinkerou:~/xyz/src/ou$ ls
main.go vendor
thinkerou@MacBook-Pro-thinkerou:~/xyz/src/ou$ cd vendor/
thinkerou@MacBook-Pro-thinkerou:~/xyz/src/ou/vendor$ ls
vendor.json
thinkerou@MacBook-Pro-thinkerou:~/xyz/src/ou/vendor$ cd ../..
thinkerou@MacBook-Pro-thinkerou:~/xyz/src$ ls
github.com ou
thinkerou@MacBook-Pro-thinkerou:~/xyz/src$ cd github.com/
thinkerou@MacBook-Pro-thinkerou:~/xyz/src/github.com$ ls
zenazn

vendor.json is this:

{
    "comment": "",
    "package": [
        {
            "path": "github.com/zenazn/goji"
        }
    ]
}

then, I should use what commands? why have no use vendor? My go version is 1.6.2.

Clash

What tool are you trying to use to do the vendoring? go get doesn't natively vendor packages so you will probably want to use one of the tools listed here: github.com/golang/go/wiki/PackageManagementTools – Chainman

OK, I have read it, thanks a lot @Chainman – Clash

Answer 1

With Go1.6, vendoring is built in as you read. What does this mean? Only one thing to keep in mind:

When using the go tools such as go build or go run, they first check to see if the dependencies are located in ./vendor/. If so, use it. If not, revert to the $GOPATH/src/ directory.

The actual "lookup paths" in Go 1.6 are, in order:

./vendor/github.com/zenazn/goji
$GOPATH/src/github.com/zenazn/goji
$GOROOT/src/github.com/zenazn/goji

With that said, go get will continue to install into you $GOPATH/src; and, go install will install into $GOPATH/bin for binaries or $GOPATH/pkg for package caching.

So, how do I use ./vendor?!?!

Hehe, armed with the knowledge above, it's pretty simple:

mkdir -p $GOPATH/src/ou/vendor/github.com/zenazn/goji
cp -r $GOPATH/src/github.com/zenazn/goji/ $GOPATH/src/ou/vendor/github.com/zenazn/goji

In short, to use vendoring, you copy the files using the same github.com/zenazn/goji full path, into your vendor director.

Now, the go build/install/run tooling will see and use your vendor folder.

An easier way instead of copying everything manually

Instead of finding and copying all 25+ vendor items, managing their versions, updating other projects etc... It would be better to use a dependency management tool. There are many out there and a little googling will point to you several.

Let me mention two that works with the vendor folder and doesn't fight you:

• godep
• govendor

In short, these tools will inspect your ou code, find the remote dependencies, and copy them from your $GOPATH/src to your $GOPATH/src/ou/vendor directory (actually, whatever current directory you are in when you run them).

For example, say you have all of your dependencies installed and working normally in your $GOPATH/src/ou/ project using the normal GOPATH/src/github installation of your dependencies. Your project runs and your tests validate everything is working with the exact version of the repos you have. With Godep as an example, you'd run this from your project root folder $GOPATH/src/ou/:

godep save ./...

This would copy all dependencies your project uses into your ./vendor folder.

Godep is by far and large the most popular. They have their own Slack channel on the Gopher Slack group. And, it's the one I use on my teams.

Govendor is another alternative I read has a nice sync feature. I haven't used it though.

Over Usage of Dependency Management Tool

This is purely opinion, and I'm sure haters will downvote... But as I need to finish my blog post on the subject, let me mention here that most people worry too much about depdency management in Go.

Yes, there is a need to lock in a repo to a version you depend on so you can ensure your system builds in production. Yes there is a need to ensure no breaking changes to a way a dependency is interrupting something.

Use dependency management for those, absolutely.

But, there is overuse of simple projects that lock in huge amounts of dependencies when in reality...

You may only need to lock in only 1 dependencies; otherwise, you want the latest version of MySQL drivers and test assertion frameworks for bug fixes.

This is where using the ./vendor/ folder apart from dependency managrment tools can really shine: you'd only need to copy that repo that need you lock in.

You selectively pick the one misbehaving repo and put it into your ./vendor/ folder. By doing this, you are telling your consumers:

Hey, this one repo needs to be held back at this revision. All others are fine and use the latest of those and update often with go get -u ./...; but, this one failed with newer versions so don't upgrade this one repo.

But if blanketly saving all your dependencies with a dependency management tool, you are basically telling your consumers:

There may or may not be a problem with one or more repos out of the 20 in the vendor folder. You may or may not be able to update them. You may or may not be able to get the latest MySQL driver. We simply don't know which may or may not be causing problems and just locked in something that worked at the time that I ran godep save. So yeah, upgrade at your own risk.

Personally, I have ran into this several times. A dependency was updated with a breaking change, and we have dozens of repos dependent on it. Vendoring just that one repo in /vendor allows us to use that one version of dependency, while go get ./... continues to run normally for all other repos to get the latest. We run with the latest bug fixes in PSQL and MySQL and others (there are constant fixes for these!) and so on.

Macrography

Anyway, if I not use other Dependency Management Tool, I should use copy method. So the vendor/vendor.json file is used Dependency Management Tool, not used the command of go. In fact, I want to use vendor but not want to use Dependency Management Tool, this need to use copy file method. – Clash

Yes, just copy the file using the same github.com/zenazn/goj path, like vendor/github.com/zenazn/goj – Macrography

I am so confused, because every where I read it says go always checks in vendor/ first. But that is simply not case for me using go 1.7 on Windows. It totally ignores anything I have in my vendor directory? $GOPATH/src/vendor, if I stand in the src/ dir and do go run it simply won't look there – Sergei

@Sergei i am pretty sure the directories in the $GOPATH/src/ folder are treated as projects themselves. In your example, ./src/vendor is treated as a project itself named "vendor". Instead, create a project folder,like $GOPATH/src/acmecorp/. Then, a vendor folder in it, like $GOPATH/src/acmecorp/vendor. Then CD to $GOPATH/src/acmecorp/ and build ur project. Read up on how to organize your Go code: golang.org/doc/code.html#Organization – Macrography

Just to clarify, does ./vendor/ refers to any folder named vendor inside actual package sources (as opposed to in $GOPATH but not in $GOPATH/src/)? Is this recursive? (ie: $GOPATH/src/acmecorp/some_sub_package/vendor) or must be exactly one level down from your 'project' (ie: exactly $GOPATH/src/acmecorp/vendor)? – Amaranthaceous

@Amaranthaceous sorry, i'll edit to clarify when back at laptop tomorrow. ./vendor/ refers to the root of your project - yes, the root of the actual package source. Technically, it is not recursive: you only have 1 at the root location you go get or go install from, meaning if u have multiple go get of multiple sub-packages, then each can have their own ./vendor folder. Just one per project. For example, a fatuous CLI installed from $GOPATH/src/github.com/eduncan911/mspec/cmd can have one vendor folder at $GOPATH/src/github.com/eduncan911/mspec/cmd/vendor. – Macrography

Good point on dependency management overkill! No one even hated :D – Castanon

When you copy the code of the module to your vendor directory, keep in mind that if the original code was a GitHub repo it will include a .git and .github folder on it. If you commit those directories to your repo and push it to GitHub the code will be registered as a submodule and not be in your GitHub repo. This might be OK if you want a submodule, but if you don't just delete the .git and .github folders from the vendor directory before you push to GitHub. – Wolverine

Just a note, your own code cannot live outside of the $GOPATH structure for vendor to work. – Selfpollination

What is your source on the first statement? I don't doubt it, but for reference. :) – Spanish

@Jonas My source is me in that statement. :) If you follow the logic through the source code, and see enough error messages of when something is missing, you derive the facts I state in the answer above. – Macrography

Okay, well, if you want one, seems you can use golang.org/src/go/build/build.go?s=9943:9963#L352 which is part of the go build code. – Spanish