Getting all dependencies of a Go project

Question

For my project I am trying to get all the dependencies and sub dependencies of my project. I need to specific version of each of these dependencies. Not only do I need the dependencies of my project, but the dependencies of the dependencies and so on until the root.

For my project, go list -m all works for everything except indirect dependencies that have not opted into using go.mod files. Right now my workflow is taking an initial batch of repositories, downloading them from git then using “GO111MODULE=on go build ./…”. and “GO111MODULE=on go list -m -json all” to get the list of dependencies. I do not check for go.mod as all of the repositories I am scanning are using go.mod files.

For the list of dependencies that come out of this initial list I have some questions, for files without go.mod files, I used this as a reference: “https://blog.golang.org/using-go-modules”

-Path = Received from go list -m all, it can be GitHub, gopkg, or whatever is used to dl the go package.

Without go.mod

-“GO111MODULE=on go mod init <PATH from parent go.mod>”

-“GO111MODULE=on go build ./…”

-“GO111MODULE=on go mod tidy”

-“GO111MODULE=on go list -m -json all”

-From there I get a list of the dependencies of this module. 

With go.mod

-“GO111MODULE=on go build ./…”

-“GO111MODULE=on go mod tidy”

-“GO111MODULE=on go list -m -json all”

Should I be running go build on each dependency that has a go.mod file? For ones without a go.mod file, I understand this should be done, as how else will we populate the go.mod file with the dependencies. But for files with a go.mod file, will I be pulling extra stuff that is not necessarily being used by my project with go build, like test files and other files that might not be used when I am simply importing that project? I understand that its better to get more unused dependencies rather than missing some, but it is getting a bit overwhelming with how massive the amount of dependencies is.

Kicksorter

go list works no matter if a package is module or not just do not use the -m flag. For the rest of the questions: There is nothing to worry about, all works fine. Unused dependencies are not a problem and stuff inside a package but unused during your build also is not a problem to worry about. – Logorrhea

Having some difficulty understanding the question. I think the answer is "no, I don't think you should be running go build on each dep that uses go modules". Perhaps you can understand why better from this: github.com/golang/go/wiki/… – Collide

The issue with using go list is that it does not show the version of the dependency used, which is something I need and is why I am using the -m tag – Kicksorter

It has no notion of non-module dependencies' versions. It only understands versions for modules. – Aftercare

does it help: https://mcmap.net/q/302449/-what-39-s-the-go-mod-equivalent-of-npm-outdated ? – Adman

Answer 1

I can try to analyze go.sum file (when you execute go list -u, go.sum was created)

The go command uses the go.sum file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both go.mod and go.sum should be checked into version control. (Using Go Modules - Adding a dependency)

go.sum file lists down the checksum (and version tag) of direct and indirect dependency required by the module.

% cat go.sum
...
github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b/go.mod h1:ac9efd0D1fsDb3EJvhqgXRbFx7bs2wqZ10HQPeU8U/Q=
github.com/c2h5oh/datasize v0.0.0-20171227191756-4eba002a5eae/go.mod h1:S/7n9copUssQ56c7aAgHqftWO4LTf4xY6CGWt8Bc+3M=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ=
...

Deadhead

Answer 2

The phrase “all the dependencies” is unfortunately ambiguous.

go list -m lists all modules whose selected versions are determined by your go.mod file. That is one possible definition of "all the dependencies", but it is a much broader set of modules than I think most people intend when they talk about the “dependencies” of a module.

In practice, go list -test all (without the -m flag) is the broadest set of dependencies I would generally care about: it includes all of the packages listed in import statements within your module (i.e. everything you need in order to run go test ./... within your module), plus all of the packages transitively needed to run go test on those packages.

(In particular, go list -test all is also the set of packages that will be resolved when you run go mod tidy.)

Gothar

Answer 3

Not only do I need the dependencies of my project, but the dependencies of the dependencies and so on until the root.

For compiling or running your project?
Because you also have tools, which don't directly build or run your project, but assist your source generation or static analysis (linter).

For that, you have a proposal for "Adding tool dependencies to go.mod".
It is accepted and implemented in golang/go issue 48429.
You will have go get -tool example.com/m1 to add a tool line to your mod file and add any missing dependencies.

And when go tool is run in module mode with an argument that does not match a Go built-in tool, it will search the current for a tool directive that matches the last path segment. It will then compile and run that tool similarly to go run.

Persistence