How to view full dependency tree for nested Go dependencies

B

6

42

I'm trying to debug the following build error in our CI where "A depends on B which can't build because it depends on C." I'm building my data service which doesn't directly depend on kafkaAvailMonitor.go which makes this error hard to trace. In other words:

data (what I'm building) depends on (?) which depends on kafkaAvailMonitor.go

It may seem trivial to fix for a developer they just do "go get whatever" but I can't do that as part of the release process - I have to find the person that added the dependency and ask them to fix it.

I'm aware that there are tools to visualize the dependency tree and other more sophisticated build systems, but this seems like a pretty basic issue: is there any way I can view the full dependency tree to see what's causing the build issue?

go build -a -v

../../../msgq/kafkaAvailMonitor.go:8:2: cannot find package 
  "github.com/Shopify/sarama/tz/breaker" in any of:
  /usr/lib/go-1.6/src/github.com/Shopify/sarama/tz/breaker (from $GOROOT)
  /home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker (from $GOPATH)
  /home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker
  /home/jenkins/go/src/github.com/Shopify/sarama/tz/breaker
  /home/jenkins/vendor-library/src/github.com/Shopify/sarama/tz/breaker

Bulky answered 13/6, 2017 at 12:7 Comment(4)

Or are you looking for go list ? Take a look at Imports and Deps. – Thaddeusthaddus 13/6, 2017 at 12:30

I'm not sure why this is diverging into a discussion about the technical meaning of stack trace - this always happens to me on SO. I have a CI job building dozen of services (that I didn't write) and this is what I see in the logs but that doesn't help me. I have no idea who is importing kafkaAvailMonitor.go i.e. who is breaking the build. go list was a helpful suggestion but didn't show anything - would love to know more about that. – Bulky 13/6, 2017 at 16:8

go list -f '{{ .GoFiles }}' github.com/Shopify/sarama/tz/breaker – Claribelclarice 27/6, 2017 at 13:44

The OP isn't looking for a stack trace, he wants the dependency tree. – Allonym 1/7, 2017 at 15:24

N

25

if the following isn't a stack trace what is it?

It is the list of path where Go is looking for your missing package.

I have no idea who is importing kafkaAvailMonitor.go

It is not "imported", just part of your sources and compiled.
Except it cannot compile, because it needs github.com/Shopify/sarama/tz/breaker, which is not in GOROOT or GOPATH.

Still, check what go list would return on your direct package, to see if kafkaAvailMonitor is mentioned.

go list can show both the packages that your package directly depends, or its complete set of transitive dependencies.

% go list -f '{{ .Imports }}' github.com/davecheney/profile
[io/ioutil log os os/signal path/filepath runtime runtime/pprof]
% go list -f '{{ .Deps }}' github.com/davecheney/profile
[bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect run

You can then script go list in order to list all dependencies.
See this bash script for instance, by Noel Cower (nilium)

#!/usr/bin/env bash
# Usage: lsdep [PACKAGE...]
#
# Example (list github.com/foo/bar and package dir deps [the . argument])
# $ lsdep github.com/foo/bar .
#
# By default, this will list dependencies (imports), test imports, and test
# dependencies (imports made by test imports).  You can recurse further by
# setting TESTIMPORTS to an integer greater than one, or to skip test
# dependencies, set TESTIMPORTS to 0 or a negative integer.

: "${TESTIMPORTS:=1}"

lsdep_impl__ () {
    local txtestimps='{{range $v := .TestImports}}{{print . "\n"}}{{end}}'
    local txdeps='{{range $v := .Deps}}{{print . "\n"}}{{end}}'

    {
        go list -f "${txtestimps}${txdeps}" "$@"
        if [[ -n "${TESTIMPORTS}" ]] && [[ "${TESTIMPORTS:-1}" -gt 0 ]]
        then
            go list -f "${txtestimps}" "$@" |
            sort | uniq |
            comm -23 - <(go list std | sort) |
                TESTIMPORTS=$((TESTIMPORTS - 1)) xargs bash -c 'lsdep_impl__ "$@"' "$0"
        fi
    } |
    sort | uniq |
    comm -23 - <(go list std | sort)
}
export -f lsdep_impl__

lsdep_impl__ "$@"

Ninanincompoop answered 26/6, 2017 at 20:45 Comment(5)

Are you familiar with the build "output" of other language like Java/C#/Python etc? Where you build something and it tells you the dependency chain e.g. A import B at line 3 which imports C at line 2? The issue with your answer is that I have to run "go list" on every dependency so if A depends on B depends on C etc. I have go all the way down each one which on a large project is impossible. The whole point is that I don't want to add tz/breaker - someone is adding it by mistake but I can't figure out who because it's buried under several other dependencies. – Bulky 26/6, 2017 at 22:14

"The issue with your answer is that I have to run "go list" on every dependency so if A depends on B depends on C etc": that is easily scriptable, I have edited the answer accordingly. – Ninanincompoop 27/6, 2017 at 4:34

"The whole point is that I don't want to add tz/breaker - someone is adding it by mistake". Someone? That "someone" is kafkaAvailMonitor.go, line 8. The question becomes: do you need kafkaAvailMonitor.go – Ninanincompoop 27/6, 2017 at 4:35

I think the point is: the OP doesn't know what is depending on kafkaAvailMonitor.go. – Allonym 1/7, 2017 at 15:27

@Fli hence the script, in order to list all dependencies on all packages, in order to find the one importing kafkaAvailMonitor – Ninanincompoop 1/7, 2017 at 17:10

S

51

When using modules you may be able to get what you need from go mod graph.

usage: go mod graph

Graph prints the module requirement graph (with replacements applied)
in text form. Each line in the output has two space-separated fields: a module
and one of its requirements. Each module is identified as a string of the form
path@version, except for the main module, which has no @version suffix.

I.e., for the original question, run go mod graph | grep github.com/Shopify/sarama then look more closely at each entry on the left-hand side.

Stretcher answered 24/7, 2020 at 12:56 Comment(0)

N

25

if the following isn't a stack trace what is it?

It is the list of path where Go is looking for your missing package.

I have no idea who is importing kafkaAvailMonitor.go

It is not "imported", just part of your sources and compiled.
Except it cannot compile, because it needs github.com/Shopify/sarama/tz/breaker, which is not in GOROOT or GOPATH.

Still, check what go list would return on your direct package, to see if kafkaAvailMonitor is mentioned.

go list can show both the packages that your package directly depends, or its complete set of transitive dependencies.

% go list -f '{{ .Imports }}' github.com/davecheney/profile
[io/ioutil log os os/signal path/filepath runtime runtime/pprof]
% go list -f '{{ .Deps }}' github.com/davecheney/profile
[bufio bytes errors fmt io io/ioutil log math os os/signal path/filepath reflect run

You can then script go list in order to list all dependencies.
See this bash script for instance, by Noel Cower (nilium)

#!/usr/bin/env bash
# Usage: lsdep [PACKAGE...]
#
# Example (list github.com/foo/bar and package dir deps [the . argument])
# $ lsdep github.com/foo/bar .
#
# By default, this will list dependencies (imports), test imports, and test
# dependencies (imports made by test imports).  You can recurse further by
# setting TESTIMPORTS to an integer greater than one, or to skip test
# dependencies, set TESTIMPORTS to 0 or a negative integer.

: "${TESTIMPORTS:=1}"

lsdep_impl__ () {
    local txtestimps='{{range $v := .TestImports}}{{print . "\n"}}{{end}}'
    local txdeps='{{range $v := .Deps}}{{print . "\n"}}{{end}}'

    {
        go list -f "${txtestimps}${txdeps}" "$@"
        if [[ -n "${TESTIMPORTS}" ]] && [[ "${TESTIMPORTS:-1}" -gt 0 ]]
        then
            go list -f "${txtestimps}" "$@" |
            sort | uniq |
            comm -23 - <(go list std | sort) |
                TESTIMPORTS=$((TESTIMPORTS - 1)) xargs bash -c 'lsdep_impl__ "$@"' "$0"
        fi
    } |
    sort | uniq |
    comm -23 - <(go list std | sort)
}
export -f lsdep_impl__

lsdep_impl__ "$@"

Ninanincompoop answered 26/6, 2017 at 20:45 Comment(5)

Are you familiar with the build "output" of other language like Java/C#/Python etc? Where you build something and it tells you the dependency chain e.g. A import B at line 3 which imports C at line 2? The issue with your answer is that I have to run "go list" on every dependency so if A depends on B depends on C etc. I have go all the way down each one which on a large project is impossible. The whole point is that I don't want to add tz/breaker - someone is adding it by mistake but I can't figure out who because it's buried under several other dependencies. – Bulky 26/6, 2017 at 22:14

"The issue with your answer is that I have to run "go list" on every dependency so if A depends on B depends on C etc": that is easily scriptable, I have edited the answer accordingly. – Ninanincompoop 27/6, 2017 at 4:34

"The whole point is that I don't want to add tz/breaker - someone is adding it by mistake". Someone? That "someone" is kafkaAvailMonitor.go, line 8. The question becomes: do you need kafkaAvailMonitor.go – Ninanincompoop 27/6, 2017 at 4:35

I think the point is: the OP doesn't know what is depending on kafkaAvailMonitor.go. – Allonym 1/7, 2017 at 15:27

@Fli hence the script, in order to list all dependencies on all packages, in order to find the one importing kafkaAvailMonitor – Ninanincompoop 1/7, 2017 at 17:10

P

11

I just want to mention here that go mod why can also help. Anyway you cannot get and display the whole tree. But you can trace back one single branch of a child dependency until its parent root.

Example:

$ go mod why github.com/childdep
# github.com/childdep
github.com/arepo.git/service
github.com/arepo.git/service.test
github.com/anotherrepo.git/mocks
github.com/childdep

That means, you have imported 'childdep' finally in 'anotherrepo.git/mocks'.

Penetrant answered 4/4, 2022 at 8:20 Comment(0)

M

9

can try this https://github.com/vc60er/deptree

 redis git:(master) go mod graph | deptree -d 3
package: github.com/go-redis/redis/v9
dependence tree:

┌── github.com/cespare/xxhash/[email protected]
├── github.com/dgryski/[email protected]
├── github.com/fsnotify/[email protected]
│    └── golang.org/x/[email protected]
├── github.com/nxadm/[email protected]
│    ├── github.com/fsnotify/[email protected]
│    │    └── golang.org/x/[email protected]
│    └── gopkg.in/[email protected]
├── github.com/onsi/[email protected]
│    ├── github.com/go-task/[email protected]
│    │    ├── github.com/davecgh/[email protected]
│    │    └── github.com/stretchr/[email protected]
│    │         └── ...

Mcfadden answered 13/7, 2022 at 9:6 Comment(1)

Excellent, Go built-in feature :) – Didymous 22/11, 2022 at 22:6

B

2

The above answer still doesn't show me a dependency tree so I've taken the time to write a Python script to do what I need - hopefully that helps other people.

The issue with the above solution (the others proposed like go list) is that it only tells me the top level. They don't "traverse the tree." This is the output I get - which doesn't help any more than what go build gives me.

.../npd/auth/
   .../mon/mlog
   .../auth/service

This is what I'm trying to get - I know that auth is broken (top) and that breaker is broken (bottom) from go build but I have no idea what's in between - my script below gives me this output.

.../npd/auth/
    .../npd/auth/service                
        .../npd/auth/resource
            .../npd/auth/storage
               .../npd/middleware
                  .../npd/metrics/persist
                    .../npd/kafka
                        .../vendor-library/src/github.com/Shopify/sarama
                            .../vendor-library/src/github.com/Shopify/sarama/vz/breaker

My Python script:

import subprocess
import os

folder_locations=['.../go/src','.../vendor-library/src']

def getImports(_cwd):
    #When the commands were combined they overflowed the bugger and I couldn't find a workaround
    cmd1 = ["go", "list", "-f", " {{.ImportPath}}","./..."]
    cmd2 = ["go", "list", "-f", " {{.Imports}}","./..."]

    process = subprocess.Popen(' '.join(cmd1), cwd=_cwd,shell=True,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)

    out1, err = process.communicate()

    process = subprocess.Popen(' '.join(cmd2), cwd=_cwd,shell=True,
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE)

    out2, err = process.communicate()
    out2clean=str(out2).replace("b'",'').replace('[','').replace(']','').replace("'",'')

    return str(out1).split('\\n'),out2clean.split('\\n')

def getFullPath(rel_path):
    for i in folder_locations:
        if os.path.exists(i+'/'+rel_path):
            return i+'/'+rel_path
    return None

def getNextImports(start,depth):

    depth=depth+1
    indent = '\t'*(depth+1)

    for i,val in enumerate(start.keys()):

        if depth==1:
            print (val)

        out1,out2=getImports(val)

        noDeps=True
        for j in out2[i].split(' '):
            noDeps=False
    
            _cwd2=getFullPath(j)
            new_tree = {_cwd2:[]}
            not_exists = (not _cwd2 in alltmp)
    
            if not_exists:
                print(indent+_cwd2)
                start[val].append(new_tree)
                getNextImports(new_tree,depth)
                alltmp.append(_cwd2)

        if noDeps:
            print(indent+'No deps')

_cwd = '/Users/.../npd/auth'

alltmp=[]
start_root={_cwd:[]}
getNextImports(start_root,0)

Bulky answered 6/7, 2017 at 22:25 Comment(2)

Yes - the the real disappointment is that go list has a recursive option but it flattens the tree - so you can see all the dependencies but not the order. go list -f "{{.ImportPath}} {{.Deps}}" ./... – Bulky 7/7, 2017 at 12:41

not required, since there's a Go built-in feature (see answer from @Gordon). but for the record: good job on the Pythonscript – Didymous 22/11, 2022 at 22:8

S

0

go mod graph and https://github.com/vc60er/deptree can only handle go mod, but NOT package.

I find the tool depth can handle package dependency tree in command line.

And it supports -explain to show the package import chain, like go mod why.

https://github.com/KyleBanks/depth depth is tool to retrieve and visualize Go source code dependency trees.

If you want show the dependencies in dot graph, try godepgraph. It runs faster but you need dot installed.

https://github.com/kisielk/godepgraph godepgraph is a program for generating a dependency graph of Go packages.

Sabbatarian answered 8/4 at 6:14 Comment(0)

Recommended topics

Hot tags