dbt: check what packages are installed

Question

Is there a way to check what packages are installed? I would expect something like dbt list packages?

The context is:

• Until I run dbt deps the content of packages.yml gives me nothing. And there are some situations when the models could be triggered without running dbt deps
• I would like to check the packages in runtime

I searched over google and dbt --help but I didn't find anything.

Liturgy

Not sure I understand your comment about packages.yml because to include any packages you must first add them to this list. If you are using imported packages then you should ALWAYS run dbt deps as a first step. – Blatt

@NathanGriffiths basically I want to check what the packages are REALLY installed in runtime, not the ones that SHOULD be installed – Liturgy

There is no way to do that I am aware of, and I don't understand why you would need to do that if you're running dbt in the intended manner. – Blatt

Answer 1

Recently I had the opportunity to work with packages closely and I published an article The Practical Guide to Utilizing DBT Packages for Data Transformation which explains dbt packages and its underhood

Here is the quote:

To verify that a package is installed in your dbt project, you can check the packages.yml file and run the dbt deps command.

1. Check the packages.yml file: This file lists all of the packages that are installed in your dbt project. Look for the name of the package you want to verify. If it is listed in the packages list, then it is installed.

2. Run the dbt deps command:

   1. This command will show you a list of all of the packages that are installed in your dbt project. Look for the name of the package you want to verify. If it is listed, then it is installed.
   2. In the root dbt project dir, you observe a new dir dbt_modules/ which contains the compiled packages that are ready to be used. NOTE: dir dbt_modules/ has to be added to .gitignore.

>>> tree -L 1 .
.
├── data
├── dbt_modules
├── dbt_project.yml
├── macros
├── models
├── packages
├── packages.yml
├── profiles.yml
├── snapshots
└── target

If your packages.yml file contains package that is not installed then you would not be able to run any dbt command:

>>> dbt list
Encountered an error:
Compilation Error
  dbt found 1 package(s) specified in packages.yml, but only 0 package(s) installed in dbt_modules. Run dbt deps to install package dependencies.

So this is our guarantee that in runtime we would not have any issues related to the package installation.

Liturgy

Answer 2

No, there is no dbt list packages command; there is no --list option for dbt deps; packages are not listed in dbt debug; installed packages are not listed in manifest.json; nor is there another officially supported analog of pip list to determine all of the currently-installed packages in your runtime.

You should open an issue for this!

Your current options (v1.3.0) are:

1. Run dbt clean and dbt deps before any other dbt command, to ensure your installed dependencies match what is in your packages.yml file.
2. Run dbt compile and inspect manifest.json to ensure it has the macro(s) and/or model(s) you need from a package (note that dbt compile will fail if packages are listed in packages.yml and not installed).
3. Inspect your project's dbt_packages subdirectory. Each installed package will have its own directory; if that package has any dependencies, they will be nested under subsequent dbt_packages directories.

dbt clean and dbt deps are pretty fast, so the cost of (1) is low, and you don't have to manage the logic/complexity of various bad states if you go that route. But I think you should absolutely open an issue for this!

Contrayerva

I suggest that the reason dbt doesn't provide any of the things you list in the first paragraph is that the intended usage of dbt is option 1) i.e. always run dbt deps as a first step. If this is done then I can't imagine a scenario where it would be necessary to programatically query the "installed" packages. Can you describe a use case where this would be necessary, e.g. some kind of deployment where it's impossible to run dbt deps? – Blatt

Answer 3

You should have a packages.yml file in your repo that contains all the packages that are installed via dbt deps.

Following the dbt deps, a directory called /dbt_modules (by default, can be changed in your dbt_project.yml) will appear in your repo (it is included, also by default, in your .gitignore file to avoid duplicating the source code for the package).

Spavined

Please, read my initial message and thought about packages.yml: Until I run dbt deps the content of packages.yml gives me nothing. And there are some situations when the models could be triggered without running dbt deps – Liturgy

packages.yml is the “list” you are looking for, there is no more. Once you run ‘dbt deps’, packages are installed, and it is in that moment that a ‘dbt run’ will run any of the models included in those installed packages. – Spavined

packages.yml is the list of the libs that SHOULD be installed. but it's not necessary will be installed. Imagine, you have package1 in the above-mentioned file. For some reason, you didn't run dbt deps. So it means that you cannot rely on the content of this file. What I'm looking for - is the way to check what libs are already installed and are ready to be used. – Liturgy