How do I run DBT models from a Python script or program?

Question

I have a DBT project, and a python script will be grabbing data from the postgresql to produce output.

However, part of the python script will need to make the DBT run. I haven't found the library that will let me cause a DBT run from an external script, but I'm pretty sure it exists. How do I do this?

ETA: The correct answer may be to download the DBT CLI and then use python system calls to use that.... I was hoping for a library, but I'll take what I can get.

Polyanthus

Answer 1

Update: v1.5 has arrived!

With v1.5 of dbt, we get a stable and officially supported Python API for invoking dbt operations; this API has functional parity with the CLI.

From the docs:

from dbt.cli.main import dbtRunner, dbtRunnerResult

# initialize
dbt = dbtRunner()

# create CLI args as a list of strings
cli_args = ["run", "--select", "tag:my_tag"]

# run the command
res: dbtRunnerResult = dbt.invoke(cli_args)

# inspect the results
for r in res.result:
    print(f"{r.node.name}: {r.status}")

There are some caveats about the stability of artifacts returned by dbt.invoke; read the docs for more details.

Original Answer

(As of Jan 2023) There is not a public Python API for dbt, yet. It is expected in v1.5, which should be out in a couple months.

Right now, your safest option is to use the CLI. If you don't want to use subprocess, the CLI uses Click now, and Click provides a runner that you can use to invoke Click commands. It's usually used for testing, but I think it would work for your use case, too. The CLI command is here. That would look something like:

from click.testing import CliRunner
from dbt.cli.main import run

dbt_runner = CliRunner()
dbt_runner.invoke(run, args="-s my_model")

You could also invoke dbt the way they do in the test suite, using run_dbt.

Sapient

"(As of Jan 2023) There is not a public Python API for dbt, yet. It is expected in v1.5, which should be out in a couple months." Can you provide a source for this? Thanks! – Sulphurbottom

github.com/dbt-labs/dbt-core/milestone/82 – Sapient

github.com/dbt-labs/dbt-core/issues/… – Divisive

Answer has been updated to reflect the new state of the art in v1.5 – Sapient

How do I invoke a dbt command from a file in a different directory than the dbt project? – Hayfield

dbt_runner.invoke(["run", "--project-dir", "/path/to/dir"]) should work just fine. – Sapient

@NicholasHansen-Feruch @Sapient you may also have to specify the --profiles-dir arg – Gyrose

@Sapient - have you had success with programmatically invoking run-operation (["run-operation", "my_macro", "--args", f"""{{"param1": "{x}"}}"""]) and retrieving the return value? The dbtRunnerResult doesn't have anything relevant to the return value of the macro. Thank you. – Personable