How can we work with multiple DBT projects

Question

can anyone share how to organise multiple projects in dbt , given the best practices. My present abstract architecture hierarchy is as follows

Analytics 
--.dbt/
-----profiles.yml
--projects/
-----project_1/
----------models/
----------dbt_project.yml
-----project_2/
----------models/
----------dbt_project.yml
--tests/
-----projects/
----------project_1/
----------project_2/

To the point of creating models for either projects_1 or project_2 is working perfectly fine.

but the problem comes when i try to run the test (model unit tests) for project_2.when I run it gives the error ' no dbt_project.yml found at expected path in temp/project_1/ ... ' (the unit tests that i am trying to run is for project_2) however, the paths are absolutely correct but the lookup for dbt_project.yml is in wrong directory (in the temp directory). On the side note, some of the project_2 models does depends on few project_1 models.

can anyone share or reference where i can get the help to solve this sort of multi dbt project issue.

Utta

Answer 1

Like many things, what you're asking here is two sub questions:

1. Is it possible?
2. Is it a good idea?

Is it possible?

While you didn't mention what data platform / dialect you are running on, yes, it is possible. Examples and recommendations here:

• dbt Discourse: Can we use multiple Bigquery datasets (or projects) in the same dbt project?

Is it a good idea?

I think Jeremy breaks out the main arguments for and against really well in this post reply:

• dbt Discourse: Should I have an organisation wide project (a monorepo) or should each work flow have their own?

But if you've decided that you are already going to do this and don't want a monorepo and you're most interested in the "best" way to implement this, I'd recommend this post by Stefano:

• Practical tips to get the best out of Data Build Tool

TLDR: Yes, it's possible(ish). Best suggested methods are managing the generate_schema_name / generate_database_name macros OR have low level models defined in packages that your main projects imports.

Airlee