DBT on Databricks Unity Catalog

Question

I've been considering turning on Databricks Unity Catalog in our primary (only) workspace, but I'm concerned about how this might impact our existing dbt loads with the new three-part object references.

I see from the dbt-databricks release notes that you need >= 1.1.1 to get unity support.
The snippet with it only shows setting the catalog property in the profile. I was planning on having some of the sources in separate catalog's for the dbt generated objects.

I might even choose to have the dbt generated objects in separate catalogues if this was available.
As turning on Unity Catalog is a one way road in a workspace, I don't wish to wing it and see what happens.

Has anyone used dbt with Unity Catalog and used numerous catalogs in the project?

If so, are there any gotcha's and how do you specify the catalog for sources and specific models?

Regards,

Ashley

Whack

Answer 1

specifying 2-part object in schema indeed causes problems, at least in incremental models, instead specify catalog

sql-serverless:
  outputs:
    dev:
      host: ***.cloud.databricks.com
      http_path: /sql/1.0/endpoints/***
      catalog: hive_metastore
      schema: tube_silver_prod
      threads: 4
      token: ***
      type: databricks
  target: dev

Courses

Answer 2

Thanks Anton, I ended up resolving this. I created a temporary workspace to test it before applying to the main workspace. The catalog attribute can be applied almost anywhere you can specify the schema attribute, not just the profile.yml. I now have a dbt project which targets multiple catalog's. These are set in the dbt_project.yml at appropriate model level.

Whack

Hi Ashley, I want to use one unity catalog data source and build the table into two different unity catalog. But these unity catalogs are in prod and test workspace separately. How do you control the which unity catalog is used? – Johnajohnath