How to override global DBT macros in my dbt package that will be used by other projects

Question

I have a DBT package named dbt_helpers, where i intend to override some of dbt's in built global macros. In this example i intend to override the macro dbt_spark_validate_get_file_format, which is present in the dbt spark adapter here.

I have referred the dbt docs specified here to implement my use case. Here is how i have implemented the macro in my package under package's macros folder.

{% macro dbt_spark_validate_get_file_format(raw_file_format) -%}
    {{ return(adapter.dispatch('dbt_spark_validate_get_file_format','dbt_helpers')(raw_file_format)) }}
{%- endmacro %}


{% macro default__dbt_spark_validate_get_file_format(raw_file_format) %}
    {% do log('overriding global macro', info=true) %}
    {#  Custom implementation here  #}
    
{% endmacro %}

I have used the macro namespace dbt_helpers same as my package name. I have specified this in my main DBT project as a package in the packages.yml and I am able to see the macros defined in the dbt_packages directory after running the command dbt deps. In my main dbt project's dbt_project.yml I have included the project level dispatch config to take the macro from my package as shown, as directed in this section of the dbt docs.

dispatch:
  - macro_namespace: dbt
    search_order: ['dbt_helpers','dbt']

However when I run my dbt model the macro defined in my package is not being called, rather the inbuilt global macro is still being called. I am able to override the macro by placing it directly inside my projects macros folder, but i need to override the macro from my dbt_helpers package. How can i manage to do this?

Furrow

Answer 1

I can't see a macro in the spark adapter called containing the string validate_get_file_format.

If you are able to identify the exact name of the macro you are trying to override, you can just create a macro with the same name and put it in your package without worrying about dispatch configuration at all. You only need to worry about this if you're trying to override from another package, not a macro.

I believe this precedence applies for all macros and not just materializations.

Winter

This is correct for macros too. Just give it the same name in your package. – Bridoon