Skip to content

Randomize temp table __dbt_new_data name to allow concurrent dbt-clickhouse jobs #150

@winterrobert

Description

@winterrobert

Is your feature request related to a problem? Please describe.
We are running into an issue with the dbt-clickhouse adapter when using delete+insert. When running multiple dbt-clickhouse jobs at the same time, we often get a 500 complaining that the temporary xxx_dbt_new_data already exists:

09:36:29    :HTTPDriver for clickhouse:8123 returned response code 500)
09:36:29     Code: 57. DB::Exception: Table default.tablenamet__dbt_new_data already exists. (TABLE_ALREADY_EXISTS) (version 23.3.2.37 (official build))

My guess is that it's related to this row here

{% macro clickhouse__incremental_delete_insert(existing_relation, unique_key, incremental_predicates) %}
    {% set new_data_relation = existing_relation.incorporate(path={"identifier": model['name'] + '__dbt_new_data'}) %}
    {{ drop_relation_if_exists(new_data_relation) }}
....
{% endmacro %}

Describe the solution you'd like
Would it be a good PR to add a random suffix to the table so that the temp tables created by dbt-clickhouse doesn't collide with each other? Could somebody at the team fix this (seems like a quick fix) or would you accept a PR?

Describe alternatives you've considered
Change our pipelines to make sure we always just run one dbt-clickhouse job at a time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions