Skip to content

Conversation

@0xRobin
Copy link
Collaborator

@0xRobin 0xRobin commented Jul 6, 2023

  1. Copied generic spark tests to a _legacy version
  2. Changed all current test refs to use the _legacy version
  3. Included DuneSQL versions of the check_seed generic macro and tests
  4. added support for varbinary in seed columns for DuneSQL seeds

@0xRobin 0xRobin requested a review from jeff-dude July 6, 2023 08:58
@0xRobin 0xRobin assigned Hosuke and couralex6 and unassigned Hosuke and couralex6 Jul 6, 2023
@0xRobin 0xRobin requested review from Hosuke and couralex6 July 6, 2023 08:58
@0xRobin
Copy link
Collaborator Author

0xRobin commented Jul 6, 2023

After this is merged and you migrate a model that uses seed tests you should do the following steps to allow the seed test to run in DuneSQL.

  1. add a _schema.yml file for your seed, add a dunesql tag and type any varbinary and uint256 columns
version: 2

seeds:
  - name: your_seed
    config:
      tags: dunesql
      column_types:
        tx_hash: varbinary
        amount: uint256
  1. Check that any tests reference your new migrated model and not the _legacy version.
  2. if using generic tests, change to the non-legacy version in the model schema file.

@0xRobin 0xRobin added dune team created by dune team DuneSQL migration labels Jul 6, 2023
@0xRobin 0xRobin mentioned this pull request Jul 6, 2023
count(model_{{seed_matching_columns[0]}}) as `result_model`,
1 as `expected_seed`,
cast(count(model_{{seed_matching_columns[0]}}) as varchar) as result_model,
cast(1 as varchar) as expected_seed,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the casts here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In spark these columns had all kind of types (depending on the type of the column we're comparing), but for trino we need to unify them to 1 type (hence the varchar choice).

Note these casted columns are not used for the actual comparison, they are just included in the result table so you can easily see the cause of errors in the test results.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, makes sense. Could we add a comment on this? E.g., cast all columns to varchar to unify column types

@jkylling jkylling merged commit e7f0e60 into duneanalytics:main Jul 6, 2023
@aalan3 aalan3 mentioned this pull request Aug 23, 2023
olgafetisova pushed a commit to olgafetisova/spellbook that referenced this pull request Nov 14, 2023
* seed test support for dunesql

* forgot one _legacy

* add comment
@0xRobin 0xRobin deleted the migration-seed-support branch May 20, 2024 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dune team created by dune team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants