Datafusion Query Planning tests in CI

[edmondop]

Quoting @andygrove on Discord:

we have rust tests for query planning, but they require some tpc-h data to be available
ideally we should be running the tpc-h queries against a small data set (sf=1) in CI and checking that the results are correct
I expect we can re-use some of the CI setup from core DataFusion

[edmondop]

At the moment, we already run some query planning tests, see https://github.com/edmondop/datafusion-ray/actions/runs/11133252895/job/30938921925

[andygrove]

At the moment, we already run some query planning tests, see https://github.com/edmondop/datafusion-ray/actions/runs/11133252895/job/30938921925

These tests do not actually do anything currently, unless they run on my computer 😞

async fn do_test(n: u8) -> Result<()> {
    let data_path = "/mnt/bigdata/tpch/sf10-parquet";
    if !Path::new(&data_path).exists() {
        return Ok(());
    }

[andygrove]

We have a Python script at https://github.com/apache/datafusion-benchmarks/tree/main/tpch for generating TPC-H data:

https://github.com/apache/datafusion-benchmarks/tree/main/tpch

We could call this in CI and then figure out how to update theses tests to use that data

[edmondop]

Excellent, I'll do this as a first iteration, thank for the hint. Then we can decide how to do the comparison between datafusion results and ray results, what do you think?

[andygrove]

Excellent, I'll do this as a first iteration, thank for the hint. Then we can decide how to do the comparison between datafusion results and ray results, what do you think?

Sounds good.

We could run each query first with DF Python and then with DF Ray and compare the results. This could be a good general approach to testing.

[edmondop]

Closed with #12 #16 #17