Bump DataFusion to 50 and arrow to 56 #4577
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AdamGS
force-pushed
the
adamg/patch-datafusion
branch
from
1f1416a to
018f180
Compare
45 tasks
CodSpeed Performance ReportMerging #4577 will improve performances by 41.79%Comparing Summary
Benchmarks breakdown
|
AdamGS
commented
Sep 9, 2025
Codecov Report❌ Patch coverage is ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
AdamGS
changed the title
AdamGS
changed the title
Contributor
Benchmarks: TPC-H SF=1 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=1 on S3Summary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=10 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: Statistical and Population GeneticsSummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=10 on S3Summary
Detailed Results Table
|
Contributor
Benchmarks: Clickbench on NVMESummary
Detailed Results Table
|
0ax1
reviewed
Sep 9, 2025
| to_arrow_primitive::<Float64Type>(array) | ||
| } | ||
| (Canonical::Decimal(array), DataType::Decimal32(precision, scale)) => { | ||
| if array.decimal_dtype().precision() != *precision |
Contributor
There was a problem hiding this comment.
Is this check too restrictive? Why not check whether the target type can hold at least the same precision and scale? Maybe I'm misreading something here.
array.decimal_dtype().precision() <= *precision
array.decimal_dtype().scale() <= *scale
Contributor
There was a problem hiding this comment.
As in bail if array.decimal_dtype().precision() > *precision.
Contributor
Author
There was a problem hiding this comment.
I think I agree, I'll think about it a bit more but probably include it
Contributor
Author
There was a problem hiding this comment.
After spending some more time with with arrow's and datafusion's decimal-handling code, I think we shouldn't change precision and scale here. Keeping it this way guarantees predictable behavior and zero-copy, unless explicitly cast.
Contributor
Author
|
Some work here depends on the outcome of apache/datafusion#17489, if DataFusion's support for smaller decimals doesn't improve, I'll have to rethink some stuff here. |
xudong963
reviewed
Sep 15, 2025
| } | ||
|
|
||
| pub(crate) fn can_be_pushed_down(expr: &PhysicalExprRef, schema: &Schema) -> bool { | ||
| // We currently do not support pushdown of dynamic expressions in DF |
There was a problem hiding this comment.
I think the dynamic filter will be gradually stable in the next few versions. How about recording an issue for this? There may be potential contributors to participate
Contributor
Author
There was a problem hiding this comment.
I guess I should link to the issue in the code, I've also expanded there a bit about my current understanding of the issue, which might not be correct but probably worth externalizing.
AdamGS
force-pushed
the
adamg/patch-datafusion
branch
from
e407032 to
0a99805
Compare
Signed-off-by: Adam Gutglick <[email protected]>
Signed-off-by: Adam Gutglick <[email protected]>
Signed-off-by: Adam Gutglick <[email protected]>
Signed-off-by: Adam Gutglick <[email protected]>
AdamGS
force-pushed
the
adamg/patch-datafusion
branch
from
0a99805 to
88eaa65
Compare
AdamGS
added
benchmark-sql
and removed
do not merge
robert3005
approved these changes
Sep 16, 2025
Contributor
Benchmarks: TPC-DS SF=1 on NVMESummary
Detailed Results Table
|
zhuqi-lucas
pushed a commit
to massive-com/vortex
that referenced
this pull request
Sep 23, 2025
Just made sure there's no breaking changes coming our way. Includes support for the two new Arrow decimal types (32 and 64 bits), but without converting from Vortex dtype into them. Codspeed improvements are due to a new inline hint on arrow's `BitIterator::next` (found by @0ax1 🥳). Open issues: - [ ] apache/datafusion#17489 - mostly affects if/how much support we want to provide to decimals, or whether we need to provide some other solution for DataFusion. - [ ] Issue with followups - vortex-data#4668 --------- Signed-off-by: Adam Gutglick <[email protected]> (cherry picked from commit cc426c8)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Just made sure there's no breaking changes coming our way. Includes support for the two new Arrow decimal types (32 and 64 bits), but without converting from Vortex dtype into them.
Codspeed improvements are due to a new inline hint on arrow's
BitIterator::next(found by @0ax1 🥳).Open issues: