[logical-types] update working branch #12812
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Make support schemas * Set default name to table * Remove print statements and stale comment * Add tests for create table * Fix typo * Update datafusion/sql/src/statement.rs Co-authored-by: Jonah Gao <[email protected]> * convert create_external_table to objectname * Add sqllogic tests * Fix failing tests --------- Co-authored-by: Jonah Gao <[email protected]>
* Fix Regex signature types * Uncomment the shared tests in string_query.slt.part and removed tests copies everywhere else * Test `LIKE` and `MATCH` with flags; Remove new tests from regexp.slt
Remove conditions where unnecessary. Refactor to improve readability.
* Remove aggregate functions dependency on frontend DataFusion is a SQL query engine and also a reusable library for building query engines. The core functionality should not depend on frontend related functionalities like `sqlparser` or `datafusion-sql`. * Remove duplicate license header
* rm clone Signed-off-by: jayzhan211 <[email protected]> * fmt Signed-off-by: jayzhan211 <[email protected]> --------- Signed-off-by: jayzhan211 <[email protected]>
… constants (apache#12702) * Refactor tests for union sorting properties * update doc test * Undo import reordering * remove unecessary static lifetimes
…nerics (apache#12703) * Reduce code duplication in `PrimitiveGroupValueBuilder` with const generics * Fix docs
* Disallow duplicated qualified field names * Fix tests
…ster) (apache#12675) * add bench * replace macro with generic function * remove duplicated code * optimize base64/hex decode
…odes (apache#12685) * Update trait `UserDefinedLogicalNodeCore` Signed-off-by: Austin Liu <[email protected]> * Update corresponding interface Signed-off-by: Austin Liu <[email protected]> Add rewrite rule for `push-down-limit` for `Extension` Signed-off-by: Austin Liu <[email protected]> * Add rewrite rule for `push-down-limit` for `Extension` and tests Signed-off-by: Austin Liu <[email protected]> * Update corresponding interface Signed-off-by: Austin Liu <[email protected]> * Reorganize to match guard Signed-off-by: Austin Liu <[email protected]> * Clena up Signed-off-by: Austin Liu <[email protected]> Clean up Signed-off-by: Austin Liu <[email protected]> --------- Signed-off-by: Austin Liu <[email protected]>
…12686) * Add union resolving for nested struct arrays * Add test * Change test * Reproduce index error * fmt --------- Co-authored-by: Andrew Lamb <[email protected]>
…pache#12693) * Adds macro for udwf singleton * Adds a doc comment parameter to macro * Add doc comment for `create_udwf` macro * Uses default constructor * Update `Cargo.lock` in `datafusion-cli` * Fixes: expand `$FN_NAME` in doc strings * Adds example for macro usage * Renames macro * Improve doc comments * Rename udwf macro * Minor: doc copy edits * Adds macro for creating fluent-style expression API * Adds support for 1 or more parameters in expression function * Rewrite doc comments * Rename parameters * Minor: formatting * Adds doc comment for `create_udwf_expr` macro * Improve example docs * Hides extraneous code in doc comments * Add a one-line readme * Adds doc test assertions + minor formatting fixes * Adds common macro for defining user-defined window functions * Adds doc comment for `define_udwf_and_expr` * Defines `RowNumber` using common macro * Add usage example for common macro * Adds usage for custom constructor * Add examples for remaining patterns * Improve doc comments for usage examples * Rewrite inner line docs * Rewrite `create_udwf_expr!` doc comments * Minor doc improvements * Fix doc test and usage example * Add inline comments for macro patterns * Minor: change doc comment in example
…pache#12705) * Support unparsing plans with both Aggregation and Window functions (apache#35) * Fix unparsing for aggregation grouping sets * Add test for grouping set unparsing * Update datafusion/sql/src/unparser/utils.rs Co-authored-by: Jax Liu <[email protected]> * Update datafusion/sql/src/unparser/utils.rs Co-authored-by: Jax Liu <[email protected]> * Update * More tests --------- Co-authored-by: Jax Liu <[email protected]>
In 1b3608d `strpos` signature was modified to indicate it supports dictionary as input argument, but the invoke method doesn't support them.
…provide an "out of the box" query engine (apache#12666) * Update DataFusion introduction to show that DataFusion offers packaged versions for end users * change order * Update README.md Co-authored-by: Andrew Lamb <[email protected]> * refine wording and update user guide for consistency * prettier --------- Co-authored-by: Andrew Lamb <[email protected]>
…on (apache#12668) * Initial work on apache#12432 to allow for generation of udf docs from embedded documentation in the code * Add missing license header. * Fixed examples. * Fixing a really weird RustRover/wsl ... something. No clue what happened there. * permission change * Cargo fmt update. * Refactored Documentation to allow it to be used in a const. * Add documentation for syntax_example * Refactoring Documentation based on PR feedback. * Cargo fmt update. * Doc update * Fixed copy/paste error. * Minor text updates. --------- Co-authored-by: Andrew Lamb <[email protected]>
* imdb dataset * cargo fmt * Add 113 queries for IMDB(JOB) Signed-off-by: Austin Liu <[email protected]> * Add `get_query_sql` from `query_id` string Signed-off-by: Austin Liu <[email protected]> * Fix CSV reader & Remove Parquet partition Signed-off-by: Austin Liu <[email protected]> * Add benchmark IMDB runner Signed-off-by: Austin Liu <[email protected]> * Add `run_imdb` script Signed-off-by: Austin Liu <[email protected]> * Add checker for imdb option Signed-off-by: Austin Liu <[email protected]> * Add SLT for IMDB Signed-off-by: Austin Liu <[email protected]> * Fix `get_query_sql()` for CI roundtrip test Signed-off-by: Austin Liu <[email protected]> Fix `get_query_sql()` for CI roundtrip test Signed-off-by: Austin Liu <[email protected]> Fix `get_query_sql()` for CI roundtrip test Signed-off-by: Austin Liu <[email protected]> * Clean up Signed-off-by: Austin Liu <[email protected]> * Add missing license Signed-off-by: Austin Liu <[email protected]> * Add IMDB(JOB) queries `2b` to `5c` Signed-off-by: Austin Liu <[email protected]> * Add `INCLUDE_IMDB` in CI verify-benchmark-results Signed-off-by: Austin Liu <[email protected]> * Prepare IMDB dataset Signed-off-by: Austin Liu <[email protected]> Prepare IMDB dataset Signed-off-by: Austin Liu <[email protected]> * use uint as id type * format * Seperate `tpch` and `imdb` benchmarking CI jobs Signed-off-by: Austin Liu <[email protected]> Fix path Signed-off-by: Austin Liu <[email protected]> Fix path Signed-off-by: Austin Liu <[email protected]> Remove `tpch` in `imdb` benchmark Signed-off-by: Austin Liu <[email protected]> * Remove IMDB(JOB) slt in CI Signed-off-by: Austin Liu <[email protected]> Remove IMDB(JOB) slt in CI Signed-off-by: Austin Liu <[email protected]> --------- Signed-off-by: Austin Liu <[email protected]> Co-authored-by: DouPache <[email protected]>
…ache#12722) * Minor: avoid clone while calculating union equivalence properties * Update datafusion/physical-expr/src/equivalence/properties.rs * fmt
* simplify streaming_merge function parameters * revert test change * change StreamingMergeConfig into builder pattern
…ith null fields. (apache#12729) * test: reproducer for missing schema metadata on cross join * fix: pass thru schema metadata on cross join * fix: preserve metadata when transforming to view types * test: reproducer for missing field metadata in left hand NULL field of union * fix: preserve field metadata from right side of union * chore: safe indexing
* cleanup make array coercion rule Signed-off-by: jayzhan211 <[email protected]> * change to type union resolution Signed-off-by: jayzhan211 <[email protected]> * change value too Signed-off-by: jayzhan211 <[email protected]> * fix tpyo Signed-off-by: jayzhan211 <[email protected]> --------- Signed-off-by: jayzhan211 <[email protected]>
…te (apache#12747) * Add `DocumentationBuilder::with_standard_expression` to reduce copy/paste * fix doc * fix standard argument * Update docs * Improve documentation to explain what is different
* fix `equal_to` in `PrimitiveGroupValueBuilder`. * fix typo. * add uts. * reduce calling of `is_null`.
* Fix stack overflow calculating projected orderings * fix docs
* Update to arrow/parquet 53.1.0 * Update some API * update for changed file sizes * Use non deprecated APIs * Use ParquetMetadataReader from @etseidl * remove upstreamed implementation * Update CSV schema * Use upstream is_null and is_not_null kernels
* Add support for serializing and deserializing Substrait ExtendedExpr message * Address clippy reviews * Reuse existing rename method
) Co-authored-by: Andrew Lamb <[email protected]>
…ache#12571) * Fix grouping sets behavior when data contains nulls * PR suggestion comment * Update new test case * Add grouping_id to the logical plan * Add doc comment next to INTERNAL_GROUPING_ID * Fix unparsing of Aggregate with grouping sets --------- Co-authored-by: Andrew Lamb <[email protected]>
….md to code (apache#12775) * Added documentation for string and unicode functions. * Fixed issues with aliases. * Cargo fmt. * Minor doc fixes. * Update docs for var_pop/samp --------- Co-authored-by: Andrew Lamb <[email protected]>
…integer (apache#12751) * fix sig Signed-off-by: jayzhan211 <[email protected]> * fix Signed-off-by: jayzhan211 <[email protected]> * fix error Signed-off-by: jayzhan211 <[email protected]> * fix all signature Signed-off-by: jayzhan211 <[email protected]> * fix all signature Signed-off-by: jayzhan211 <[email protected]> * change default type Signed-off-by: jayzhan211 <[email protected]> * clippy Signed-off-by: jayzhan211 <[email protected]> * fix docs Signed-off-by: jayzhan211 <[email protected]> * rm deadcode Signed-off-by: jayzhan211 <[email protected]> * cleanup Signed-off-by: jayzhan211 <[email protected]> * cleanup Signed-off-by: jayzhan211 <[email protected]> * rm test Signed-off-by: jayzhan211 <[email protected]> --------- Signed-off-by: jayzhan211 <[email protected]>
github-actions
bot
added
documentation
alamb
approved these changes
Oct 8, 2024
Contributor
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Looks good to me @notfilippo -- thanks.
The CI appears to be failing on the logical-types branch -- perhaps you can make a follow on PR to fix that?
Contributor
|
Merging as this is to a feature branch |
Contributor
Author
Filed this PR: #12820 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
22 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update the
logical-typesbranch to reflect changes in main (especially the CI fix) cc @jayzhan211 @alamb