Skip to content

Deprecate ScalarUDFImpl::return_type #13717

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

Deprecate ScalarUDFImpl::return_type #13717

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e17342c
Deprecate ScalarUDFImpl::return_type and revert a breaking API change
andygrove Dec 10, 2024
b64cf14
clippy
andygrove Dec 10, 2024
97b037b
revert change
andygrove Dec 11, 2024
4e08770
address feedback
andygrove Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions datafusion/expr/src/udf.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ impl ScalarUDF {
///
/// See [`ScalarUDFImpl::return_type`] for more details.
pub fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
#[allow(deprecated)]
Copy link
Member

@findepi findepi Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we also deprecate this (ScalarUDF::return_type) method if we deprecate underlying ScalarUDFImpl method?

Copy link
Contributor

@alamb alamb Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

self.inner.return_type(arg_types)
}

Expand Down Expand Up @@ -450,6 +451,7 @@ pub trait ScalarUDFImpl: Debug + Send + Sync {
/// is recommended to return [`DataFusionError::Internal`].
///
/// [`DataFusionError::Internal`]: datafusion_common::DataFusionError::Internal
#[deprecated(since = "44.0.0", note = "Use `return_type_from_exprs` instead")]
Copy link
Contributor

@alamb alamb Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we are going to deprecate this API I think we should add an example to return_type_from_exprs to help the migration effort.

I can help if we proceed

fn return_type(&self, arg_types: &[DataType]) -> Result<DataType>;

/// What [`DataType`] will be returned by this function, given the
Expand Down Expand Up @@ -483,6 +485,7 @@ pub trait ScalarUDFImpl: Debug + Send + Sync {
_schema: &dyn ExprSchema,
arg_types: &[DataType],
) -> Result<DataType> {
#[allow(deprecated)]
self.return_type(arg_types)
}

Expand Down Expand Up @@ -756,6 +759,7 @@ impl ScalarUDFImpl for AliasedScalarUDFImpl {
}

fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
#[allow(deprecated)]
self.inner.return_type(arg_types)
}

Expand Down
4 changes: 3 additions & 1 deletion datafusion/functions/src/datetime/date_part.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,9 @@ impl ScalarUDFImpl for DatePartFunc {
}

fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
internal_err!("return_type_from_exprs shoud be called instead")
// return type could be Float32 or Int32 depending on the input arguments
// return_type_from_exprs should be called instead
Ok(DataType::Float64)
Copy link
Member

@findepi findepi Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would appreciate exception more than successfully retrieving a false value.
the false value may easily cause some hard to track down problem, while exception is so much easier to find

Copy link
Member Author

@andygrove andygrove Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reverted this change now.

Copy link
Member Author

@andygrove andygrove Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, this is a breaking change and is currently blocking Comet from upgrading its DataFusion version. We do not have access to the logical Expr to call the newer return_type_from_exprs because we are only working with DataFusion's physical plans. I think I would prefer the non-breaking behavior of maintaining the functionality from DataFusion 43 and marking this as deprecated until we can find a better solution.

Maybe @alamb has thoughts on how we can best handle this to avoid causing pain for downstream projects?

Copy link
Member

@findepi findepi Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a breaking change and is currently blocking Comet from upgrading its DataFusion version.

i agree this is breaking and sorry to hear it affects Comet
i wish i knew how to make it non-breaking!

it seems this all boils down to the problem that expressions don't know their types -- #12604

}

fn return_type_from_exprs(
Expand Down
1 change: 1 addition & 0 deletions datafusion/functions/src/string/concat.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,7 @@ pub fn simplify_concat(args: Vec<Expr>) -> Result<ExprSimplifyResult> {
_ => None,
})
.collect();
#[allow(deprecated)]
ConcatFunc::new().return_type(&data_types)
}?;

Expand Down
Loading