Skip to content

Conversation

@yuancu
Copy link
Collaborator

@yuancu yuancu commented Oct 16, 2025

Description

This PR fixes a bug in Calcite-enabled PPL queries where group-by fields cannot be aliased to their original field names, causing queries to fail with "field not found" errors.

When Calcite is enabled, PPL queries that use span functions with aliases matching the original field names fail with errors like: field [value] not found; input fields are: [value0, count()]

Affected Query Patterns:

  • source=time_test | stats count() by span(value, 2000) as value
  • source=time_test | stats count() by span(timestamp, 1h) as timestamp

Root Cause Analysis

The issue occurs during Calcite's aggregation processing:

  1. Input Dependencies: Calcite identifies that both the input field value and output alias value have the same name
  2. Name Collision Resolution: To avoid conflicts, Calcite automatically renames fields by adding numeric suffixes (value → value0)
  3. Reference Resolution Failure: Subsequent query processing still references the original name value, but Calcite has internally renamed it to value0, causing a "field not found" error

Solution Implementation

This PR implements a post-aggregation field renaming strategy that preserves intended aliases.

Related Issues

Resolves #4580

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@yuancu yuancu added the bug Something isn't working label Oct 16, 2025
Signed-off-by: Yuanchun Shen <[email protected]>
@yuancu yuancu changed the title WIP: Allow renaming to existing field names for group-by fields Allow renaming to existing field names for group-by fields Oct 17, 2025
@yuancu yuancu changed the title Allow renaming to existing field names for group-by fields Allow renaming group-by fields to existing field names Oct 17, 2025
@yuancu yuancu marked this pull request as ready for review October 17, 2025 02:43
Comment on lines +1038 to +1046
private boolean isInputRef(RexNode node) {
return switch (node.getKind()) {
case AS, DESCENDING, NULLS_FIRST, NULLS_LAST -> {
final List<RexNode> operands = ((RexCall) node).operands;
yield isInputRef(operands.getFirst());
}
default -> node instanceof RexInputRef;
};
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can the PlanUtil.getInputRefs be used to replace this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think they serve different purposes. PlanUtil.getInputRefs returns all referred input refs. Besides, if a node refers multiple inputs, it will return all of them. Yet here I just want to check whether a node is an input ref (optionally aliased), keeping the node as is.

// During aggregation, Calcite projects both input dependencies and output group-by fields.
// When names conflict, Calcite adds numeric suffixes (e.g., "value0").
// Apply explicit renaming to restore the intended aliases.
if (names.size() == reResolved.getLeft().size()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when the names.size not equals to reResolved.getLeft().size()? seems the condition is always true

Copy link
Collaborator Author

@yuancu yuancu Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lengths do not equal when a group key is not aliased -- under which circumstance extractAliasLiteral will return empty:

private Optional<RexLiteral> extractAliasLiteral(RexNode node) {
  if (node == null) {
    return Optional.empty();
  } else if (node.getKind() == AS) {
    return Optional.of((RexLiteral) ((RexCall) node).getOperands().get(1));
  } else {
    return Optional.empty();
  }

Although it seems that all group keys are aliased in practice, this defense check was to prevent unintended future changes to avoid in-correspondent renaming. Should I remove it?

Pair<List<RexNode>, List<AggCall>> reResolved =
resolveAttributesForAggregation(groupExprList, aggExprList, context);

List<String> names = getGroupKeyNamesAfterAggregation(reResolved.getLeft());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you rename the var names to make it more meaningful

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed

Signed-off-by: Yuanchun Shen <[email protected]>
qianheng-aws
qianheng-aws previously approved these changes Oct 20, 2025
* Imitates {@code Registrar.registerExpression} of {@link RelBuilder} to derive the output order
* of group-by keys after aggregation.
*
* <p>The projected input reference comes first, while any other computed expression follows.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Registrar.registerExpression, seems the other computed expression won't promise following the original order if there is expression duplication.

But since our PPL only allow span expr in our group by and it cannot be combined with other span expr. This logic may be right and I cannot find any bad case so far.

Copy link
Collaborator Author

@yuancu yuancu Oct 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a bad case: stats count() by value, value, @timestamp. I'll fix it.

Update: Fixed by checking duplication

/** Whether a rex node is an aliased input reference */
private boolean isInputRef(RexNode node) {
return switch (node.getKind()) {
case AS, DESCENDING, NULLS_FIRST, NULLS_LAST -> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any case that we have DESCENDING, NULLS_FIRST, NULLS_LAST in our stats .. by ... command

Copy link
Collaborator Author

@yuancu yuancu Oct 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I didn't manage to create any. It seems there is always a projection after sorting and before aggregation.

E.g.

LogicalAggregate(group=[{0}], count()=[COUNT()])
  LogicalProject(value=[$2])
    LogicalSort(sort0=[$2], dir0=[DESC-nulls-last])

@qianheng-aws qianheng-aws merged commit a86a5a7 into opensearch-project:main Oct 24, 2025
33 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 24, 2025
* Rename fields to intended ones after aggregation

Signed-off-by: Yuanchun Shen <[email protected]>

* Add a defense check

Signed-off-by: Yuanchun Shen <[email protected]>

* Remove defense check

Signed-off-by: Yuanchun Shen <[email protected]>

* Handle cases where there exist duplicated group keys

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
(cherry picked from commit a86a5a7)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
LantaoJin pushed a commit that referenced this pull request Oct 28, 2025
…names (#4653)

* Allow renaming group-by fields to existing field names (#4586)

* Rename fields to intended ones after aggregation

Signed-off-by: Yuanchun Shen <[email protected]>

* Add a defense check

Signed-off-by: Yuanchun Shen <[email protected]>

* Remove defense check

Signed-off-by: Yuanchun Shen <[email protected]>

* Handle cases where there exist duplicated group keys

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
(cherry picked from commit a86a5a7)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Downgrade language level to java 11

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Yuanchun Shen <[email protected]>
asifabashar added a commit to asifabashar/sql that referenced this pull request Oct 28, 2025
* default-main: (34 commits)
  Enhance dynamic source clause to support only metadata filters (opensearch-project#4554)
  Make nested alias type support referring to outer context (opensearch-project#4673)
  Update big5 ppl queries and check plans (opensearch-project#4668)
  Support push down sort after limit (opensearch-project#4657)
  Use table scan rowType in filter pushdown could fix rename issue (opensearch-project#4670)
  Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (opensearch-project#4621)
  Fix bin nested fields issue (opensearch-project#4606)
  Add `per_minute`, `per_hour`, `per_day` function support (opensearch-project#4531)
  Pushdown sort aggregate metrics (opensearch-project#4603)
  Followup: Change ComparableLinkedHashMap to compare Key than Value (opensearch-project#4648)
  Mitigate the CI failure caused by 500 Internal Server Error (opensearch-project#4646)
  Allow renaming group-by fields to existing field names (opensearch-project#4586)
  Publish internal modules separately for downstream reuse (opensearch-project#4484)
  Revert "Update grammar files and developer guide (opensearch-project#4301)" (opensearch-project#4643)
  Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (opensearch-project#4599)
  Replace all dots in fields of table scan's PhysType (opensearch-project#4633)
  Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue (opensearch-project#4629)
  Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT (opensearch-project#4623)
  Pushdown case function in aggregations as range queries (opensearch-project#4400)
  Update GEOIP function to support IP types as input (opensearch-project#4613)
  ...

# Conflicts:
#	docs/user/ppl/functions/conversion.rst
asifabashar added a commit to asifabashar/sql that referenced this pull request Oct 28, 2025
* default-main: (34 commits)
  Enhance dynamic source clause to support only metadata filters (opensearch-project#4554)
  Make nested alias type support referring to outer context (opensearch-project#4673)
  Update big5 ppl queries and check plans (opensearch-project#4668)
  Support push down sort after limit (opensearch-project#4657)
  Use table scan rowType in filter pushdown could fix rename issue (opensearch-project#4670)
  Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (opensearch-project#4621)
  Fix bin nested fields issue (opensearch-project#4606)
  Add `per_minute`, `per_hour`, `per_day` function support (opensearch-project#4531)
  Pushdown sort aggregate metrics (opensearch-project#4603)
  Followup: Change ComparableLinkedHashMap to compare Key than Value (opensearch-project#4648)
  Mitigate the CI failure caused by 500 Internal Server Error (opensearch-project#4646)
  Allow renaming group-by fields to existing field names (opensearch-project#4586)
  Publish internal modules separately for downstream reuse (opensearch-project#4484)
  Revert "Update grammar files and developer guide (opensearch-project#4301)" (opensearch-project#4643)
  Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (opensearch-project#4599)
  Replace all dots in fields of table scan's PhysType (opensearch-project#4633)
  Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue (opensearch-project#4629)
  Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT (opensearch-project#4623)
  Pushdown case function in aggregations as range queries (opensearch-project#4400)
  Update GEOIP function to support IP types as input (opensearch-project#4613)
  ...

Signed-off-by: Asif Bashar <[email protected]>
@yuancu yuancu deleted the issues/4580 branch November 4, 2025 02:15
expani pushed a commit to vinaykpud/sql that referenced this pull request Nov 4, 2025
…oject#4586)

* Rename fields to intended ones after aggregation

Signed-off-by: Yuanchun Shen <[email protected]>

* Add a defense check

Signed-off-by: Yuanchun Shen <[email protected]>

* Remove defense check

Signed-off-by: Yuanchun Shen <[email protected]>

* Handle cases where there exist duplicated group keys

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.19-dev bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Group-by fields can not be aliased to the original field names with Calcite

3 participants