Publish internal modules separately for downstream reuse #4484

Swiddis · 2025-10-09T19:17:37Z

Description

Copy of #4385 but merged with main, trying to see what's broken with tests

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…project#3763) * Add common Gradle task for all published modules Signed-off-by: Chen Dai <[email protected]> * Add publish workflow file Signed-off-by: Chen Dai <[email protected]> * Rename group ID from sql to query Signed-off-by: Chen Dai <[email protected]> --------- Signed-off-by: Chen Dai <[email protected]>

* Add api module with API and UT Signed-off-by: Chen Dai <[email protected]> * Refactor catalog API and clean up build.gradle Signed-off-by: Chen Dai <[email protected]> * Add cache schema API and refactor UT Signed-off-by: Chen Dai <[email protected]> * Add readme Signed-off-by: Chen Dai <[email protected]> * Add comment for hardcoding query size limit Signed-off-by: Chen Dai <[email protected]> * Add default namespace API with more UTs Signed-off-by: Chen Dai <[email protected]> --------- Signed-off-by: Chen Dai <[email protected]>

…-project#4015) * add datasources submodule Signed-off-by: cnoramut <[email protected]> * add legacy since datasources need it Signed-off-by: cnoramut <[email protected]> --------- Signed-off-by: cnoramut <[email protected]> Signed-off-by: Louis Chu <[email protected]> Co-authored-by: Louis Chu <[email protected]>

Signed-off-by: Simeon Widdis <[email protected]>

Swiddis · 2025-10-09T20:18:13Z

api/src/main/java/org/opensearch/sql/api/EmptyDataSourceService.java

+/**
+ * A DataSourceService that assumes no access to data sources
+ */
+public class EmptyDataSourceService implements DataSourceService {


Since CalciteRelNodeVisitor needs a data source service now, but this notion doesn't make any sense outside of a cluster, let's just fill in an empty one that always returns no results and reports any given source as nonexistent.

Better solution long-term would be to make CalciteRelNodeVisitor not have this dependency to begin with, seems like it crosses boundaries.

Signed-off-by: Simeon Widdis <[email protected]>

dai-chen · 2025-10-22T18:23:54Z

.github/workflows/maven-publish-modules.yml

+      - 2.*
+
+env:
+  SNAPSHOT_REPO_URL: https://aws.oss.sonatype.org/content/repositories/snapshots/


Is this already deprecated? If so, probably you need a PR for previous feature branch to verify the publish?

agh, sonatype is the gift that keeps on giving...

Interesting that tests pass when this shouldn't be able to build

Signed-off-by: Simeon Widdis <[email protected]>

.github/workflows/maven-publish-modules.yml

Swiddis · 2025-10-23T16:10:08Z

~~Don't merge -- trying to test the publish flow on the feature branch currently, it says "success" but no artifact shows up in central~~

Fixed, successful publish: https://github.com/opensearch-project/sql/actions/runs/18763687236/job/53534148205

Signed-off-by: Simeon Widdis <[email protected]>

Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Chen Dai <[email protected]> Co-authored-by: Mebsina <[email protected]> (cherry picked from commit 05d6594) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* default-main: (34 commits) Enhance dynamic source clause to support only metadata filters (opensearch-project#4554) Make nested alias type support referring to outer context (opensearch-project#4673) Update big5 ppl queries and check plans (opensearch-project#4668) Support push down sort after limit (opensearch-project#4657) Use table scan rowType in filter pushdown could fix rename issue (opensearch-project#4670) Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (opensearch-project#4621) Fix bin nested fields issue (opensearch-project#4606) Add `per_minute`, `per_hour`, `per_day` function support (opensearch-project#4531) Pushdown sort aggregate metrics (opensearch-project#4603) Followup: Change ComparableLinkedHashMap to compare Key than Value (opensearch-project#4648) Mitigate the CI failure caused by 500 Internal Server Error (opensearch-project#4646) Allow renaming group-by fields to existing field names (opensearch-project#4586) Publish internal modules separately for downstream reuse (opensearch-project#4484) Revert "Update grammar files and developer guide (opensearch-project#4301)" (opensearch-project#4643) Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (opensearch-project#4599) Replace all dots in fields of table scan's PhysType (opensearch-project#4633) Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue (opensearch-project#4629) Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT (opensearch-project#4623) Pushdown case function in aggregations as range queries (opensearch-project#4400) Update GEOIP function to support IP types as input (opensearch-project#4613) ... # Conflicts: # docs/user/ppl/functions/conversion.rst

* default-main: (34 commits) Enhance dynamic source clause to support only metadata filters (opensearch-project#4554) Make nested alias type support referring to outer context (opensearch-project#4673) Update big5 ppl queries and check plans (opensearch-project#4668) Support push down sort after limit (opensearch-project#4657) Use table scan rowType in filter pushdown could fix rename issue (opensearch-project#4670) Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (opensearch-project#4621) Fix bin nested fields issue (opensearch-project#4606) Add `per_minute`, `per_hour`, `per_day` function support (opensearch-project#4531) Pushdown sort aggregate metrics (opensearch-project#4603) Followup: Change ComparableLinkedHashMap to compare Key than Value (opensearch-project#4648) Mitigate the CI failure caused by 500 Internal Server Error (opensearch-project#4646) Allow renaming group-by fields to existing field names (opensearch-project#4586) Publish internal modules separately for downstream reuse (opensearch-project#4484) Revert "Update grammar files and developer guide (opensearch-project#4301)" (opensearch-project#4643) Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (opensearch-project#4599) Replace all dots in fields of table scan's PhysType (opensearch-project#4633) Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue (opensearch-project#4629) Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT (opensearch-project#4623) Pushdown case function in aggregations as range queries (opensearch-project#4400) Update GEOIP function to support IP types as input (opensearch-project#4613) ... Signed-off-by: Asif Bashar <[email protected]>

Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Chen Dai <[email protected]> Co-authored-by: Mebsina <[email protected]>

…project#4484) Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Chen Dai <[email protected]> Co-authored-by: Mebsina <[email protected]>

…project#4484) Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Chen Dai <[email protected]> Co-authored-by: Mebsina <[email protected]> Signed-off-by: Chen Dai <[email protected]>

…m reuse (#4723) * Publish internal modules separately for downstream reuse (#4484) Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Chen Dai <[email protected]> Co-authored-by: Mebsina <[email protected]> Signed-off-by: Chen Dai <[email protected]> * Add test branch and downgrade JDK in maven publish workflow Signed-off-by: Chen Dai <[email protected]> * Add 2.19-dev branch to workflow and bump Calcite version in api module Signed-off-by: Chen Dai <[email protected]> --------- Signed-off-by: Chen Dai <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Mebsina <[email protected]>

…project#4484) Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Chen Dai <[email protected]> Co-authored-by: Mebsina <[email protected]>

dai-chen and others added 5 commits June 16, 2025 08:56

add datasources submodule (opensearch-project#3935)

99724d9

Merge branch 'main' into feature/unified-ppl

ec5e8fe

Swiddis added the enhancement New feature or request label Oct 9, 2025

Swiddis changed the title ~~Unified PPL as a library~~ Publish internal modules separately for downstream reuse Oct 9, 2025

Swiddis added 4 commits October 9, 2025 12:37

Unlock Spotless version

87396b7

Signed-off-by: Simeon Widdis <[email protected]>

Don't run publish on feature fork

99f181b

Signed-off-by: Simeon Widdis <[email protected]>

Merge remote-tracking branch 'upstream/main' into feature/unified-ppl

8c428c6

Fill in empty data source service for external API

f6af494

Signed-off-by: Simeon Widdis <[email protected]>

Swiddis commented Oct 9, 2025

View reviewed changes

Run spotless

7dc10bf

Signed-off-by: Simeon Widdis <[email protected]>

Swiddis marked this pull request as ready for review October 9, 2025 20:54

Swiddis requested review from GumpacG, MaxKsyunz, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, penghuo, ps48, seankao-az and vamsimanohar as code owners October 9, 2025 20:54

Swiddis requested review from LantaoJin, RyanL1997, noCharger, qianheng-aws and yuancu as code owners October 9, 2025 20:54

dai-chen reviewed Oct 22, 2025

View reviewed changes

Swiddis added 4 commits October 22, 2025 18:53

Switch sonatype repo

2bb0223

Signed-off-by: Simeon Widdis <[email protected]>

Fix url again (weird it didn't push the first time)

42b9dd5

Signed-off-by: Simeon Widdis <[email protected]>

Merge remote-tracking branch 'upstream/main' into HEAD

6195a56

Deal with upstream breaking stuff again

05dd1a4

Signed-off-by: Simeon Widdis <[email protected]>

dai-chen previously approved these changes Oct 23, 2025

View reviewed changes

.github/workflows/maven-publish-modules.yml Outdated Show resolved Hide resolved

Attempt to publish to S3 again

eed569d

Signed-off-by: Simeon Widdis <[email protected]>

Swiddis dismissed dai-chen’s stale review via eed569d October 23, 2025 22:52

Tweak publish branch rules

98e16bc

Signed-off-by: Simeon Widdis <[email protected]>

dai-chen approved these changes Oct 23, 2025

View reviewed changes

RyanL1997 approved these changes Oct 23, 2025

View reviewed changes

Swiddis merged commit 05d6594 into opensearch-project:main Oct 23, 2025
33 checks passed

Swiddis added the backport 3.3 label Oct 24, 2025

opensearch-trigger-bot bot mentioned this pull request Oct 24, 2025

[Backport 3.3] Publish internal modules separately for downstream reuse #4661

Open

dai-chen added a commit that referenced this pull request Oct 30, 2025

Publish internal modules separately for downstream reuse (#4484)

4314feb

Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Chen Dai <[email protected]> Co-authored-by: Mebsina <[email protected]>

dai-chen mentioned this pull request Nov 3, 2025

[Backport 2.19-dev] Publish internal modules separately for downstream reuse #4723

Merged

dai-chen mentioned this pull request Nov 4, 2025

[FEATURE] Export PPL-Calcite engine as reusable library #3734

Open

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Publish internal modules separately for downstream reuse #4484

Publish internal modules separately for downstream reuse #4484

Uh oh!

Swiddis commented Oct 9, 2025

Uh oh!

Swiddis Oct 9, 2025 •

edited

Loading

Uh oh!

dai-chen Oct 22, 2025

Uh oh!

Swiddis Oct 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

Swiddis commented Oct 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Publish internal modules separately for downstream reuse #4484

Publish internal modules separately for downstream reuse #4484

Uh oh!

Conversation

Swiddis commented Oct 9, 2025

Description

Related Issues

Check List

Uh oh!

Swiddis Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dai-chen Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Swiddis Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Swiddis commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Swiddis Oct 9, 2025 •

edited

Loading

Swiddis Oct 22, 2025 •

edited

Loading

Swiddis commented Oct 23, 2025 •

edited

Loading