Add the base FlightProducer for getStream API #17446

rishabhmaurya · 2025-02-24T23:36:59Z

Description

With recently added Arrow SPIs #16691 and flight server support #16962, this is a follow up change which introduces -

implementation of getStream() and getFlightInfo() APIs. It also adds support for default StreamManager, FlightProducer, FlightStreamReader which these APIs make use of.
adds way for consumers of StreamManagerPlugin to get instance of default StreamManager.
adds ProxyStreamProvider which acts as forward proxy for FlightStream. This is useful when stream is not present locally and needs to be fetched from a different node in the cluster.

Related Issues

Resolves #17065

Check List

Functionality includes testing.
~~- [ ] API changes companion pull request created, if applicable.~~
~~- [ ] Public documentation issue/PR created, if applicable.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-02-25T00:31:38Z

✅ Gradle check result for 7738286: SUCCESS

codecov · 2025-02-25T00:33:31Z

Codecov Report

Attention: Patch coverage is 82.29508% with 54 lines in your changes missing coverage. Please review.

Project coverage is 72.40%. Comparing base (7b6108b) to head (02c7595).
Report is 29 commits behind head on main.

Files with missing lines	Patch %	Lines
...ensearch/arrow/flight/impl/FlightStreamTicket.java	70.45%	4 Missing and 9 partials ⚠️
server/src/main/java/org/opensearch/node/Node.java	0.00%	12 Missing and 1 partial ⚠️
...ensearch/arrow/flight/impl/BaseFlightProducer.java	91.81%	8 Missing and 1 partial ⚠️
...nsearch/arrow/flight/impl/FlightStreamManager.java	91.37%	3 Missing and 2 partials ⚠️
...ch/arrow/flight/bootstrap/FlightClientManager.java	77.77%	3 Missing and 1 partial ⚠️
...light/impl/CustomCallbackBackpressureStrategy.java	70.00%	1 Missing and 2 partials ⚠️
...c/main/java/org/opensearch/common/cache/Cache.java	75.00%	3 Missing ⚠️
...va/org/opensearch/plugins/StreamManagerPlugin.java	0.00%	2 Missing ⚠️
...rch/arrow/flight/bootstrap/FlightStreamPlugin.java	0.00%	0 Missing and 1 partial ⚠️
...nsearch/arrow/flight/impl/ProxyStreamProducer.java	95.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17446      +/-   ##
============================================
- Coverage     72.44%   72.40%   -0.05%     
- Complexity    66483    66583     +100     
============================================
  Files          5409     5417       +8     
  Lines        308282   308550     +268     
  Branches      44759    44776      +17     
============================================
+ Hits         223344   223408      +64     
- Misses        66608    66857     +249     
+ Partials      18330    18285      -45

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

...flight-rpc/src/internalClusterTest/java/org/opensearch/arrow/flight/ArrowFlightServerIT.java

plugins/arrow-flight-rpc/src/main/java/org/opensearch/arrow/flight/impl/BaseFlightProducer.java

Signed-off-by: Rishabh Maurya <[email protected]>

rishabhmaurya · 2025-04-09T22:25:05Z

@reta I have addressed your comments. I appreciate your thorough review of these PRs, so thank you for doing it.
It would be nice if it can make to 3.0 as a lot of POC code (spread across plugins) is dependent on it and it is becoming complex to manage feature branches.
we can merge #17738 prior to it.

github-actions · 2025-04-09T22:31:24Z

❕ Gradle check result for 90cf37a: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

...ins/arrow-flight-rpc/src/main/java/org/opensearch/arrow/flight/impl/ProxyStreamProducer.java

plugins/arrow-flight-rpc/src/main/java/org/opensearch/arrow/flight/impl/BaseFlightProducer.java

reta

Thanks @rishabhmaurya , great work! @mch2 anything left from your side? thanks!

Signed-off-by: Rishabh Maurya <[email protected]>

github-actions · 2025-04-11T01:06:51Z

❕ Gradle check result for 02c7595: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

mch2

lgtm, Thanks @rishabhmaurya excited to see where this goes!

gradle/run.gradle

* make use of extendedPlugins provide runtime dependencies for arrow-memrory-core Signed-off-by: Rishabh Maurya <[email protected]> * merged opensearch-arrow-core into arrow-flight-rpc plugin Signed-off-by: Rishabh Maurya <[email protected]> * moved libs:arrow-spi to server module Signed-off-by: Rishabh Maurya <[email protected]> * Flight producer changes Signed-off-by: Rishabh Maurya <[email protected]> * add hook in StreamManager plugin to get its instance on initialization Signed-off-by: Rishabh Maurya <[email protected]> * Minor refactor Signed-off-by: Rishabh Maurya <[email protected]> * Add more unit tests and address PR comment Signed-off-by: Rishabh Maurya <[email protected]> * Address PR comments Signed-off-by: Rishabh Maurya <[email protected]> * defer setting BufferAllocator in FlightStreamManager Signed-off-by: Rishabh Maurya <[email protected]> * Address PR comments Signed-off-by: Rishabh Maurya <[email protected]> * Refactor BaseFlightProducer and improved error handling Signed-off-by: Rishabh Maurya <[email protected]> * Address PR comments Signed-off-by: Rishabh Maurya <[email protected]> * refactor and improve test coverage Signed-off-by: Rishabh Maurya <[email protected]> * throw UnsupportedOperationException for ProxyStreamProducer#getAction() Signed-off-by: Rishabh Maurya <[email protected]> * refactor error handling Signed-off-by: Rishabh Maurya <[email protected]> --------- Signed-off-by: Rishabh Maurya <[email protected]> Signed-off-by: Sriram Ganesh <[email protected]>

* make use of extendedPlugins provide runtime dependencies for arrow-memrory-core Signed-off-by: Rishabh Maurya <[email protected]> * merged opensearch-arrow-core into arrow-flight-rpc plugin Signed-off-by: Rishabh Maurya <[email protected]> * moved libs:arrow-spi to server module Signed-off-by: Rishabh Maurya <[email protected]> * Flight producer changes Signed-off-by: Rishabh Maurya <[email protected]> * add hook in StreamManager plugin to get its instance on initialization Signed-off-by: Rishabh Maurya <[email protected]> * Minor refactor Signed-off-by: Rishabh Maurya <[email protected]> * Add more unit tests and address PR comment Signed-off-by: Rishabh Maurya <[email protected]> * Address PR comments Signed-off-by: Rishabh Maurya <[email protected]> * defer setting BufferAllocator in FlightStreamManager Signed-off-by: Rishabh Maurya <[email protected]> * Address PR comments Signed-off-by: Rishabh Maurya <[email protected]> * Refactor BaseFlightProducer and improved error handling Signed-off-by: Rishabh Maurya <[email protected]> * Address PR comments Signed-off-by: Rishabh Maurya <[email protected]> * refactor and improve test coverage Signed-off-by: Rishabh Maurya <[email protected]> * throw UnsupportedOperationException for ProxyStreamProducer#getAction() Signed-off-by: Rishabh Maurya <[email protected]> * refactor error handling Signed-off-by: Rishabh Maurya <[email protected]> --------- Signed-off-by: Rishabh Maurya <[email protected]> Signed-off-by: Harsh Kothari <[email protected]>

github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Performance labels Feb 24, 2025

rishabhmaurya self-assigned this Feb 24, 2025

rishabhmaurya added the skip-changelog label Feb 24, 2025

rishabhmaurya added this to Performance Roadmap Feb 24, 2025

github-project-automation bot moved this to Todo in Performance Roadmap Feb 24, 2025

rishabhmaurya added this to Search Project Board Feb 24, 2025

rishabhmaurya moved this to 🏗 In progress in Search Project Board Feb 24, 2025

rishabhmaurya moved this from Todo to In Progress in Performance Roadmap Feb 24, 2025

mch2 reviewed Feb 28, 2025

View reviewed changes

rishabhmaurya marked this pull request as ready for review March 5, 2025 22:05

rishabhmaurya requested review from Bukhtawar, CEHENKLE, anasalkouz, andrross, ashking94, bugmakerrrrrr, cwperks, dblock, dbwiddis, gbbafna, jainankitk, kotwanikunal, linuxpi, msfroh, nknize, owaiskazi19 and reta as code owners March 5, 2025 22:05

refactor and improve test coverage

90cf37a

Signed-off-by: Rishabh Maurya <[email protected]>

rishabhmaurya force-pushed the flight-base-impl branch from 82b10fc to 90cf37a Compare April 9, 2025 21:37

rishabhmaurya mentioned this pull request Apr 9, 2025

Arrow stream framework usage guidelines #17872

Closed

reta reviewed Apr 10, 2025

View reviewed changes

...ins/arrow-flight-rpc/src/main/java/org/opensearch/arrow/flight/impl/ProxyStreamProducer.java Show resolved Hide resolved

reta reviewed Apr 10, 2025

View reviewed changes

plugins/arrow-flight-rpc/src/main/java/org/opensearch/arrow/flight/impl/BaseFlightProducer.java Show resolved Hide resolved

reta approved these changes Apr 10, 2025

View reviewed changes

github-project-automation bot moved this from 🏗 In progress to 👀 In review in Search Project Board Apr 10, 2025

rishabhmaurya added 2 commits April 10, 2025 16:10

throw UnsupportedOperationException for ProxyStreamProducer#getAction()

2954b99

Signed-off-by: Rishabh Maurya <[email protected]>

refactor error handling

02c7595

Signed-off-by: Rishabh Maurya <[email protected]>

reta approved these changes Apr 11, 2025

View reviewed changes

mch2 approved these changes Apr 11, 2025

View reviewed changes

mch2 reviewed Apr 11, 2025

View reviewed changes

gradle/run.gradle Show resolved Hide resolved

mch2 merged commit 07cb4c9 into opensearch-project:main Apr 11, 2025
36 checks passed

github-project-automation bot moved this from In Progress to Done in Performance Roadmap Apr 11, 2025

github-project-automation bot moved this from 👀 In review to ✅ Done in Search Project Board Apr 11, 2025

jngz-es mentioned this pull request Apr 28, 2025

Best practice for dependencies of extensible plugins #18113

Open

opensearch-ci-bot mentioned this pull request Apr 29, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SearchIT #18129

Closed

opensearch-ci-bot mentioned this pull request May 15, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreMultipartCoreTestCase #18302

Open

opensearch-ci-bot mentioned this pull request May 14, 2025

[AUTOCUT] Gradle Check Flaky Test Report for WorkloadManagementIT #17154

Open

opensearch-ci-bot mentioned this pull request Sep 8, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ArrowFlightServerIT #19257

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add the base FlightProducer for getStream API #17446

Add the base FlightProducer for getStream API #17446

Uh oh!

rishabhmaurya commented Feb 24, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Feb 25, 2025

Uh oh!

codecov bot commented Feb 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rishabhmaurya commented Apr 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

Uh oh!

Uh oh!

reta left a comment

Uh oh!

github-actions bot commented Apr 11, 2025

Uh oh!

mch2 left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add the base FlightProducer for getStream API #17446

Add the base FlightProducer for getStream API #17446

Uh oh!

Conversation

rishabhmaurya commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Feb 25, 2025

Uh oh!

codecov bot commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rishabhmaurya commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

Uh oh!

Uh oh!

reta left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 11, 2025

Uh oh!

mch2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rishabhmaurya commented Feb 24, 2025 •

edited

Loading

codecov bot commented Feb 25, 2025 •

edited

Loading

rishabhmaurya commented Apr 9, 2025 •

edited

Loading