[GRPC] Extend transport-grpc-spi to support GRPC KNN queries

[karenyrx]

Description

Used in conjunction with opensearch-project/OpenSearch#18949

Related Issues

Part of #2816
Related to opensearch-project/OpenSearch#18513

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[karenyrx]

Hi @navneet1v @shatejas @finnroblin, the CI failure seems unrelated to this PR, as I didn't chnage any of the files related to CMake or JNI. Could you share any insights on how this could be fixed? Thanks!

CMake Error at external/nmslib/similarity_search/CMakeLists.txt:73 (message):

  Unrecognized compiler (use GCC, Clang, Intel compiler, or MSVC (on
> Task :cmakeJniLib FAILED
  Windows)!

[finnroblin]

Hi @peterzhuamazon , do you know why the MacOS build would fail (with no changes to C++/JNI that would break the compilation)? Has any infrastructure changed recently on the MacOS CI runners? Compilation is working for me locally on MacOS 15.3.2 with both JDK 21 and JDK 24. Thanks

[finnroblin]

LGTM

[Vikasht34]

Can we have complete unit test and if possible unit test for this PR?

[navneet1v]

@karenyrx can you please resolve the comments and get this PR in good shape. The KNN build is failing without this change

[karenyrx]

Can we have complete unit test and if possible unit test for this PR?

A good test is that that previous unit tests continue to pass (as this is mostly a refactor), but added some new ones as well to cover more edge cases

[navneet1v]

@karenyrx I have created an issue for mac os Fix: #2876.

Lets resolve all the comments and then merge this code since the builds are blocked.

[navneet1v]

@karenyrx the CIs are failing

[navneet1v]

Approving the code, since CI is only left. Overall code looks good to me.

[navneet1v]

@Vikasht34 can you please review this PR now, this PR needs to be merged since this is blocking the builds.

[Vikasht34]

LGTM

[bzhangam]

@karenyrx @finnroblin @Vikasht34 @navneet1v After this PR is merged we can run into new issues(test):

> Could not resolve all files for configuration ':testFixturesCompileClasspath'.
   > Could not resolve com.google.errorprone:error_prone_annotations:2.26.1.
     Required by:
         root project : > com.google.guava:guava:33.2.1-jre
      > Conflict found for module 'com.google.errorprone:error_prone_annotations': between versions 2.41.0 and 2.26.1
> There is 1 more failure with an identical cause.

I have tried to do below to fix it in this PR and seems like it can pass the github checks.

    // com.google.guava:guava:33.2.1-jre is using com.google.errorprone:error_prone_annotations:2.26.1 but OpenSearch
    // core is using 2.41.0 so manually force to use 2.41.0 to avoid the conflict.
    testFixturesImplementation('com.google.guava:guava:33.2.1-jre'){
        exclude group: 'com.google.errorprone', module: 'error_prone_annotations'
    }
    testFixturesImplementation group: 'com.google.errorprone', name: 'error_prone_annotations', version: '2.41.0'

But when I start the cluster locally and try to test the knn query it will fail with error:

» java.lang.NoClassDefFoundError: com/google/common/util/concurrent/internal/InternalFutureFailureAccess » at java.base/java.lang.ClassLoader.defineClass1(Native Method) » at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)

Which seems related to the change about

   compileOnly("org.opensearch.plugin:transport-grpc:${opensearch_version}") {
        exclude group: 'com.google.guava', module: 'guava'
        exclude group: 'com.google.guava', module: 'failureaccess'
        exclude group: 'com.google.errorprone', module: 'error_prone_annotations'
    }

Could you help take a look?

[karenyrx]

Opened up https://github.com/opensearch-project/k-NN/pull/2883/files for the fix

[karenyrx]

opensearch-project/OpenSearch@b9c5bc7#diff-1767b7f709666659fcea718930d017913630119903232332e9b9b08627ca7c0fR29 may be the culprit of the failures rather than this PR