Merge branch 'main' into fips_build_tooling2

Author: iigonin

CHANGELOG.md

@@ -12,9 +12,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add a mapper for context aware segments grouping criteria ([#19233](https://github.com/opensearch-project/OpenSearch/pull/19233))
 - Return full error for GRPC error response ([#19568](https://github.com/opensearch-project/OpenSearch/pull/19568))
 - Add pluggable gRPC interceptors with explicit ordering([#19005](https://github.com/opensearch-project/OpenSearch/pull/19005))
+- Add BindableServices extension point to transport-grpc-spi ([#19304](https://github.com/opensearch-project/OpenSearch/pull/19304))
 - Add metrics for the merged segment warmer feature ([#18929](https://github.com/opensearch-project/OpenSearch/pull/18929))
 - Add pointer based lag metric in pull-based ingestion ([#19635](https://github.com/opensearch-project/OpenSearch/pull/19635))
 - Introduced internal API for retrieving metadata about requested indices from transport actions  ([#18523](https://github.com/opensearch-project/OpenSearch/pull/18523))
+- Add cluster defaults for merge autoThrottle, maxMergeThreads, and maxMergeCount; Add segment size filter to the merged segment warmer ([#19629](https://github.com/opensearch-project/OpenSearch/pull/19629))
 - Add build-tooling to run in FIPS environment ([#18921](https://github.com/opensearch-project/OpenSearch/pull/18921))
 
 ### Changed
@@ -27,15 +29,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Wrap checked exceptions in painless.DefBootstrap to support JDK-25 ([#19706](https://github.com/opensearch-project/OpenSearch/pull/19706))
 - Refactor the ThreadPoolStats.Stats class to use the Builder pattern instead of constructors ([#19317](https://github.com/opensearch-project/OpenSearch/pull/19317))
 - Refactor the IndexingStats.Stats class to use the Builder pattern instead of constructors ([#19306](https://github.com/opensearch-project/OpenSearch/pull/19306))
-
+- Remove FeatureFlag.MERGED_SEGMENT_WARMER_EXPERIMENTAL_FLAG. ([#19715](https://github.com/opensearch-project/OpenSearch/pull/19715))
+-
 ### Fixed
 - Fix Allocation and Rebalance Constraints of WeightFunction are incorrectly reset ([#19012](https://github.com/opensearch-project/OpenSearch/pull/19012))
 - Fix flaky test FieldDataLoadingIT.testIndicesFieldDataCacheSizeSetting ([#19571](https://github.com/opensearch-project/OpenSearch/pull/19571))
+- Fix case-insensitive wildcard + aggregation query crash ([#19489](https://github.com/opensearch-project/OpenSearch/pull/19489))
 - Avoid primary shard failure caused by merged segment warmer exceptions ([#19436](https://github.com/opensearch-project/OpenSearch/pull/19436))
 - Fix pull-based ingestion out-of-bounds offset scenarios and remove persisted offsets ([#19607](https://github.com/opensearch-project/OpenSearch/pull/19607))
 - Fix issue with updating core with a patch number other than 0 ([#19377](https://github.com/opensearch-project/OpenSearch/pull/19377))
 - [Java Agent] Allow JRT protocol URLs in protection domain extraction ([#19683](https://github.com/opensearch-project/OpenSearch/pull/19683))
 - Fix potential concurrent modification exception when updating allocation filters ([#19701])(https://github.com/opensearch-project/OpenSearch/pull/19701))
+- Fix file-based ingestion consumer to handle start point beyond max line number([#19757])(https://github.com/opensearch-project/OpenSearch/pull/19757))
 
 ### Dependencies
 - Update to Gradle 9.1 ([#19575](https://github.com/opensearch-project/OpenSearch/pull/19575))

buildSrc/build.gradle

@@ -133,6 +133,14 @@ dependencies {
   integTestImplementation('org.spockframework:spock-core:2.4-M6-groovy-4.0') {
     exclude module: "groovy"
   }
+  // Integration with JUnit 4 features for Spock 2+ (which internally uses JUnit Platform -
+  // part of JUnit 5) - https://spockframework.org/spock/docs/2.4-M6/modules.html
+  integTestImplementation('org.spockframework:spock-junit4:2.4-M6-groovy-4.0') {
+    exclude module: "groovy"
+  }
+  integTestImplementation(platform("org.junit:junit-bom:${props.getProperty('junit5')}"))
+  integTestRuntimeOnly("org.junit.platform:junit-platform-launcher")
+  integTestRuntimeOnly("org.junit.vintage:junit-vintage-engine")
 }
 
 configurations.all {
@@ -253,6 +261,7 @@ if (project != rootProject) {
   }
 
   tasks.register("integTest", Test) {
+    useJUnitPlatform()
     inputs.dir(file("src/testKit")).withPropertyName("testkit dir").withPathSensitivity(PathSensitivity.RELATIVE)
     systemProperty 'test.version_under_test', version
     onlyIf { org.opensearch.gradle.info.BuildParams.isInFipsJvm() == false }

buildSrc/src/integTest/groovy/org/opensearch/gradle/DistributionDownloadPluginFuncTest.groovy

@@ -56,8 +56,8 @@ class DistributionDownloadPluginFuncTest extends AbstractGradleFuncTest {
         where:
         version                              | platform                                   | distType
         VersionProperties.getOpenSearch()    | OpenSearchDistribution.Platform.LINUX      | "current"
-        "8.1.0-SNAPSHOT"                     | OpenSearchDistribution.Platform.LINUX   | "bwc"
-        "7.0.0"                              | OpenSearchDistribution.Platform.WINDOWS | "released"
+        "2.19.0-SNAPSHOT"                    | OpenSearchDistribution.Platform.LINUX   | "bwc"
+        "3.3.0"                              | OpenSearchDistribution.Platform.WINDOWS | "released"
     }
 
 
@@ -72,7 +72,8 @@ class DistributionDownloadPluginFuncTest extends AbstractGradleFuncTest {
         """
 
         when:
-        def runner = gradleRunner('clean', 'setupDistro', '-i')
+        def customGradleUserHome = testProjectDir.newFolder().absolutePath;
+        def runner = gradleRunner('clean', 'setupDistro', '-i', '-g', customGradleUserHome)
         def result = withMockedDistributionDownload(version, platform, runner) {
             // initial run
             build()
@@ -82,7 +83,8 @@ class DistributionDownloadPluginFuncTest extends AbstractGradleFuncTest {
 
         then:
         result.task(":setupDistro").outcome == TaskOutcome.SUCCESS
-        assertOutputContains(result.output, "Skipping ${SymbolicLinkPreservingUntarTransform.class.simpleName}")
+        result.output.count("Unpacking opensearch-${version}-linux-x64.tar.gz " +
+            "using SymbolicLinkPreservingUntarTransform.") == 0
     }
 
     def "transforms are reused across projects"() {
@@ -91,6 +93,7 @@ class DistributionDownloadPluginFuncTest extends AbstractGradleFuncTest {
         def platform = OpenSearchDistribution.Platform.LINUX
 
         3.times {
+            testProjectDir.newFolder("sub-$it")
             settingsFile << """
                 include ':sub-$it'
             """

buildSrc/src/integTest/groovy/org/opensearch/gradle/JdkDownloadPluginFuncTest.groovy

@@ -108,6 +108,7 @@ class JdkDownloadPluginFuncTest extends AbstractGradleFuncTest {
         def mockRepoUrl = urlPath(jdkVendor, jdkVersion, platform)
         def mockedContent = filebytes(jdkVendor, platform)
         3.times {
+            testProjectDir.newFolder("sub-$it")
             settingsFile << """
                 include ':sub-$it'
             """
@@ -193,7 +194,7 @@ class JdkDownloadPluginFuncTest extends AbstractGradleFuncTest {
         }
 
         then:
-        assertOutputContains(result.output, "Skipping $transformType")
+        result.output.count("Unpacking ${platform}-12.0.2-x64.tar.gz using ${transformType}.") == 0
 
         where:
         platform  | transformType
@@ -246,6 +247,7 @@ class JdkDownloadPluginFuncTest extends AbstractGradleFuncTest {
                 p.repositories.all { repo ->
                     if(repo.name == "jdk_repo_${jdkVendor}_${jdkVersion}") {
                       repo.setUrl('${server.baseUrl()}')
+                      repo.setAllowInsecureProtocol(true)
                     }
                 }
            }"""

buildSrc/src/integTest/groovy/org/opensearch/gradle/internal/InternalDistributionArchiveSetupPluginFuncTest.groovy

@@ -98,6 +98,9 @@ class InternalDistributionArchiveSetupPluginFuncTest extends AbstractGradleFuncT
         given:
         file('someFile.txt') << "some content"
 
+        testProjectDir.newFolder("consumer")
+        testProjectDir.newFolder("producer-tar")
+
         settingsFile << """
             include ':consumer'
             include ':producer-tar'

buildSrc/src/integTest/groovy/org/opensearch/gradle/internal/InternalDistributionDownloadPluginFuncTest.groovy

@@ -122,6 +122,8 @@ class InternalDistributionDownloadPluginFuncTest extends AbstractGradleFuncTest
         internalBuild()
         bwcMinorProjectSetup()
         buildFile << """
+            import org.opensearch.gradle.JavaPackageType
+
             apply plugin: 'opensearch.internal-distribution-download'
 
             opensearch_distributions {
@@ -130,7 +132,7 @@ class InternalDistributionDownloadPluginFuncTest extends AbstractGradleFuncTest
                   type = "archive"
                   platform = "linux"
                   architecture = Architecture.current();
-                  bundledJdk = false
+                  bundledJdk = JavaPackageType.NONE
               }
             }
             tasks.register("createExtractedTestDistro") {

buildSrc/src/integTest/resources/org/opensearch/gradle/internal/fake_git/remote/distribution/archives/build.gradle

@@ -32,7 +32,7 @@ subprojects {
     apply plugin:'base'
 
     tasks.register('tar', Tar) {
-        from('.')
+        from('build.gradle')
         destinationDirectory.set(file('build/distributions'))
         archiveBaseName.set("opensearch")
         archiveVersion.set("8.0.1-SNAPSHOT")

buildSrc/src/testKit/testingConventions/build.gradle

@@ -21,7 +21,7 @@ allprojects {
     mavenCentral()
   }
   dependencies {
-    testImplementation "junit:junit:4.13.2"
+    testImplementation "junit:junit:${versions.junit}"
   }
 
   ext.licenseFile = file("$buildDir/dummy/license")

gradle/libs.versions.toml

@@ -75,6 +75,7 @@ picocli           = "4.7.7"
 # test dependencies
 randomizedrunner  = "2.7.1"
 junit             = "4.13.2"
+junit5            = "6.0.0"
 hamcrest          = "2.1"
 mockito           = "5.16.1"
 objenesis         = "3.3"

modules/transport-grpc/spi/README.md

@@ -9,6 +9,7 @@ The `transport-grpc-spi` module enables plugin developers to:
 - Extend gRPC protocol buffer handling
 - Register custom query types that can be processed via gRPC
 - Register gRPC interceptors with explicit ordering
+- Register `BindableService` implementation to the gRPC transport
 
 ## Key Components
 
@@ -47,6 +48,10 @@ public interface GrpcInterceptorProvider {
 }
 ```
 
+### GrpcServiceFactory
+
+Interface for providing a `BindableService` factory to be registered on the grpc transport.
+
 ## Usage for Plugin Developers
 
 ### 1. Add Dependency
@@ -63,7 +68,7 @@ dependencies {
 
 ### 2. Declare Extension in build.gradle
 
-In your `build.gradle`, declare that your plugin extends `transport-grpc`. This automatically adds the `extended.plugins=transport-grpc` entry to the auto-generated `plugin-descriptor.properties` file:  :
+In your `build.gradle`, declare that your plugin extends `transport-grpc`. This automatically adds the `extended.plugins=transport-grpc` entry to the auto-generated `plugin-descriptor.properties` file:
 
 ```groovy
 opensearchplugin {
@@ -81,20 +86,26 @@ opensearchplugin {
 
 Create a service file denoting your plugin's implementation of a service interface.
 
-For QueryBuilderProtoConverter implementations:
+For `QueryBuilderProtoConverter` implementations:
 `src/main/resources/META-INF/services/org.opensearch.transport.grpc.spi.QueryBuilderProtoConverter`:
 
 ```
 org.opensearch.mypackage.MyCustomQueryConverter
 ```
 
-For `GrpcInterceptorProvider` implementations: `src/main/resources/META-INF/services/org.opensearch.transport.grpc.spi.GrpcInterceptorProvider`:
+For `GrpcInterceptorProvider` implementations:
+`src/main/resources/META-INF/services/org.opensearch.transport.grpc.spi.GrpcInterceptorProvider`:
 
 ```
 org.opensearch.mypackage.SampleInterceptorProvider
 ```
 
+For `GrpcServiceFactory` implementations:
+`src/main/resources/META-INF/services/org.opensearch.transport.grpc.spi.GrpcServiceFactory`:
 
+```
+org.opensearch.mypackage.MyCustomGrpcServiceFactory
+```
 
 ## QueryBuilderProtoConverter
 ### 1. Implement Custom Query Converter
@@ -383,3 +394,25 @@ Each interceptor must have a unique order value. If duplicate order values are d
 IllegalArgumentException: Multiple gRPC interceptors have the same order value: 10.
 Each interceptor must have a unique order value.
 ```
+
+## GrpcServiceFactory
+
+### 1. Implement Custom Query Converter
+
+Several node resources are exposed to a `GrpcServiceFactory` for use within services such as client, settings, and thread pools.
+A plugin's `GrpcServiceFactory` implementation will be discovered through the SPI registration file and registered on the gRPC transport.
+
+```java
+public static class MockServiceProvider implements GrpcServiceFactory {
+
+    @Override
+    public String plugin() {
+        return "MockExtendingPlugin";
+    }
+
+    @Override
+    public List<BindableService> build() {
+        return List.of(new MockChannelzService());
+    }
+}
+```