Merge branch 'main' into fix-rescore-named-queries

Signed-off-by: Marc Handalian <[email protected]>

Author: mch2

CHANGELOG.md

@@ -7,8 +7,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ### Added
 - Add support for Warm Indices Write Block on Flood Watermark breach ([#18375](https://github.com/opensearch-project/OpenSearch/pull/18375))
 - Add support for custom index name resolver from cluster plugin ([#18593](https://github.com/opensearch-project/OpenSearch/pull/18593))
+- Rename WorkloadGroupTestUtil to WorkloadManagementTestUtil ([#18709](https://github.com/opensearch-project/OpenSearch/pull/18709))
 - Disallow resize for Warm Index, add Parameterized ITs for close in remote store ([#18686](https://github.com/opensearch-project/OpenSearch/pull/18686))
 - Ability to run Code Coverage with Gradle and produce the jacoco reports locally ([#18509](https://github.com/opensearch-project/OpenSearch/issues/18509))
+- [Workload Management] Update logging and Javadoc, rename QueryGroup to WorkloadGroup ([#18711](https://github.com/opensearch-project/OpenSearch/issues/18711))
 - Add NodeResourceUsageStats to ClusterInfo ([#18480](https://github.com/opensearch-project/OpenSearch/issues/18472))
 - Introduce SecureHttpTransportParameters experimental API (to complement SecureTransportParameters counterpart) ([#18572](https://github.com/opensearch-project/OpenSearch/issues/18572))
 - Create equivalents of JSM's AccessController in the java agent ([#18346](https://github.com/opensearch-project/OpenSearch/issues/18346))
@@ -21,6 +23,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add support for non-timing info in profiler ([#18460](https://github.com/opensearch-project/OpenSearch/issues/18460))
 - Extend Approximation Framework to other numeric types ([#18530](https://github.com/opensearch-project/OpenSearch/issues/18530))
 - Add Semantic Version field type mapper and extensive unit tests([#18454](https://github.com/opensearch-project/OpenSearch/pull/18454))
+- Pass index settings to system ingest processor factories. ([#18708](https://github.com/opensearch-project/OpenSearch/pull/18708))
 - Include named queries from rescore contexts in matched_queries array ([#18697](https://github.com/opensearch-project/OpenSearch/pull/18697))
 
 ### Changed
@@ -42,6 +45,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `org.bouncycastle:bcpkix-fips` from 2.0.7 to 2.0.8 ([#18668](https://github.com/opensearch-project/OpenSearch/pull/18668))
 - Bump `org.bouncycastle:bcpg-fips` from 2.0.10 to 2.0.11 ([#18668](https://github.com/opensearch-project/OpenSearch/pull/18668))
 - Bump `com.password4j:password4j` from 1.8.2 to 1.8.3 ([#18668](https://github.com/opensearch-project/OpenSearch/pull/18668))
+- Bump `com.azure:azure-core` from 1.55.3 to 1.55.5 ([#18691](https://github.com/opensearch-project/OpenSearch/pull/18691))
 
 ### Deprecated
 
@@ -56,6 +60,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix the visit of sub queries for HasParentQuery and HasChildQuery ([#18621](https://github.com/opensearch-project/OpenSearch/pull/18621))
 - Fix the backward compatibility regression with COMPLEMENT for Regexp queries introduced in OpenSearch 3.0 ([#18640](https://github.com/opensearch-project/OpenSearch/pull/18640))
 - Fix Replication lag computation ([#18602](https://github.com/opensearch-project/OpenSearch/pull/18602))
+- Fixed Staggered merge -  load average replace with AverageTrackers, some Default thresholds modified ([#18666](https://github.com/opensearch-project/OpenSearch/pull/18666))
 
 ### Security
 

MAINTAINERS.md

@@ -28,6 +28,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
 | Owais Kazi        | [owaiskazi19](https://github.com/owaiskazi19)           | Amazon      |
 | Pan Guixin        | [bugmakerrrrrr](https://github.com/bugmakerrrrrr)       | ByteDance   |
 | Peter Nied        | [peternied](https://github.com/peternied)               | Amazon      |
+| Prudhvi Godithi   | [prudhvigodithi](https://github.com/prudhvigodithi)     | Amazon      |
 | Rishabh Maurya    | [rishabhmaurya](https://github.com/rishabhmaurya)       | Amazon      |
 | Rishikesh Pasham  | [Rishikesh1159](https://github.com/Rishikesh1159)       | Amazon      |
 | Sachin Kale       | [sachinpkale](https://github.com/sachinpkale)           | Amazon      |

modules/autotagging-commons/common/src/main/java/org/opensearch/rule/autotagging/Rule.java

@@ -31,7 +31,7 @@
  * of a rule. The indexed view may differ in representation.
  * {
  *     "id": "fwehf8302582mglfio349==",
- *     "description": "Assign Query Group for Index Logs123"
+ *     "description": "Assign Workload Group for Index Logs123"
  *     "index_pattern": ["logs123"],
  *     "workload_group": "dev_workload_group_id",
  *     "updated_at": "01-10-2025T21:23:21.456Z"

plugins/examples/system-ingest-processor/src/main/java/org/opensearch/example/systemingestprocessor/ExampleSystemIngestProcessorFactory.java

@@ -8,15 +8,20 @@
 
 package org.opensearch.example.systemingestprocessor;
 
+import org.opensearch.common.settings.Settings;
 import org.opensearch.ingest.AbstractBatchingSystemProcessor;
 
 import java.util.ArrayList;
+import java.util.Collection;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 
+import static org.opensearch.example.systemingestprocessor.ExampleSystemIngestProcessorPlugin.TRIGGER_SETTING;
 import static org.opensearch.plugins.IngestPlugin.SystemIngestPipelineConfigKeys.INDEX_MAPPINGS;
+import static org.opensearch.plugins.IngestPlugin.SystemIngestPipelineConfigKeys.INDEX_SETTINGS;
 import static org.opensearch.plugins.IngestPlugin.SystemIngestPipelineConfigKeys.INDEX_TEMPLATE_MAPPINGS;
+import static org.opensearch.plugins.IngestPlugin.SystemIngestPipelineConfigKeys.INDEX_TEMPLATE_SETTINGS;
 
 /**
  * A factory to create the example system ingest processor
@@ -55,17 +60,26 @@ protected ExampleSystemIngestProcessorFactory() {
     @Override
     protected AbstractBatchingSystemProcessor newProcessor(String tag, String description, Map<String, Object> config) {
         final List<Map<String, Object>> mappings = new ArrayList<>();
+        final List<Settings> settings = new ArrayList<>();
         final Object mappingFromIndex = config.get(INDEX_MAPPINGS);
         final Object mappingFromTemplates = config.get(INDEX_TEMPLATE_MAPPINGS);
+        final Object settingsFromIndex = config.get(INDEX_SETTINGS);
+        final Object settingsFromTemplates = config.get(INDEX_TEMPLATE_SETTINGS);
         if (mappingFromTemplates instanceof List) {
             mappings.addAll((List<Map<String, Object>>) mappingFromTemplates);
         }
         if (mappingFromIndex instanceof Map) {
             mappings.add((Map<String, Object>) mappingFromIndex);
         }
+        if (settingsFromTemplates instanceof List) {
+            settings.addAll((Collection<? extends Settings>) settingsFromTemplates);
+        }
+        if (settingsFromIndex instanceof Settings) {
+            settings.add((Settings) settingsFromIndex);
+        }
 
         // If no config we are not able to create a processor so simply return a null to show no processor created
-        if (mappings.isEmpty()) {
+        if (mappings.isEmpty() && settings.isEmpty()) {
             return null;
         }
 
@@ -87,6 +101,15 @@ protected AbstractBatchingSystemProcessor newProcessor(String tag, String descri
             }
         }
 
+        // If the trigger setting is configured then use it directly.
+        // When we rely on the v1 template to create the index there can be multiple settings and the later one can
+        // override the previous one so we need to loop through all the settings.
+        for (final Settings setting : settings) {
+            if (setting.hasValue(TRIGGER_SETTING.getKey())) {
+                isTriggerFieldFound = TRIGGER_SETTING.get(setting);
+            }
+        }
+
         return isTriggerFieldFound ? new ExampleSystemIngestProcessor(tag, description, DEFAULT_BATCH_SIZE) : null;
     }
 

plugins/examples/system-ingest-processor/src/main/java/org/opensearch/example/systemingestprocessor/ExampleSystemIngestProcessorPlugin.java

@@ -8,10 +8,12 @@
 
 package org.opensearch.example.systemingestprocessor;
 
+import org.opensearch.common.settings.Setting;
 import org.opensearch.ingest.Processor;
 import org.opensearch.plugins.IngestPlugin;
 import org.opensearch.plugins.Plugin;
 
+import java.util.List;
 import java.util.Map;
 
 /**
@@ -23,8 +25,23 @@ public class ExampleSystemIngestProcessorPlugin extends Plugin implements Ingest
      */
     public ExampleSystemIngestProcessorPlugin() {}
 
+    /**
+     * A custom index setting which is used to control if we should create the example system ingest processor.
+     */
+    public static final Setting<Boolean> TRIGGER_SETTING = Setting.boolSetting(
+        "index.example_system_ingest_processor_plugin.trigger_setting",
+        false,
+        Setting.Property.IndexScope,
+        Setting.Property.Dynamic
+    );
+
     @Override
     public Map<String, Processor.Factory> getSystemIngestProcessors(Processor.Parameters parameters) {
         return Map.of(ExampleSystemIngestProcessorFactory.TYPE, new ExampleSystemIngestProcessorFactory());
     }
+
+    @Override
+    public List<Setting<?>> getSettings() {
+        return List.of(TRIGGER_SETTING);
+    }
 }

plugins/examples/system-ingest-processor/src/test/java/org/opensearch/example/systemingestprocessor/ExampleSystemIngestProcessorFactoryTests.java

@@ -8,6 +8,7 @@
 
 package org.opensearch.example.systemingestprocessor;
 
+import org.opensearch.common.settings.Settings;
 import org.opensearch.ingest.AbstractBatchingSystemProcessor;
 import org.opensearch.test.OpenSearchTestCase;
 
@@ -18,8 +19,11 @@
 import static org.opensearch.example.systemingestprocessor.ExampleSystemIngestProcessorFactory.DOC;
 import static org.opensearch.example.systemingestprocessor.ExampleSystemIngestProcessorFactory.PROPERTIES;
 import static org.opensearch.example.systemingestprocessor.ExampleSystemIngestProcessorFactory.TRIGGER_FIELD_NAME;
+import static org.opensearch.example.systemingestprocessor.ExampleSystemIngestProcessorPlugin.TRIGGER_SETTING;
 import static org.opensearch.plugins.IngestPlugin.SystemIngestPipelineConfigKeys.INDEX_MAPPINGS;
+import static org.opensearch.plugins.IngestPlugin.SystemIngestPipelineConfigKeys.INDEX_SETTINGS;
 import static org.opensearch.plugins.IngestPlugin.SystemIngestPipelineConfigKeys.INDEX_TEMPLATE_MAPPINGS;
+import static org.opensearch.plugins.IngestPlugin.SystemIngestPipelineConfigKeys.INDEX_TEMPLATE_SETTINGS;
 
 public class ExampleSystemIngestProcessorFactoryTests extends OpenSearchTestCase {
     public void testNewProcessor_whenWithTriggerField_thenReturnProcessor() {
@@ -55,4 +59,37 @@ public void testNewProcessor_whenNoMapping_thenReturnNull() {
 
         assertNull("Should not create an example system ingest processor when the mapping is not found.", processor);
     }
+
+    public void testNewProcessor_whenWithTriggerSettingFromIndex_thenReturnProcessor() {
+        final ExampleSystemIngestProcessorFactory factory = new ExampleSystemIngestProcessorFactory();
+        Settings triggerEnabled = Settings.builder().put(TRIGGER_SETTING.getKey(), true).build();
+
+        AbstractBatchingSystemProcessor processor = factory.newProcessor("tag", "description", Map.of(INDEX_SETTINGS, triggerEnabled));
+
+        assertNotNull("Should create an example system ingest processor when the trigger_setting is true.", processor);
+        assertTrue(processor instanceof ExampleSystemIngestProcessor);
+    }
+
+    public void testNewProcessor_whenWithTriggerSettingFromTemplate_thenReturnProcessor() {
+        final ExampleSystemIngestProcessorFactory factory = new ExampleSystemIngestProcessorFactory();
+        Settings triggerEnabled = Settings.builder().put(TRIGGER_SETTING.getKey(), true).build();
+
+        AbstractBatchingSystemProcessor processor = factory.newProcessor(
+            "tag",
+            "description",
+            Map.of(INDEX_TEMPLATE_SETTINGS, List.of(triggerEnabled))
+        );
+
+        assertNotNull("Should create an example system ingest processor when the trigger_setting is true.", processor);
+        assertTrue(processor instanceof ExampleSystemIngestProcessor);
+    }
+
+    public void testNewProcessor_whenWithTriggerSettingDisabled_thenReturnProcessor() {
+        final ExampleSystemIngestProcessorFactory factory = new ExampleSystemIngestProcessorFactory();
+        Settings triggerDisabled = Settings.builder().put(TRIGGER_SETTING.getKey(), false).build();
+
+        AbstractBatchingSystemProcessor processor = factory.newProcessor("tag", "description", Map.of(INDEX_SETTINGS, triggerDisabled));
+
+        assertNull("Should not create an example system ingest processor when the trigger_setting is false.", processor);
+    }
 }

plugins/examples/system-ingest-processor/src/test/java/org/opensearch/example/systemingestprocessor/ExampleSystemIngestProcessorPluginTests.java

@@ -11,8 +11,10 @@
 import org.opensearch.ingest.Processor;
 import org.opensearch.test.OpenSearchTestCase;
 
+import java.util.List;
 import java.util.Map;
 
+import static org.opensearch.example.systemingestprocessor.ExampleSystemIngestProcessorPlugin.TRIGGER_SETTING;
 import static org.mockito.Mockito.mock;
 
 public class ExampleSystemIngestProcessorPluginTests extends OpenSearchTestCase {
@@ -27,4 +29,8 @@ public void testGetSystemIngestProcessors() {
             factories.get(ExampleSystemIngestProcessorFactory.TYPE) instanceof ExampleSystemIngestProcessorFactory
         );
     }
+
+    public void testGetSettings() {
+        assertEquals(List.of(TRIGGER_SETTING), plugin.getSettings());
+    }
 }

plugins/examples/system-ingest-processor/src/yamlRestTest/resources/rest-api-spec/test/example-system-ingest-processor/20_system_ingest_processor.yml

@@ -29,6 +29,28 @@ teardown:
   - match:
       _source.field_auto_added_by_system_ingest_processor: "This field is auto added by the example system ingest processor."
 
+---
+"Processor injects a field on indexing a doc to existing index when trigger_setting is true":
+  - do:
+      indices.create:
+        index: test-index
+        body:
+          settings:
+            index.example_system_ingest_processor_plugin.trigger_setting: true
+  - do:
+      index:
+        index: test-index
+        id: 1
+        body:
+          system_ingest_processor_trigger_field: "dummy value"
+        refresh: true
+  - do:
+      get:
+        index: test-index
+        id: 1
+  - match:
+      _source.field_auto_added_by_system_ingest_processor: "This field is auto added by the example system ingest processor."
+
 ---
 "Processor should not inject a field on indexing a doc to existing index when trigger field is not defined in the index mapping":
   - do:
@@ -56,6 +78,7 @@ teardown:
   - skip:
       features: allowed_warnings
   - do:
+      # test v1 template
       indices.put_template:
         name: example-template
         body:
@@ -79,7 +102,41 @@ teardown:
         id: 1
   - match:
       _source.field_auto_added_by_system_ingest_processor: "This field is auto added by the example system ingest processor."
+  - do:
+      indices.delete_template:
+        name: example-template
+---
 
+"Processor injects field when index is created from matching template where trigger_setting is true":
+  - skip:
+      features: allowed_warnings
+  - do:
+      # test v2 template
+      indices.put_index_template:
+        name: example-template
+        body:
+          index_patterns: ["template-*"]
+          template:
+            settings:
+              index.example_system_ingest_processor_plugin.trigger_setting: true
+  - do:
+      allowed_warnings:
+        - "index [template-index-1] matches multiple legacy templates [example-template, global], composable templates will only match a single template"
+      index:
+        index: template-index-1
+        id: 1
+        body:
+          system_ingest_processor_trigger_field: "dummy value"
+        refresh: true
+  - do:
+      get:
+        index: template-index-1
+        id: 1
+  - match:
+      _source.field_auto_added_by_system_ingest_processor: "This field is auto added by the example system ingest processor."
+  - do:
+      indices.delete_index_template:
+        name: example-template
 ---
 "Processor injects field on bulk indexing to existing index":
   - do:
@@ -156,7 +213,9 @@ teardown:
         id: 2
   - match:
       _source.field_auto_added_by_system_ingest_processor: "This field is auto added by the example system ingest processor."
-
+  - do:
+      indices.delete_template:
+        name: bulk-template
 ---
 "Processor injects field on bulk update, upsert on existing/new documents":
   # Temporarily disable system ingest pipelines to insert without triggering the system ingest field

plugins/repository-azure/build.gradle

@@ -44,7 +44,7 @@ opensearchplugin {
 }
 
 dependencies {
-  api 'com.azure:azure-core:1.55.3'
+  api 'com.azure:azure-core:1.55.5'
   api 'com.azure:azure-json:1.5.0'
   api 'com.azure:azure-xml:1.2.0'
   api 'com.azure:azure-storage-common:12.29.0'