[PoC] Dynamic loading of Analysis Plugin

[amankr-amazon]

Description

Overview
OpenSearch plugins are add-on components that extend the functionality of OpenSearch, providing specialized features for text analysis, security, monitoring, data transformation, and external system integration. Plugin installation typically requires restarting OpenSearch process to complete the registration adding to operational overhead.

The requirement of restarting OpenSearch process comes from the fact that plugins can only be loaded during node bootstrap while all required services/modules (e.g. pluginsService, AnalysisModule) are being initialized. We are working on a project for dynamic loading of plugins (RFC will be published soon) i.e. adding/removing plugin without restarting the OpenSearch process. As part of the project, we did a PoC for dynamic loading of Analysis Plugin. This PR consolidates all modifications and implementations from the PoC work.

Implementation Summary:

1. Added 3 APIs to load, register and remove plugins on demand
   a. _plugins/load: Loads plugin jars from a path specified. Optionally, it registers the plugin if specified.
   b. _plugins/register: Registers the plugin i.e. update the AnalysisRegistry with the components of newly loaded plugin
   c. _plugins/remove: Removes plugin components from Analysis Registry

2. Updated PluginsService class with following:
   a. Added dynamic plugin loading methods e.g. loadPluginDynamically(Path pluginPath), loadPluginByName(String pluginName, Path pluginsDirectory)
   b. Updated loadPluginClass() method with class loader isolation
   c. Updated loadBundle() method with lenient class loader validation

3. Updates in AnalysisRegistry Class:
   a. Changed immutable maps to ConcurrentHashMap to update AnalysisRegistry with plugin components at runtime
   b. Added dynamic plugin component management methods to update the AnalysisRegistry
   c. Added cache invalidation when new plugins are added

Additionally, this PoC includes plugin management and security-related modifications.
Note: These changes were implemented to expedite PoC development and are not intended for production use. These should be replaced with dedicated plugin management APIs and refactored plugin loading mechanisms to maintain or enhance security standards.

1. Updated _cat/plugins API to show status of loaded plugins loaded/active
   • loaded: Plugin classes are loaded in OpenSearch JVM
   • active: AnalysisRegistry is updated with plugin components
2. Relaxed file permissions to allow OpenSearch process to read plugin from any directory specified while executing _plugins/load API.

Testing:

With above mentioned changes were able to add multiple analysis plugin at runtime. Following are the test execution details of adding/removing 'analysis-icu' plugin with single node cluster at runtime:

1. Executing '_cat/plugins' API to check installed plugins

(25-10-28 7:11:22) <0> [~]
% curl -s "localhost:9200/_cat/plugins?v"
name component version load_status

2. Currently, no plugins are installed. Let's try to create an index using analysis-icu plugin

(25-10-28 7:11:23) <0> [~]  
% curl -H'Content-Type: application/json'  -XPUT 'localhost:9200/icu_analyzer_index?pretty' -d '{ "settings": { "index": { "analysis": { "analyzer": { "folded": { "tokenizer": "icu_tokenizer", "filter": [ "icu_folding" ] } } } } } }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Custom Analyzer [folded] failed to find tokenizer under name [icu_tokenizer]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Custom Analyzer [folded] failed to find tokenizer under name [icu_tokenizer]"
  },
  "status" : 400
}

3. Index creation failed as expected since analysis-icu plugin is not installed currently. Let's load the analysis-icu plugin using '_plugins/load' API

(25-10-28 7:11:36) <0> [~]  
% curl -s -X POST "localhost:9200/_plugins/load?plugin_path=/home/akmarma/analysis-icu"
{"success":true,"message":"Successfully loaded 1 plugin(s)","loaded_plugins":["AnalysisICUPlugin"]}%                                                                                          

4. API executed successfully. Let's check the status using '_cat/plugins' API

(25-10-28 7:11:51) <0> [~]  
% curl -s "localhost:9200/_cat/plugins?v"
name      component    version load_status
runTask-0 analysis-icu 2.19.0  loaded

5. Plugin status is 'loaded' i.e. the plugin classes are loaded in OpenSearch JVM. Let's register the plugin using '_plugins/register' API

(25-10-28 7:11:54) <0> [~]  
% curl -X POST "localhost:9200/_plugins/register"
{"success":true,"message":"Analysis registry registered successfully","registered_components":["Analysis: CommonAnalysisPlugin","Analysis: AnalysisICUPlugin"]}%                              

6. '_plugins/register' API executed successfully. Let's check the status again using '_cat/plugins' API

(25-10-28 7:12:03) <0> [~]  
% curl -s "localhost:9200/_cat/plugins?v"                                              
name      component    version load_status
runTask-0 analysis-icu 2.19.0  active

7. Plugin status is updated to 'active' i.e. AnalysisRegistry is updated with plugin components. Let's try creating the index again.

(25-10-28 7:12:07) <0> [~]  
% curl -H'Content-Type: application/json'  -XPUT 'localhost:9200/icu_analyzer_index?pretty' -d '{ "settings": { "index": { "analysis": { "analyzer": { "folded": { "tokenizer": "icu_tokenizer", "filter": [ "icu_folding" ] } } } } } }'
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "icu_analyzer_index"
}

8. Index creation using analysis-icu plugin is successful this time. Let's remove the plugin using '_plugins/remove' API

(25-10-28 7:12:12) <0> [~]  
% curl -s -X DELETE "localhost:9200/_plugins/remove?plugin_name=analysis-icu&refresh_analysis=true"
{"success":false,"message":"Failed to remove 1 plugin(s): analysis-icu (unsafe to remove: actively used by 1 indices: icu_analyzer_index. Use force=true to override)","removed_plugins":[]}% 

9. Plugin is not removed as we have 1 index using the plugin. Let's delete that index and try removing the plugin again

(25-10-28 7:12:46) <0> [~]  
% curl -X DELETE localhost:9200/icu_analyzer_index                                                 
{"acknowledged":true}%                                                                                                                                                                        

(25-10-28 7:13:06) <0> [~]  
% curl -s -X DELETE "localhost:9200/_plugins/remove?plugin_name=analysis-icu&refresh_analysis=true"
{"success":true,"message":"Successfully removed 1 plugin(s)","removed_plugins":["analysis-icu"]}%                                                                                             

10. '_plugins/remove' API execution is successful. Let's confirm with '_cat/plugins' API

(25-10-28 7:13:12) <0> [~]  
% curl -s "localhost:9200/_cat/plugins?v"
name component version load_status

11. Plugin is removed from '_cat/plugins' API info as well.

Related Issues

Placeholder RFC link

[github-actions]

❌ Gradle check result for 8ccaae5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[cwperks]

@amankr-amazon this is showing 3k+ commits. Does it have the right base branch?

[amankr-amazon]

@amankr-amazon this is showing 3k+ commits. Does it have the right base branch?

updated base branch to 2.19. Thanks