Merge branch 'main' into searchable-system-index

Author: cwperks

CHANGELOG-3.0.md

@@ -14,12 +14,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - [WLM] Add WLM support for search scroll API ([#16981](https://github.com/opensearch-project/OpenSearch/pull/16981))
 - Allow to pass the list settings through environment variables (like [], ["a", "b", "c"], ...) ([#10625](https://github.com/opensearch-project/OpenSearch/pull/10625))
 - Views, simplify data access and manipulation by providing a virtual layer over one or more indices ([#11957](https://github.com/opensearch-project/OpenSearch/pull/11957))
+- Add systemd configurations to strengthen OS core security ([#17107](https://github.com/opensearch-project/OpenSearch/pull/17107))
 - Added pull-based Ingestion (APIs, for ingestion source, a Kafka plugin, and IngestionEngine that pulls data from the ingestion source) ([#16958](https://github.com/opensearch-project/OpenSearch/pull/16958))
 - Added ConfigurationUtils to core for the ease of configuration parsing [#17223](https://github.com/opensearch-project/OpenSearch/pull/17223)
+- Add cluster and index level settings to limit the total primary shards per node and per index [#17295](https://github.com/opensearch-project/OpenSearch/pull/17295)
+- Add execution_hint to cardinality aggregator request (#[17312](https://github.com/opensearch-project/OpenSearch/pull/17312))
+- Arrow Flight RPC plugin with Flight server bootstrap logic and client for internode communication ([#16962](https://github.com/opensearch-project/OpenSearch/pull/16962))
+- Added offset management for the pull-based Ingestion ([#17354](https://github.com/opensearch-project/OpenSearch/pull/17354))
 
 ### Dependencies
 - Update Apache Lucene to 10.1.0 ([#16366](https://github.com/opensearch-project/OpenSearch/pull/16366))
 - Bump Apache HttpCore5/HttpClient5 dependencies from 5.2.5/5.3.1 to 5.3.1/5.4.1 to support ExtendedSocketOption in HttpAsyncClient ([#16757](https://github.com/opensearch-project/OpenSearch/pull/16757))
+- Bumps `jetty` version from 9.4.55.v20240627 to 9.4.57.v20241219
 
 ### Changed
 - Changed locale provider from COMPAT to CLDR  ([#14345](https://github.com/opensearch-project/OpenSearch/pull/14345))
@@ -37,6 +43,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Stop minimizing automata used for case-insensitive matches ([#17268](https://github.com/opensearch-project/OpenSearch/pull/17268))
 - Refactor the `:server` module `org.opensearch.client` to `org.opensearch.transport.client` to eliminate top level split packages for JPMS support ([#17272](https://github.com/opensearch-project/OpenSearch/pull/17272))
 - Use Lucene `BM25Similarity` as default since the `LegacyBM25Similarity` is marked as deprecated ([#17306](https://github.com/opensearch-project/OpenSearch/pull/17306))
+- Wildcard field index only 3gram of the input data [#17349](https://github.com/opensearch-project/OpenSearch/pull/17349)
 
 ### Deprecated
 
@@ -66,6 +73,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Don't over-allocate in HeapBufferedAsyncEntityConsumer in order to consume the response ([#9993](https://github.com/opensearch-project/OpenSearch/pull/9993))
 - Fix swapped field formats in nodes API where `total_indexing_buffer_in_bytes` and `total_indexing_buffer` values were reversed ([#17070](https://github.com/opensearch-project/OpenSearch/pull/17070))
 - Add HTTP/2 protocol support to HttpRequest.HttpVersion ([#17248](https://github.com/opensearch-project/OpenSearch/pull/17248))
+- Fix missing bucket in terms aggregation with missing value ([#17418](https://github.com/opensearch-project/OpenSearch/pull/17418))
 
 ### Security
 

CHANGELOG.md

@@ -11,16 +11,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Introduce a setting to disable download of full cluster state from remote on term mismatch([#16798](https://github.com/opensearch-project/OpenSearch/pull/16798/))
 - Added ability to retrieve value from DocValues in a flat_object filed([#16802](https://github.com/opensearch-project/OpenSearch/pull/16802))
 - Improve performace of NumericTermAggregation by avoiding unnecessary sorting([#17252](https://github.com/opensearch-project/OpenSearch/pull/17252))
+- [Rule Based Auto-tagging] Add in-memory attribute value store ([#17342](https://github.com/opensearch-project/OpenSearch/pull/17342))
 - Add a flag to set whether a system index is readable on SystemIndexDescriptor ([#17296](https://github.com/opensearch-project/OpenSearch/pull/17296))
 
 ### Dependencies
-- Bump `org.awaitility:awaitility` from 4.2.0 to 4.2.2 ([#17230](https://github.com/opensearch-project/OpenSearch/pull/17230))
+- Bump `org.awaitility:awaitility` from 4.2.0 to 4.3.0 ([#17230](https://github.com/opensearch-project/OpenSearch/pull/17230), [#17439](https://github.com/opensearch-project/OpenSearch/pull/17439))
 - Bump `dnsjava:dnsjava` from 3.6.2 to 3.6.3 ([#17231](https://github.com/opensearch-project/OpenSearch/pull/17231))
 - Bump `com.google.code.gson:gson` from 2.11.0 to 2.12.1 ([#17229](https://github.com/opensearch-project/OpenSearch/pull/17229))
 - Bump `org.jruby.joni:joni` from 2.2.1 to 2.2.3 ([#17136](https://github.com/opensearch-project/OpenSearch/pull/17136))
 - Bump `org.apache.ant:ant` from 1.10.14 to 1.10.15 ([#17288](https://github.com/opensearch-project/OpenSearch/pull/17288))
 - Bump netty from 4.1.117.Final to 4.1.118.Final ([#17320](https://github.com/opensearch-project/OpenSearch/pull/17320))
 - Bump `reactor_netty` from 1.1.26 to 1.1.27 ([#17322](https://github.com/opensearch-project/OpenSearch/pull/17322))
+- Bump `me.champeau.gradle.japicmp` from 0.4.5 to 0.4.6 ([#17375](https://github.com/opensearch-project/OpenSearch/pull/17375))
+- Bump `com.google.api.grpc:proto-google-common-protos` from 2.37.1 to 2.52.0 ([#17379](https://github.com/opensearch-project/OpenSearch/pull/17379))
+- Bump `net.minidev:json-smart` from 2.5.1 to 2.5.2 ([#17378](https://github.com/opensearch-project/OpenSearch/pull/17378))
 
 ### Changed
 - Convert transport-reactor-netty4 to use gradle version catalog [#17233](https://github.com/opensearch-project/OpenSearch/pull/17233)
@@ -35,6 +39,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix exists queries on nested flat_object fields throws exception ([#16803](https://github.com/opensearch-project/OpenSearch/pull/16803))
 - Add highlighting for wildcard search on `match_only_text` field ([#17101](https://github.com/opensearch-project/OpenSearch/pull/17101))
 - Fix illegal argument exception when creating a PIT ([#16781](https://github.com/opensearch-project/OpenSearch/pull/16781))
+- Fix HTTP API calls that hang with 'Accept-Encoding: zstd' ([#17408](https://github.com/opensearch-project/OpenSearch/pull/17408))
 
 ### Security
 

MAINTAINERS.md

@@ -5,17 +5,17 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
 ## Current Maintainers
 
 | Maintainer               | GitHub ID                                               | Affiliation |
-|--------------------------|---------------------------------------------------------|-------------|
+| ------------------------ | ------------------------------------------------------- | ----------- |
 | Anas Alkouz              | [anasalkouz](https://github.com/anasalkouz)             | Amazon      |
 | Andrew Ross              | [andrross](https://github.com/andrross)                 | Amazon      |
-| Andriy Redko             | [reta](https://github.com/reta)                         | Aiven       |
+| Andriy Redko             | [reta](https://github.com/reta)                         | Independent |
 | Ankit Jain               | [jainankitk](https://github.com/jainankitk)             | Amazon      |
 | Ashish Singh             | [ashking94](https://github.com/ashking94)               | Amazon      |
 | Bukhtawar Khan           | [Bukhtawar](https://github.com/Bukhtawar)               | Amazon      |
 | Charlotte Henkle         | [CEHENKLE](https://github.com/CEHENKLE)                 | Amazon      |
 | Craig Perkins            | [cwperks](https://github.com/cwperks)                   | Amazon      |
 | Dan Widdis               | [dbwiddis](https://github.com/dbwiddis)                 | Amazon      |
-| Daniel "dB." Doubrovkine | [dblock](https://github.com/dblock)                     | Amazon      |
+| Daniel "dB." Doubrovkine | [dblock](https://github.com/dblock)                     | Independent |
 | Binlong Gao              | [gaobinlong](https://github.com/gaobinlong)             | Amazon      |
 | Gaurav Bafna             | [gbbafna](https://github.com/gbbafna)                   | Amazon      |
 | Jay Deng                 | [jed326](https://github.com/jed326)                     | Amazon      |
@@ -35,14 +35,14 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
 
 ## Emeritus
 
-| Maintainer             | GitHub ID                                   | Affiliation |
-| ---------------------- |-------------------------------------------- | ----------- |
-| Megha Sai Kavikondala  | [meghasaik](https://github.com/meghasaik)   | Amazon      |
-| Xue Zhou               | [xuezhou25](https://github.com/xuezhou25)   | Amazon      |
-| Kartik Ganesh          | [kartg](https://github.com/kartg)           | Amazon      |
-| Abbas Hussain          | [abbashus](https://github.com/abbashus)     | Meta        |
-| Himanshu Setia         | [setiah](https://github.com/setiah)         | Amazon      |
-| Ryan Bogan             | [ryanbogan](https://github.com/ryanbogan)   | Amazon      |
-| Rabi Panda             | [adnapibar](https://github.com/adnapibar)   | Independent |
-| Tianli Feng            | [tlfeng](https://github.com/tlfeng)         | Amazon      |
-| Suraj Singh            | [dreamer-89](https://github.com/dreamer-89) | Amazon      |
+| Maintainer            | GitHub ID                                   | Affiliation |
+| --------------------- | ------------------------------------------- | ----------- |
+| Megha Sai Kavikondala | [meghasaik](https://github.com/meghasaik)   | Amazon      |
+| Xue Zhou              | [xuezhou25](https://github.com/xuezhou25)   | Amazon      |
+| Kartik Ganesh         | [kartg](https://github.com/kartg)           | Amazon      |
+| Abbas Hussain         | [abbashus](https://github.com/abbashus)     | Meta        |
+| Himanshu Setia        | [setiah](https://github.com/setiah)         | Amazon      |
+| Ryan Bogan            | [ryanbogan](https://github.com/ryanbogan)   | Amazon      |
+| Rabi Panda            | [adnapibar](https://github.com/adnapibar)   | Independent |
+| Tianli Feng           | [tlfeng](https://github.com/tlfeng)         | Amazon      |
+| Suraj Singh           | [dreamer-89](https://github.com/dreamer-89) | Amazon      |

codecov.yml

@@ -4,6 +4,7 @@ codecov:
 ignore:
   - "test"
   - "benchmarks"
+  - "plugins/arrow-flight-rpc/**/org/apache/arrow/flight/**"
 
 coverage:
   precision: 2

distribution/packages/src/common/env/opensearch

@@ -3,17 +3,17 @@
 ################################
 
 # OpenSearch home directory
-#OPENSEARCH_HOME=/usr/share/opensearch
+OPENSEARCH_HOME=/usr/share/opensearch
 
 # OpenSearch Java path
-#OPENSEARCH_JAVA_HOME=
+#OPENSEARCH_JAVA_HOME=/usr/lib/jvm/java-11-amazon-corretto
 
 # OpenSearch configuration directory
 # Note: this setting will be shared with command-line tools
-OPENSEARCH_PATH_CONF=${path.conf}
+OPENSEARCH_PATH_CONF=/etc/opensearch
 
 # OpenSearch PID directory
-#PID_DIR=/var/run/opensearch
+PID_DIR=/var/run/opensearch
 
 # Additional Java OPTS
 #OPENSEARCH_JAVA_OPTS=
@@ -25,11 +25,12 @@ OPENSEARCH_PATH_CONF=${path.conf}
 # OpenSearch service
 ################################
 
-# SysV init.d
-#
 # The number of seconds to wait before checking if OpenSearch started successfully as a daemon process
 OPENSEARCH_STARTUP_SLEEP_TIME=5
 
+# Notification for systemd
+OPENSEARCH_SD_NOTIFY=true
+
 ################################
 # System properties
 ################################
@@ -49,4 +50,4 @@ OPENSEARCH_STARTUP_SLEEP_TIME=5
 # Maximum number of VMA (Virtual Memory Areas) a process can own
 # When using Systemd, this setting is ignored and the 'vm.max_map_count'
 # property is set at boot time in /usr/lib/sysctl.d/opensearch.conf
-#MAX_MAP_COUNT=262144
+#MAX_MAP_COUNT=262144

distribution/packages/src/common/systemd/opensearch.service

@@ -1,18 +1,25 @@
+# Copyright OpenSearch Contributors
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# Description:
+# Default opensearch.service file
+
 [Unit]
 Description=OpenSearch
-Documentation=https://www.elastic.co
+Documentation=https://opensearch.org/
 Wants=network-online.target
 After=network-online.target
 
 [Service]
 Type=notify
 RuntimeDirectory=opensearch
 PrivateTmp=true
-Environment=OPENSEARCH_HOME=/usr/share/opensearch
-Environment=OPENSEARCH_PATH_CONF=${path.conf}
-Environment=PID_DIR=/var/run/opensearch
-Environment=OPENSEARCH_SD_NOTIFY=true
-EnvironmentFile=-${path.env}
+EnvironmentFile=-/etc/default/opensearch
+EnvironmentFile=-/etc/sysconfig/opensearch
 
 WorkingDirectory=/usr/share/opensearch
 
@@ -29,6 +36,7 @@ ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.
 # logging, you can simply remove the "quiet" option from ExecStart.
 StandardOutput=journal
 StandardError=inherit
+SyslogIdentifier=opensearch
 
 # Specifies the maximum file descriptor number that can be opened by this process
 LimitNOFILE=65535
@@ -60,6 +68,97 @@ SuccessExitStatus=143
 # Allow a slow startup before the systemd notifier module kicks in to extend the timeout
 TimeoutStartSec=75
 
+# Prevent modifications to the control group filesystem
+ProtectControlGroups=true
+
+# Prevent loading or reading kernel modules
+ProtectKernelModules=true
+
+# Prevent altering kernel tunables (sysctl parameters)
+ProtectKernelTunables=true
+
+# Set device access policy to 'closed', allowing access only to specific devices
+DevicePolicy=closed
+
+# Make /proc invisible to the service, enhancing isolation
+ProtectProc=invisible
+
+# Make /usr, /boot, and /etc read-only (less restrictive than 'strict')
+ProtectSystem=full
+
+# Prevent changes to control groups (redundant with earlier setting, can be removed)
+ProtectControlGroups=yes
+
+# Prevent changing the execution domain
+LockPersonality=yes
+
+
+# System call filtering
+# System call filterings which restricts which system calls a process can make
+# @ means allowed
+# ~ means not allowed
+SystemCallFilter=@system-service
+SystemCallFilter=~@reboot
+SystemCallFilter=~@swap
+
+SystemCallErrorNumber=EPERM
+
+# Capability restrictions
+# Remove the ability to block system suspends
+CapabilityBoundingSet=~CAP_BLOCK_SUSPEND
+
+# Remove the ability to establish leases on files
+CapabilityBoundingSet=~CAP_LEASE
+
+# Remove the ability to use system resource accounting
+CapabilityBoundingSet=~CAP_SYS_PACCT
+
+# Remove the ability to configure TTY devices
+CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG
+
+# Remov below capabilities:
+# - CAP_SYS_ADMIN: Various system administration operations
+# - CAP_SYS_PTRACE: Ability to trace processes
+# - CAP_NET_ADMIN: Various network-related operations
+CapabilityBoundingSet=~CAP_SYS_ADMIN ~CAP_SYS_PTRACE ~CAP_NET_ADMIN
+
+
+# Address family restrictions
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
+
+# Filesystem Access
+
+ReadWritePaths=/var/log/opensearch
+ReadWritePaths=/var/lib/opensearch
+ReadWritePaths=-/etc/opensearch
+ReadWritePaths=-/mnt/snapshots
+
+## Allow read access to system files
+ReadOnlyPaths=/etc/os-release /usr/lib/os-release /etc/system-release
+
+## Allow read access to Linux IO stats
+ReadOnlyPaths=/proc/self/mountinfo /proc/diskstats
+
+## Allow read access to control group stats
+ReadOnlyPaths=/proc/self/cgroup /sys/fs/cgroup/cpu /sys/fs/cgroup/cpu/-
+ReadOnlyPaths=/sys/fs/cgroup/cpuacct /sys/fs/cgroup/cpuacct/- /sys/fs/cgroup/memory /sys/fs/cgroup/memory/-
+
+
+RestrictNamespaces=true
+
+NoNewPrivileges=true
+
+# Memory and execution protection
+MemoryDenyWriteExecute=true           # Prevent creating writable executable memory mappings
+SystemCallArchitectures=native        # Allow only native system calls
+KeyringMode=private                   # Service does not share key material with other services
+LockPersonality=true                  # Prevent changing ABI personality
+RestrictSUIDSGID=true                 # Prevent creating SUID/SGID files
+RestrictRealtime=true                 # Prevent acquiring realtime scheduling
+ProtectHostname=true                  # Prevent changes to system hostname
+ProtectKernelLogs=true                # Prevent reading/writing kernel logs
+ProtectClock=true                     # Prevent tampering with the system clock
+
 [Install]
 WantedBy=multi-user.target
 

gradle/libs.versions.toml

@@ -82,7 +82,7 @@ opentelemetry         = "1.46.0"
 opentelemetrysemconv  = "1.29.0-alpha"
 
 # arrow dependencies
-arrow                 = "17.0.0"
+arrow                 = "18.1.0"
 flatbuffers           = "2.0.0"
 
 [libraries]