Make test-suite runnable under FIPS JVM #18491

beanuwave · 2025-06-11T08:36:56Z

Description

Makes the required changes to build and test under FIPS-140-3 compliance support. FIPS mode can be activated by adding the -Pcrypto.standard=FIPS-140-3 Gradle parameter.

NOTE:

This PR includes all remaining changes from the original PR#14912
Due to the netty FIPS reflection bug in v4.1.125.Final and v4.1.126.Final, the :plugins:transport-reactor-netty4:test task will fail. To run properly, netty needs to be updated to a newer version, such as v4.1.127.Final.

Related Issues

Resolves RFC

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-06-11T09:41:17Z

❌ Gradle check result for 9b5da5c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-07T12:36:21Z

❌ Gradle check result for 986dce7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

terryquigleysas · 2025-07-14T10:36:42Z

I'd like to raise a general point here to keep in mind with this development, especially as instructions will be required for Java setups other than the bundled version. The Red Hat JDK 21, for example, has a default of fips.keystore.type: PKCS12 - see https://docs.redhat.com/en/documentation/red_hat_build_of_openjdk/21/html/configuring_red_hat_build_of_openjdk_21_on_rhel_with_fips/fips_settings#fips_settings .

We'd like to ensure that code checks aren't so stringent as to prevent this setup from working.

github-actions · 2025-07-14T14:42:29Z

❌ Gradle check result for 939e6b5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T11:21:52Z

❌ Gradle check result for 11da667: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T11:31:38Z

❌ Gradle check result for 9a387a4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T12:12:16Z

❌ Gradle check result for 4e0af75: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T12:43:23Z

❌ Gradle check result for 9efd838: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T13:49:30Z

❌ Gradle check result for 4fc6b40: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T15:10:51Z

❌ Gradle check result for 0139eaa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-07-15T17:54:34Z

❌ Gradle check result for f52e720: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

beanuwave · 2025-07-16T12:21:25Z

The Red Hat JDK 21, for example, has a default of fips.keystore.type: PKCS12

@terryquigleysas Thank you for pointing out those limitations.

We'd like to ensure that code checks aren't so stringent as to prevent this setup from working.

We rely on SunPKCS12 provider to load the JVM's default truststore. In case of OpenJKD the default type is the same as RHEL's - so nothing changes for us.

terryquigleysas · 2025-07-16T13:28:01Z

The Red Hat JDK 21, for example, has a default of fips.keystore.type: PKCS12

@terryquigleysas Thank you for pointing out those limitations.

We'd like to ensure that code checks aren't so stringent as to prevent this setup from working.

We rely on SunPKCS12 provider to load the JVM's default truststore. In case of OpenJKD the default type is the same as RHEL's - so nothing changes for us.

Good news. Thank you for the reply. Much appreciated!

github-actions · 2025-08-01T16:25:22Z

❌ Gradle check result for cb7949d: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-08-04T12:12:41Z

❌ Gradle check result for 0680de8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-08-04T13:20:27Z

❌ Gradle check result for 732e412: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…folders in tests' Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

github-actions · 2025-11-04T12:19:33Z

❌ Gradle check result for 0da4c6d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

github-actions · 2025-11-04T12:59:22Z

❌ Gradle check result for fcc6a05: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

github-actions · 2025-11-04T15:08:07Z

✅ Gradle check result for bbf73cf: SUCCESS

beanuwave · 2025-11-04T15:20:38Z

@cwperks @reta I believe this PR is ready for review - can you take a look please?

reta · 2025-11-06T02:51:54Z

@cwperks @reta I believe this PR is ready for review - can you take a look please?

Thanks @beanuwave , sorry haven't looked yet but will do this week, thank you

...aller-cli/src/test/java/org/opensearch/tools/cli/fips/truststore/TrustStoreServiceTests.java

reta · 2025-11-07T15:09:31Z

client/rest/src/test/java/org/opensearch/client/RestClientFipsAwareTestCase.java

+
+import static org.opensearch.client.RestClientTestCase.inFipsJvm;
+
+public interface RestClientFipsAwareTestCase {


I don't think we should make it public - it is just for module tests

Suggested change

public interface RestClientFipsAwareTestCase {

interface RestClientFipsAwareTestCase {

reta · 2025-11-07T15:11:30Z

client/rest/src/test/java/org/opensearch/client/RestClientFipsAwareTestCase.java

+public interface RestClientFipsAwareTestCase {
+
+    default SSLContext getSslContext(boolean server) throws Exception {
+        String keyStoreType = inFipsJvm() ? "BCFKS" : "JKS";


To make it a bit more readable, please:

default SSLContext getSslContext(boolean server) throws Exception { if (inFipsJvm()) { return getSslContext(server, "BCFKS, SecureRandom.getInstance("DEFAULT", "BCFIPS"), ".bcfks"); } else { return getSslContext(server, new SecureRandom(), ".jks"); } }

reta · 2025-11-07T15:16:54Z

libs/ssl-config/src/main/java/org/opensearch/common/ssl/KeyStoreUtil.java

+        }
+    };
+
+    public static final Map<String, List<String>> TYPE_TO_EXTENSION_MAP = new HashMap<>();


Suggested change

public static final Map<String, List<String>> TYPE_TO_EXTENSION_MAP = new HashMap<>();

public static final Map<String, List<String>> TYPE_TO_EXTENSION_MAP = Map.of(

"JKS", List.of(".jks", ".ks"),

"PKCS12", List.of(".p12", ".pkcs12", ".pfx"),

"BCFKS", List.of(".bcfks")); // Bouncy Castle FIPS Keystore

reta · 2025-11-07T15:25:32Z

libs/ssl-config/src/main/java/org/opensearch/common/ssl/PemKeyConfig.java

        try {
-            final PrivateKey privateKey = PemUtils.readPrivateKey(key, () -> keyPassword);
+            final PrivateKey privateKey = PemUtils.readPrivateKey(key, () -> {
+                if (keyPassword.length == 0) {


Since we always need password now, could we move this validation to constructor? (where we do null check now)

reta · 2025-11-07T15:40:51Z

...es/transport-grpc/src/test/java/org/opensearch/transport/grpc/ssl/SecureSettingsHelpers.java

            public Optional<SSLContext> buildSecureAuxServerTransportContext(Settings settings, String auxTransportType)
                throws SSLException {
                // Choose a random protocol from among supported test defaults
-                String protocol = randomFrom(DEFAULT_SSL_PROTOCOLS);


Shouldn't we pick from FIPS_APPROVED_PROTOCOLS or DEFAULT_SSL_PROTOCOLS here?

reta · 2025-11-07T15:42:31Z

...terTest/java/org/opensearch/discovery/azure/classic/AzureDiscoveryClusterFormationTests.java

-                (PrivilegedAction<Version>) () -> Version.parse(System.getProperty("java.version"))
-            );
-            if (full.compareTo(Version.parse("12.0.1")) < 0) {
+        if (!inFipsJvm()) {


Not related, but our baseline is JDK-21, we could clean up this code

reta · 2025-11-07T15:45:38Z

.../repository-gcs/src/main/java/org/opensearch/repositories/gcs/GoogleCloudStorageService.java

+            if (Security.getProvider("BCFIPS") != null) {
+                certTrustStore = KeyStore.getInstance("BCFKS");
+                InputStream keyStoreStream = getClass().getResourceAsStream("/google.bcfks");
+                SecurityUtils.loadKeyStore(certTrustStore, keyStoreStream, "notasecret");


I believe we definitely should not hardcode password here, please have a setting for it

Let's make a setting for SSL trust as well as it's passphrase w/o a fallback and use this google.bcfks store only for tests.

reta · 2025-11-07T15:56:55Z

@cwperks @reta I believe this PR is ready for review - can you take a look please?

Thanks @beanuwave , sorry haven't looked yet but will do this week, thank you

Thanks @beanuwave , did a first pass, a few minor comments but I have two major concerns:

Incredible amount of "if" logic related to FIPS mode checks (and its variations like assumeFalse). I would like to suggest to split test cases in XxxBaseTest and XxxNoFipsTest extends XxxBaseTest (the naming is not very important). With this setup, we could easily exclude *NoFipsTest* from test runs that use FIPS mode (and XxxBaseTest would be run in any mode)
We now have to run all test suites in 2 modes - w/o FIPS. AFAICT this is not done yet, what is your plan here?

Thank you.

beanuwave · 2025-11-07T17:13:30Z

We now have to run all test suites in 2 modes - w/o FIPS. AFAICT this is not done yet, what is your plan here?

I believe @cwperks suggested creating a new GitHub workflow that supports both scheduled runs and PR comments.

…or TYPE_TO_EXTENSION_MAP; revert changes to SecureSettingsHelpers.getSecureSettingsProvider; revert changes to AzureClassic Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

reta · 2025-11-14T15:30:54Z

I believe @cwperks suggested creating a new GitHub workflow that supports both scheduled runs and PR comments.

Sorry, a bit late here, I think this is in general viable option, but:

we need to make sure out build scripts are adapted (we build on Jenkins, not GA)
we need to run the these checks in parallel

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

… google.p12 in non-FIPS mode and an exception in FIPS mode when no truststore is set up. Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

github-actions · 2025-11-17T12:21:56Z

❌ Gradle check result for 07c6358: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

opensearch-ci-bot mentioned this pull request Jun 11, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RecoveryWhileUnderLoadIT #14509

Open

beanuwave force-pushed the fips_compliance3 branch from 939e6b5 to 11da667 Compare July 15, 2025 11:17

beanuwave force-pushed the fips_compliance3 branch from 11da667 to 9a387a4 Compare July 15, 2025 11:26

beanuwave force-pushed the fips_compliance3 branch from 9a387a4 to 4e0af75 Compare July 15, 2025 12:07

beanuwave force-pushed the fips_compliance3 branch from 4e0af75 to 9efd838 Compare July 15, 2025 12:20

beanuwave force-pushed the fips_compliance3 branch from 9efd838 to 4fc6b40 Compare July 15, 2025 13:22

beanuwave force-pushed the fips_compliance3 branch from 4fc6b40 to 0139eaa Compare July 15, 2025 14:58

beanuwave force-pushed the fips_compliance3 branch from 0139eaa to f52e720 Compare July 15, 2025 15:51

beanuwave force-pushed the fips_compliance3 branch from f52e720 to cb7949d Compare August 1, 2025 14:22

opensearch-ci-bot mentioned this pull request Aug 1, 2025

[AUTOCUT] Gradle Check Flaky Test Report for FullRollingRestartIT #18490

Closed

beanuwave force-pushed the fips_compliance3 branch from cb7949d to 0680de8 Compare August 4, 2025 11:47

beanuwave changed the title ~~remaining changes from 14912~~ Make test-suite runnable under FIPS JVM Aug 4, 2025

beanuwave force-pushed the fips_compliance3 branch from 0680de8 to 732e412 Compare August 4, 2025 13:15

beanuwave force-pushed the fips_compliance3 branch from 732e412 to 6164088 Compare August 4, 2025 13:36

beanuwave force-pushed the fips_compliance3 branch from d892331 to 0da4c6d Compare November 4, 2025 12:07

use testFipsRuntimeOnly instead of fipsRuntimeOnly for :client:sniffer

fcc6a05

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

Use reflection inside libs:ssl-config instead of CryptoServicesRegistrar

bbf73cf

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

beanuwave force-pushed the fips_compliance3 branch from f8aa312 to bbf73cf Compare November 4, 2025 13:51

beanuwave marked this pull request as ready for review November 4, 2025 15:09

beanuwave requested review from a team and peternied as code owners November 4, 2025 15:09

cwperks reviewed Nov 6, 2025

View reviewed changes

...aller-cli/src/test/java/org/opensearch/tools/cli/fips/truststore/TrustStoreServiceTests.java Outdated Show resolved Hide resolved

reta reviewed Nov 7, 2025

View reviewed changes

iigonin and others added 2 commits November 13, 2025 10:45

Separate FIPS-specific tests into dedicated FipsTests classes

10bec59

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>

iigonin and others added 3 commits November 14, 2025 17:32

Merge branch 'main' into fips_compliance3

5a83e64

fix spotlessCheck

f8cd57b

Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>


		import static org.opensearch.client.RestClientTestCase.inFipsJvm;

		public interface RestClientFipsAwareTestCase {

	public interface RestClientFipsAwareTestCase {
	interface RestClientFipsAwareTestCase {

-    public static final Map<String, List<String>> TYPE_TO_EXTENSION_MAP = new HashMap<>();
+    public static final Map<String, List<String>> TYPE_TO_EXTENSION_MAP = Map.of(
+        "JKS", List.of(".jks", ".ks"),
+        "PKCS12", List.of(".p12", ".pkcs12", ".pfx"),
+        "BCFKS", List.of(".bcfks")); // Bouncy Castle FIPS Keystore

Make test-suite runnable under FIPS JVM #18491

Are you sure you want to change the base?

Make test-suite runnable under FIPS JVM #18491

Uh oh!

Conversation

beanuwave commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

terryquigleysas commented Jul 14, 2025

Uh oh!

github-actions bot commented Jul 14, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

beanuwave commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

terryquigleysas commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 1, 2025

Uh oh!

github-actions bot commented Aug 4, 2025

Uh oh!

github-actions bot commented Aug 4, 2025

Uh oh!

github-actions bot commented Nov 4, 2025

Uh oh!

github-actions bot commented Nov 4, 2025

Uh oh!

github-actions bot commented Nov 4, 2025

Uh oh!

beanuwave commented Nov 4, 2025

Uh oh!

reta commented Nov 6, 2025

Uh oh!

Uh oh!

reta Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

reta Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

reta Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

reta Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

reta Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

reta Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reta Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

beanuwave Nov 7, 2025

Choose a reason for hiding this comment

beanuwave commented Jun 11, 2025 •

edited

Loading

beanuwave commented Jul 16, 2025 •

edited

Loading

terryquigleysas commented Jul 16, 2025 •

edited

Loading

reta Nov 7, 2025 •

edited

Loading