Adding Cluster Endpoints for search failures and refactoring document status counters #19115

ajleong623 · 2025-08-20T17:15:04Z

Description

This change is a rebased and cleaned version of another pr #18601

Related Issues

Resolves #18377, resolves #18438

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-08-20T18:25:40Z

❌ Gradle check result for ab74705: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-08-21T04:58:25Z

❌ Gradle check result for ad9e107: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-08-21T05:21:26Z

❌ Gradle check result for 9e3af03: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-08-21T05:59:27Z

❌ Gradle check result for 7e9374a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-08-21T07:18:31Z

❌ Gradle check result for a345b21: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-08-21T07:48:26Z

❌ Gradle check result for ae2c661: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-08-22T19:42:40Z

❌ Gradle check result for c9ad00b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-08-22T20:49:10Z

❌ Gradle check result for 32a2fae: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

kaushalmahi12

These are some intial comments, will go through rest of it in couple of days

kaushalmahi12 · 2025-09-18T00:06:07Z

libs/core/src/main/java/org/opensearch/core/rest/RestStatus.java

+        } else {
+            return "system_failure";
+        }


Lets move the numeric values into well defined constants/Enums then directly use, StatusCode.from(getStatusFamilyCode)

kaushalmahi12 · 2025-09-22T18:12:13Z

server/src/main/java/org/opensearch/action/admin/indices/stats/DocStatusStats.java

+        String successType = RestStatus.ACCEPTED.getErrorType();
+        String userFailureType = RestStatus.BAD_REQUEST.getErrorType();
+        String systemErrorType = RestStatus.INTERNAL_SERVER_ERROR.getErrorType();


Are these the only RestStatus codes we need to track or all of them ? It would be better to encapsulate these into a collection to avoid tracking any new additions. That will help avoid any changes in this class.
It can easily be written more generically

for (RestStatus status: supportedStatusCodes) { builder.field(status.getErrorType(), errorTypeCounts.getOrDefault(status.getErrorType(), (long) 0)); }

The idea to group the possible RestStatus values was mentioned in #18438 (comment). I think it simplifies what the user views. I will add the new enums in a different file.

kaushalmahi12 · 2025-09-22T18:39:39Z

server/src/main/java/org/opensearch/action/admin/indices/stats/SearchResponseStatusStats.java

+ * @opensearch.api
+ */
+@PublicApi(since = "1.0.0")
+public class SearchResponseStatusStats implements Writeable, ToXContentFragment {


This looks pretty much duplicate of the DocStatusStats class. Can we make a generic class which will help us remove this duplicity

kaushalmahi12 · 2025-09-22T19:03:21Z

server/src/main/java/org/opensearch/action/bulk/TransportBulkAction.java

                    }
                }

-                indicesService.addDocStatusStats(stats);


If we are replacing the previous stats completely then why not just use the same method with updated logic.

kaushalmahi12 · 2025-09-22T20:28:08Z

server/src/main/java/org/opensearch/action/search/TransportSearchAction.java

+                cancellationListener.onResponse(searchResponse);
+                indicesService.getSearchResponseStatusStats().inc(searchResponse.status());


I am not clear on whether if you should put these instructions in this order it will be correct and might be flaky since it might have already sent the response via the httpChannel which is responsible for this request

I agree. I am not 100% sure if the code will continue to be run after the original listener receives the SearchResponse. I think it is safer to switch the orders just in case.

kaushalmahi12 · 2025-09-22T20:33:54Z

server/src/main/java/org/opensearch/action/update/TransportUpdateAction.java

                stats.inc(RestStatus.OK);

-                indicesService.addDocStatusStats(stats);
+                indicesService.addNewDocStatusStats(stats);


All of these changes can be avoided when you change the existing method instead.

An issue is that I could not directly change the signature of the existing method because of backwards compatibility. addDocStatusStats would have to taken in the new class for DocStatusStats. I could try to rewrite the method but with a different signature which is what I should have probably done in the first place.

kaushalmahi12 · 2025-09-22T20:37:25Z

server/src/main/java/org/opensearch/index/shard/IndexingStats.java

            this.isThrottled = isThrottled;
            this.throttleTimeInMillis = throttleTimeInMillis;
-            this.docStatusStats = docStatusStats;
+            this.docStatusStats = null;


If we are deprecating this construct then do we really need the ctor changes ?

github-actions · 2025-09-23T18:48:41Z

❌ Gradle check result for b544e35: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

…g623/OpenSearch into upgrade-cluster-endpoints

ajleong623 · 2025-09-23T21:02:24Z

@kaushalmahi12 Thank you so much for looking into my pr and reviewing it. The idea is that I would like to track search failures in some kind of counter which will be stored in IndicesService. All search requests are handled in TransportSearchAction, so that will be where the counters are updated.

Additionally, #8716 implemented a change where document response statuses are counted. Based on some comments from #18438 (comment) I refactored the logic as described in #18438 (comment).

I have addressed a couple of the comments but will continue to work through them. Thank you again for the review!

github-actions · 2025-09-23T21:22:28Z

❌ Gradle check result for 3455efb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

ajleong623 · 2025-09-24T18:29:42Z

@kaushalmahi12 I believe the requested changes have been addressed. I have made a new class AbstractStatusStats which holds the common methods between DocStatusStats and StatusCounterStats. Additionally, I believe I was able to replace the addDocStatusStats method in IndicesService without breaking backwards compatibility.

github-actions · 2025-09-24T18:33:04Z

❌ Gradle check result for a2aa335: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-09-24T19:19:21Z

❌ Gradle check result for 93f508f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-09-24T22:28:53Z

❌ Gradle check result for eb19367: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-09-25T00:29:22Z

❌ Gradle check result for 536c20b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-09-25T05:27:58Z

✅ Gradle check result for de8b27d: SUCCESS

ajleong623 · 2025-09-30T03:40:29Z

@kaushalmahi12 I believe the comments you suggested have been addressed. Could you please confirm whether that is true or not since I did not want to resolve the comments without making sure they are truly resolved? Also, I noticed that this PR is scheduled for a 3.3.0 release

opensearch-trigger-bot · 2025-11-02T15:22:05Z

This PR is stalled because it has been open for 30 days with no activity.

Signed-off-by: Anthony Leong <[email protected]>

github-actions · 2025-11-12T18:37:50Z

❌ Gradle check result for 917722c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

add first changes

ab74705

Signed-off-by: Anthony Leong <[email protected]>

ajleong623 requested a review from a team as a code owner August 20, 2025 17:15

github-actions bot added enhancement Enhancement or improvement to existing feature or request Search Search query, autocomplete ...etc Search:Query Insights labels Aug 20, 2025

This was referenced Aug 20, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreReplicationSourceTests #16683

Open

[AUTOCUT] Gradle Check Flaky Test Report for IndexStatsIT #15836

Open

ajleong623 marked this pull request as draft August 21, 2025 04:30

test for breaking detection

ad9e107

Signed-off-by: Anthony Leong <[email protected]>

test for breaking detection

9e3af03

Signed-off-by: Anthony Leong <[email protected]>

add changes

7e9374a

Signed-off-by: Anthony Leong <[email protected]>

ajleong623 added 2 commits August 20, 2025 23:27

fix compile issues

fe21417

Signed-off-by: Anthony Leong <[email protected]>

fix breaking changes

a345b21

Signed-off-by: Anthony Leong <[email protected]>

fix breaking changes

ae2c661

Signed-off-by: Anthony Leong <[email protected]>

ajleong623 added 2 commits August 22, 2025 12:01

added integration tests

ad95bff

Signed-off-by: Anthony Leong <[email protected]>

fix null pointer and breaking changes

c9ad00b

Signed-off-by: Anthony Leong <[email protected]>

test runner not found

32a2fae

Signed-off-by: Anthony Leong <[email protected]>

test compilation

8589524

Signed-off-by: Anthony Leong <[email protected]>

kaushalmahi12 reviewed Sep 22, 2025

View reviewed changes

Merge branch 'opensearch-project:main' into upgrade-cluster-endpoints

b544e35

ajleong623 added 2 commits September 23, 2025 13:52

refactor enum and stat holders

dc043dd

Signed-off-by: Anthony Leong <[email protected]>

Merge branch 'upgrade-cluster-endpoints' of https://github.com/ajleon…

3455efb

…g623/OpenSearch into upgrade-cluster-endpoints

This was referenced Sep 23, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

[AUTOCUT] Gradle Check Flaky Test Report for IngestFromKinesisIT #17678

Open

applied suggestions

a2aa335

Signed-off-by: Anthony Leong <[email protected]>

retry for flaky

93f508f

Signed-off-by: Anthony Leong <[email protected]>

Merge branch 'opensearch-project:main' into upgrade-cluster-endpoints

eb19367

retry for flaky

536c20b

Signed-off-by: Anthony Leong <[email protected]>

retry for flaky

de8b27d

Signed-off-by: Anthony Leong <[email protected]>

github-actions bot added bug Something isn't working Indexing Indexing, Bulk Indexing and anything related to indexing labels Oct 1, 2025

sandeshkr419 added v3.4.0 Issues and PRs related to version 3.4.0 and removed v3.3.0 labels Oct 1, 2025

opensearch-trigger-bot bot added the stalled Issues that have stalled label Nov 2, 2025

opensearch-trigger-bot bot removed the stalled Issues that have stalled label Nov 10, 2025

Merge branch 'main' into upgrade-cluster-endpoints

917722c

Signed-off-by: Anthony Leong <[email protected]>

		cancellationListener.onResponse(searchResponse);
		indicesService.getSearchResponseStatusStats().inc(searchResponse.status());

Adding Cluster Endpoints for search failures and refactoring document status counters #19115

Are you sure you want to change the base?

Adding Cluster Endpoints for search failures and refactoring document status counters #19115

Conversation

ajleong623 commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Aug 20, 2025

Uh oh!

github-actions bot commented Aug 21, 2025

Uh oh!

github-actions bot commented Aug 21, 2025

Uh oh!

github-actions bot commented Aug 21, 2025

Uh oh!

github-actions bot commented Aug 21, 2025

Uh oh!

github-actions bot commented Aug 21, 2025

Uh oh!

github-actions bot commented Aug 22, 2025

Uh oh!

github-actions bot commented Aug 22, 2025

Uh oh!

kaushalmahi12 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajleong623 Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajleong623 Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 23, 2025

Uh oh!

ajleong623 commented Sep 23, 2025

Uh oh!

github-actions bot commented Sep 23, 2025

Uh oh!

ajleong623 commented Sep 24, 2025

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

github-actions bot commented Sep 24, 2025

Uh oh!

github-actions bot commented Sep 25, 2025

Uh oh!

github-actions bot commented Sep 25, 2025

Uh oh!

ajleong623 commented Sep 30, 2025

Uh oh!

opensearch-trigger-bot bot commented Nov 2, 2025

Uh oh!

github-actions bot commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

ajleong623 commented Aug 20, 2025 •

edited

Loading

ajleong623 Sep 23, 2025 •

edited

Loading

ajleong623 Sep 23, 2025 •

edited

Loading