Skip to content

[segment replication] Add cluster setting for retry timeout of publish checkpoint tx action #17749

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Apr 15, 2025
Merged

[segment replication] Add cluster setting for retry timeout of publish checkpoint tx action #17749

merged 12 commits into from
Apr 15, 2025

Conversation

@guojialiang92
Copy link
Contributor

@guojialiang92 guojialiang92 commented Apr 1, 2025 •
edited
Loading

Description

Added a test. In the current situation, if the primary shard publish checkpoint fails, it will cause the replica shard and the primary shard to fail to synchronize.
TransportReplicationAction support specifying retryTimeout.
PublishCheckpointAction use the never give up retry strategy.

Related Issues

Resolves 17595

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added bug Something isn't working Indexing:Replication Issues and PRs related to core replication framework eg segrep labels Apr 1, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 1, 2025

❌ Gradle check result for 1edc0ca: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@guojialiang92
TransportReplicationAction support specifying retryTimeout, PublishCh…
e49aa81 
…eckpointAction use the never give up strategy.

Signed-off-by: guojialiang <[email protected]>
@guojialiang92 guojialiang92 force-pushed the dev/PublishCheckpointAction_use_never_give_up_retry_strategy branch from 1edc0ca to e49aa81 Compare April 1, 2025 11:09
@github-actions
Copy link
Contributor

github-actions bot commented Apr 1, 2025

✅ Gradle check result for e49aa81: SUCCESS

ashking94
ashking94 approved these changes Apr 14, 2025
View reviewed changes
Copy link
Member

@ashking94 ashking94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@github-actions
Copy link
Contributor

github-actions bot commented Apr 14, 2025

❌ Gradle check result for b744f4b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@guojialiang92
add comments
68a5e9d 
Signed-off-by: guojialiang <[email protected]>
@guojialiang92 guojialiang92 force-pushed the dev/PublishCheckpointAction_use_never_give_up_retry_strategy branch from b744f4b to 68a5e9d Compare April 14, 2025 16:16
@github-actions
Copy link
Contributor

github-actions bot commented Apr 14, 2025

❌ Gradle check result for 68a5e9d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

This was referenced Apr 14, 2025
Open
Closed
Open
Open
Open
Open
Closed
Closed
Open
@guojialiang92
Merge remote-tracking branch 'origin/main' into dev/PublishCheckpoint…
3eb976e 
…Action_use_never_give_up_retry_strategy

# Conflicts:
#	CHANGELOG.md
@github-actions
Copy link
Contributor

github-actions bot commented Apr 15, 2025

❌ Gradle check result for 3eb976e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@ashking94
Copy link
Member

ashking94 commented Apr 15, 2025

❌ Gradle check result for 3eb976e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Restarted the pr build.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 15, 2025

❌ Gradle check result for 3eb976e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

github-actions bot commented Apr 15, 2025

❌ Gradle check result for a3a23a7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@guojialiang92
update
e7b926a 
Signed-off-by: guojialiang <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Apr 15, 2025

✅ Gradle check result for e7b926a: SUCCESS

@ashking94 ashking94 merged commit c44d230 into opensearch-project:main Apr 15, 2025
31 checks passed
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Apr 15, 2025
Open
@BrewTestBot BrewTestBot mentioned this pull request May 6, 2025
Merged
Harsh-87 pushed a commit to Harsh-87/OpenSearch that referenced this pull request May 7, 2025
@guojialiang92
[segment replication] Add cluster setting for retry timeout of publis…
6c6598b 
…h checkpoint tx action (opensearch-project#17749)

* TransportReplicationAction support specifying retryTimeout, PublishCheckpointAction use the never give up strategy.

Signed-off-by: guojialiang <[email protected]>

* support  PublishCheckpointAction PUBLISH_CHECK_POINT_RETRY_TIMEOUT to override the default retry timeout

Signed-off-by: guojialiang <[email protected]>

* add TransportReplicationAction.getRetryTimeoutSetting

Signed-off-by: guojialiang <[email protected]>

* add entry to CHANGELOG.md

Signed-off-by: guojialiang <[email protected]>

* rewrite the PR title

Signed-off-by: guojialiang <[email protected]>

* modify changelog entry

Signed-off-by: guojialiang <[email protected]>

* add comments

Signed-off-by: guojialiang <[email protected]>

* update

Signed-off-by: guojialiang <[email protected]>

---------

Signed-off-by: guojialiang <[email protected]>
Signed-off-by: Harsh Kothari <[email protected]>
Harsh-87 pushed a commit to Harsh-87/OpenSearch that referenced this pull request May 7, 2025
@guojialiang92
[segment replication] Add cluster setting for retry timeout of publis…
d5a00df 
…h checkpoint tx action (opensearch-project#17749)

* TransportReplicationAction support specifying retryTimeout, PublishCheckpointAction use the never give up strategy.

Signed-off-by: guojialiang <[email protected]>

* support  PublishCheckpointAction PUBLISH_CHECK_POINT_RETRY_TIMEOUT to override the default retry timeout

Signed-off-by: guojialiang <[email protected]>

* add TransportReplicationAction.getRetryTimeoutSetting

Signed-off-by: guojialiang <[email protected]>

* add entry to CHANGELOG.md

Signed-off-by: guojialiang <[email protected]>

* rewrite the PR title

Signed-off-by: guojialiang <[email protected]>

* modify changelog entry

Signed-off-by: guojialiang <[email protected]>

* add comments

Signed-off-by: guojialiang <[email protected]>

* update

Signed-off-by: guojialiang <[email protected]>

---------

Signed-off-by: guojialiang <[email protected]>
Signed-off-by: Harsh Kothari <[email protected]>
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request May 23, 2025
Open
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Apr 1, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashking94 ashking94 ashking94 approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz

@andrross andrross Awaiting requested review from andrross

@bugmakerrrrrr bugmakerrrrrr Awaiting requested review from bugmakerrrrrr

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE

@cwperks cwperks Awaiting requested review from cwperks

@dblock dblock Awaiting requested review from dblock

@dbwiddis dbwiddis Awaiting requested review from dbwiddis

@gbbafna gbbafna Awaiting requested review from gbbafna

@jainankitk jainankitk Awaiting requested review from jainankitk

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal

@linuxpi linuxpi Awaiting requested review from linuxpi

@mch2 mch2 Awaiting requested review from mch2

@msfroh msfroh Awaiting requested review from msfroh

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19

@reta reta Awaiting requested review from reta

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159

@sachinpkale sachinpkale Awaiting requested review from sachinpkale

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli

@shwetathareja shwetathareja Awaiting requested review from shwetathareja

@sohami sohami Awaiting requested review from sohami

@VachaShah VachaShah Awaiting requested review from VachaShah

Assignees

No one assigned

Labels

bug Something isn't working Indexing:Replication Issues and PRs related to core replication framework eg segrep

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[BUG] segment replication stops when publish checkpoint fails

2 participants

@guojialiang92 @ashking94