[merged segment warmer] support remote merged segment warmer #18683
Conversation
kh3ra
requested review from
a team,
Bukhtawar,
CEHENKLE,
Rishikesh1159,
VachaShah,
anasalkouz,
andrross,
ashking94,
cwperks,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
owaiskazi19,
reta,
sachinpkale,
saratvemulapalli,
shwetathareja and
sohami
as code owners
|
❌ Gradle check result for d7f0e5b: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
kh3ra
force-pushed
the
dev/remote-merged-segment-warmer
branch
from
d7f0e5b to
dae97b6
Compare
|
❌ Gradle check result for dae97b6: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
kh3ra
force-pushed
the
dev/remote-merged-segment-warmer
branch
from
dae97b6 to
628d4c8
Compare
|
❌ Gradle check result for 628d4c8: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for e4aae33: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
kh3ra
force-pushed
the
dev/remote-merged-segment-warmer
branch
from
e4aae33 to
48d3c50
Compare
|
❌ Gradle check result for 48d3c50: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: kh3ra <[email protected]>
Signed-off-by: kh3ra <[email protected]>
Signed-off-by: kh3ra <[email protected]>
Signed-off-by: kh3ra <[email protected]>
Signed-off-by: kh3ra <[email protected]>
Signed-off-by: kh3ra <[email protected]>
kh3ra
force-pushed
the
dev/remote-merged-segment-warmer
branch
from
0457da0 to
21d8c13
Compare
|
❕ Gradle check result for 21d8c13: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Signed-off-by: kh3ra <[email protected]>
|
❌ Gradle check result for 162b3eb: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Regarding codecov: Achieved 72.43% vs expected 72.77% The slight coverage gap is due to challenges in writing automated tests mergedSegmentWarmer, as it interacts with several final/private classes in Lucene's codebase which limits our testing capabilities at this point. |
|
@kh3ra Lets add a cluster setting to control enablement of remote merged segment warmer along with the experimental setting. If we want to disable it later, it would come in handy. |
…rch-project#18683) * Changes to support upload and download of merge segments using the IndexWriter.IndexWarmer.warm() flow Signed-off-by: kh3ra <[email protected]> * Unit tests Signed-off-by: kh3ra <[email protected]> * Fixing build issues Signed-off-by: kh3ra <[email protected]> * Fixing build issues - forbiddenAPIs/spotlessApply Signed-off-by: kh3ra <[email protected]> * Upload merge segments in low priority, minor fixes Signed-off-by: kh3ra <[email protected]> * Addressing review comments + rebase main Signed-off-by: kh3ra <[email protected]> * Test fixes + javadocs Signed-off-by: kh3ra <[email protected]> * 1. Bug fix to RemoteDirectory.DownloadRateLimiterProvider 2. Fixes to tests Signed-off-by: kh3ra <[email protected]> * Bug fix to replica updates to ActiveMergesRegistry Signed-off-by: kh3ra <[email protected]> * Addressing review comments - round 2 Signed-off-by: kh3ra <[email protected]> * new tests + test fixes + minor bug fixes Signed-off-by: kh3ra <[email protected]> * Bug fix Signed-off-by: kh3ra <[email protected]> * Fixes to RemoteStorePublishMergedSegmentActionTests Signed-off-by: kh3ra <[email protected]> * Adding integration tests Signed-off-by: kh3ra <[email protected]> * Tracking stats for merged segment warmer Signed-off-by: kh3ra <[email protected]> * Revert "Tracking stats for merged segment warmer" This reverts commit 1476e22. Signed-off-by: kh3ra <[email protected]> * Addressing review comments for tests Signed-off-by: kh3ra <[email protected]> * Addressing review comments Signed-off-by: kh3ra <[email protected]> * Rebasing Signed-off-by: kh3ra <[email protected]> * spotlessApply Signed-off-by: kh3ra <[email protected]> * test fix Signed-off-by: kh3ra <[email protected]> * Empty commit Signed-off-by: kh3ra <[email protected]> * Adding tests, enhancing logs Signed-off-by: kh3ra <[email protected]> * Adding MergedSegmentWarmerFactory tests + enhancing existing tests Signed-off-by: kh3ra <[email protected]> * Applying spotless Signed-off-by: kh3ra <[email protected]> * Added test for Timeout case for RemoteStoreReplicationSource Signed-off-by: kh3ra <[email protected]> * Restored RemoteSegmentStoreDirectory PublicAPI + added changelog Signed-off-by: kh3ra <[email protected]> --------- Signed-off-by: kh3ra <[email protected]> Signed-off-by: kh3ra <[email protected]>
3 participants
Description
This PR is based on PR18255's follow-up work. It implements the core process of warming merged segments in remote-store-enabled domains.
During the
IndexReaderWarmer#warm,RemoteStorePublishMergedSegmentRequestrequest containing information about the newly created merged segment.IndexReaderWarmer#warmcompletes and the merge resumes.Notes -
IndexReaderWarmer#warmflow are ignored and the merged segments are then expected to get replicated in the regular replication flow.Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.