Skip to content

Conversation

@peteralfonsi
Copy link
Contributor

@peteralfonsi peteralfonsi commented Apr 28, 2025

Description

Switches the percentiles agg implementation from AVLTreeDigest to MergingDigest. MergingDigest is now the recommended implementation from the t-digest library, and we see significant speedups by switching to it. (See more details in the original issue). Some latencies from percentiles aggregations on http_logs:

Field Baseline latency (ms) Modififed latency (ms)
timestamp 13,085 6,293
status 196,794 6,212

Since the implementations extend the same abstract class AbstractTDigest, this is a drag-and-drop change. Our extending class TDigestState just adds some StreamInput/StreamOutput methods, which use the result of the centroids() and centroidCount() methods which are overridden from the abstract class. I don't think we need additional tests. The existing tests in TDigestPercentilesAggregatorTests.java still pass after the change. However I will still mark this PR as a draft, please let me know if I'm missing something here...

I also don't think we need any tuning of the sole parameter compression, since in the library's method createDigest that creates a digest of the default implementation, it doesn't modify the input compression based on which implementation it's actually using.

Related Issues

Resolves #18122

Check List

  • Functionality includes testing.
  • [N/A] API changes companion pull request created, if applicable.
  • [N/A] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

❌ Gradle check result for 852c178: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@peteralfonsi
Copy link
Contributor Author

On further thought I will add some tests around serialization + serialization inter-version compatibility. Both classes provide their own serialization methods. Probably the most logical thing to do would be to add a version check, and for < 3.1 use the preexisting logic assuming TDigestState extends AVLTreeDigest, and for >= 3.1 just rely on the library's serialization code.

@github-actions
Copy link
Contributor

❌ Gradle check result for 4599cff: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

✅ Gradle check result for e4caace: SUCCESS

@msfroh msfroh merged commit 22a6194 into opensearch-project:main May 28, 2025
28 of 30 checks passed
Gagan6164 pushed a commit to Gagan6164/OpenSearch that referenced this pull request Jun 8, 2025
…t#18124)

---------

Signed-off-by: Peter Alfonsi <[email protected]>
Signed-off-by: Peter Alfonsi <[email protected]>
Co-authored-by: Peter Alfonsi <[email protected]>
Gagan6164 pushed a commit to Gagan6164/OpenSearch that referenced this pull request Jun 8, 2025
…t#18124)

---------

Signed-off-by: Peter Alfonsi <[email protected]>
Signed-off-by: Peter Alfonsi <[email protected]>
Co-authored-by: Peter Alfonsi <[email protected]>
peteralfonsi pushed a commit to peteralfonsi/OpenSearch that referenced this pull request Jun 11, 2025
peteralfonsi pushed a commit to peteralfonsi/OpenSearch that referenced this pull request Jun 11, 2025
msfroh pushed a commit that referenced this pull request Jun 13, 2025
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jun 13, 2025
…18497)

---------

Signed-off-by: Peter Alfonsi <[email protected]>
Co-authored-by: Peter Alfonsi <[email protected]>
(cherry picked from commit afb08a0)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
peteralfonsi added a commit to peteralfonsi/OpenSearch that referenced this pull request Jun 13, 2025
abhita pushed a commit to abhita/OpenSearch that referenced this pull request Jun 17, 2025
owaiskazi19 pushed a commit that referenced this pull request Jun 18, 2025
…18497)

---------

Signed-off-by: Peter Alfonsi <[email protected]>
Co-authored-by: Peter Alfonsi <[email protected]>
(cherry picked from commit afb08a0)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
owaiskazi19 pushed a commit that referenced this pull request Jun 18, 2025
…18497) (#18517)

---------



(cherry picked from commit afb08a0)

Signed-off-by: Peter Alfonsi <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Peter Alfonsi <[email protected]>
neuenfeldttj added a commit to neuenfeldttj/OpenSearch that referenced this pull request Jun 26, 2025
…t#18124)

---------

Signed-off-by: Peter Alfonsi <[email protected]>
Signed-off-by: Peter Alfonsi <[email protected]>
Co-authored-by: Peter Alfonsi <[email protected]>Signed-off-by: TJ Neuenfeldt <[email protected]>
neuenfeldttj added a commit to neuenfeldttj/OpenSearch that referenced this pull request Jun 26, 2025
neuenfeldttj pushed a commit to neuenfeldttj/OpenSearch that referenced this pull request Jun 26, 2025
…t#18124)

---------

Signed-off-by: Peter Alfonsi <[email protected]>
Signed-off-by: Peter Alfonsi <[email protected]>
Co-authored-by: Peter Alfonsi <[email protected]>
neuenfeldttj pushed a commit to neuenfeldttj/OpenSearch that referenced this pull request Jun 26, 2025
tandonks pushed a commit to tandonks/OpenSearch that referenced this pull request Aug 5, 2025
…t#18124)

---------

Signed-off-by: Peter Alfonsi <[email protected]>
Signed-off-by: Peter Alfonsi <[email protected]>
Co-authored-by: Peter Alfonsi <[email protected]>
tandonks pushed a commit to tandonks/OpenSearch that referenced this pull request Aug 5, 2025
peteralfonsi pushed a commit to peteralfonsi/OpenSearch that referenced this pull request Oct 15, 2025
jainankitk pushed a commit that referenced this pull request Oct 17, 2025
rgsriram pushed a commit to rgsriram/OpenSearch that referenced this pull request Oct 18, 2025
kh3ra pushed a commit to kh3ra/OpenSearch that referenced this pull request Oct 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Enhancement or improvement to existing feature or request Search:Performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Speed up percentile aggregation by switching implementation

2 participants