[Pull-based Ingestion] Add error handling strategy to pull-based ingestion

[varunbharadwaj]

Description

This PR is a follow up for pull-based-ingestion to add error handling support. We introduce the following two strategies:

1. DROP: Drop errors and proceed with remaining updates in the ingestion source.
2. BLOCK: Block on any errors, preventing further writes.

This PR adds the drop/block support along with required interfaces. A follow up PR will add metric emission and record the errors.

When using Block strategy, we need a way to allow users to resume ingestion. Ingestion management APIs will be added to allow users more flexibility as part of #17442

Related Issues

Resolves #17085

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for dc4722f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 1b61879: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 7eee397: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 474008b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6dc359a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 441b463: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[andrross]

Looks good @varunbharadwaj. Can you add a changelog entry in CHANGELOG-3.0.md and also rebase this with the latest on main?

[github-actions]

✅ Gradle check result for 1904e14: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 88.52459% with 7 lines in your changes missing coverage. Please review.

Project coverage is 72.51%. Comparing base (218f353) to head (a431b14).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
.../indices/pollingingest/IngestionErrorStrategy.java	66.66%	3 Missing and 1 partial ⚠️
...pensearch/plugin/kafka/KafkaPartitionConsumer.java	0.00%	1 Missing ⚠️
...g/opensearch/cluster/metadata/IngestionSource.java	83.33%	0 Missing and 1 partial ⚠️
...ndices/pollingingest/MessageProcessorRunnable.java	88.88%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17427      +/-   ##
============================================
+ Coverage     72.46%   72.51%   +0.04%     
- Complexity    65707    65739      +32     
============================================
  Files          5307     5310       +3     
  Lines        304774   304825      +51     
  Branches      44193    44196       +3     
============================================
+ Hits         220850   221033     +183     
+ Misses        65898    65766     -132     
  Partials      18026    18026              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[varunbharadwaj]

Looks good @varunbharadwaj. Can you add a changelog entry in CHANGELOG-3.0.md and also rebase this with the latest on main?

Sure, updated. Please take a look.

[github-actions]

✅ Gradle check result for a431b14: SUCCESS

[yupeng9]

LGTM