Seeing Upload failure while uploading object to S3 for OpenSearch service.

[pranikum]

Describe the bug

We are seeing Issue while uploading translog data to S3 while using aws-crt client. The issue seem to be happening when data is being ingested to opensearch service under heavy load. Not seen in normal loads.

We tried the same ingestion load using Netty client and we don't see any failure.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

We should not be seeing any upload failure.

Current Behavior

We get the below exception.

025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] java.lang.IllegalStateException: The request content has fewer bytes than the specified content-length: 16912416 bytes. [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.io.SdkLengthAwareInputStream.read(SdkLengthAwareInputStream.java:82) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.FilterInputStream.read(FilterInputStream.java:95) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:55) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:92) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:112) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:80) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:925) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [2025-11-07T05:40:34,733][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.lang.Thread.run(Thread.java:1583)

[2025-11-07T05:40:34,742][ERROR][o.o.i.s.RemoteDirectory ][55a662f74df1eebcd05dc39abeb4cb07] Failed to upload blob _7s9.cfs software.amazon.awssdk.core.exception.SdkClientException: Failed to send multipart upload requests. at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130) at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) at org.opensearch.repositories.s3.async.AsyncTransferManager.handleException(AsyncTransferManager.java:333) at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$handleExceptionOrResponse$13(AsyncTransferManager.java:279) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$allOfExceptionForwarded$5(CompletableFutureUtils.java:195) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$anyFail$6(CompletableFutureUtils.java:217) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:56) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:67) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:117) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:925) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: A callback has reported failure. (SDK Attempt Count: 2) at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130) at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168) ... 26 more Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: software.amazon.awssdk.crt.http.HttpException: The connection has closed or is closing. Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: software.amazon.awssdk.crt.http.HttpException: The connection has closed or is closing. Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when acquiring a connection Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: An exception occurred when making the request Caused by: software.amazon.awssdk.crt.http.HttpException: A callback has reported failure. at software.amazon.awssdk.http.crt.internal.response.CrtResponseAdapter.onResponseComplete(CrtResponseAdapter.java:119)

[2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.utils.async.StoringSubscriber$Event.runtimeError(StoringSubscriber.java:181) [2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.utils.async.ByteBufferStoringSubscriber.transferTo(ByteBufferStoringSubscriber.java:114) [2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.http.crt.internal.request.CrtRequestBodyAdapter.sendRequestBody(CrtRequestBodyAdapter.java:43) [2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] Caused by: java.io.IOException: Stream closed [2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:188) [2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:198) [2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:352) [2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420) [2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405) [2025-11-07T05:40:34,747][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.FilterInputStream.read(FilterInputStream.java:119) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.io.SdkLengthAwareInputStream.read(SdkLengthAwareInputStream.java:75) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.io.FilterInputStream.read(FilterInputStream.java:95) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:55) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:92) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:112) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:80) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:925) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [2025-11-07T05:40:34,748][WARN ][stderr ][55a662f74df1eebcd05dc39abeb4cb07] at java.base/java.lang.Thread.run(Thread.java:1583)

Reproduction Steps

Load the data in Opensearch service using multiple threads.

Possible Solution

NA

Additional Information/Context

Related OpenSearch PR: opensearch-project/OpenSearch#18800

aws-crt-java version used

0.35.0

Java version used

JAVA21

Operating System and version

Linux

[DmitriyMusatkin]

Under what kinds of loads is this happening?
Is in test or something we can repro? Any chance to get logs around the error?

[pranikum]

@DmitriyMusatkin Thanks for looking into this.

The issue is happening when we are ingesting bulk data to OpenSearch Service.
Bulk Request Size: 25MB
Number of Threads running bulk Ingesting: 100

We were suspecting it to be related to "mark and reset".

[DmitriyMusatkin]

Can you provide more info on what kinda of requests you are doing and what do you mean by mark and reset?

I.e. how many parallel requests are you queueing up? is it ranged requests?