S3 Benchmarks - support java-based multipart client #4288

L-Applin · 2023-08-10T15:41:43Z

Changes required to run benchmarks using the MultipartS3AsyncClient (Upload and Copy)

Motivation and Context

Performance benchmarks were run as part of the release of the MultipartS3AsyncClient

Modifications

New benchmarks classes created for the Java based Multipart S3 Async Client
Add a new benchmark target java as a --version=java option
Add new options
- --connAcqTimeoutInSec the connection acquire timout. 30GB benchmark were timing out, increasing this value prevented it.
- --crtHttp which force using the CRT HTTP Cient for the java based benchmark if --crtHttp=true
- --maxConcurrency to compare the influence of this parameter on the http clients.

Testing

none - benchmark were run, in an EC2 instance

Screenshots (if appropriate)

none

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

I have read the CONTRIBUTING document
Local run of mvn install succeeds
My code follows the code style of this project
My change requires a change to the Javadoc documentation
I have updated the Javadoc documentation accordingly
I have added tests to cover my changes
All new and existing tests passed
I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

I confirm that this pull request can be released under the Apache 2 license

* Implement multipart upload in Java-based S3 async client Co-authored-by: Matthew Miller <[email protected]>

* Iterate SdkFields to convert requests * Fix flaky test * Rename convertion utils class

…4188) * Create split method in AsyncRequestBody to return SplittingPublisher * Fix Javadoc and build

* Handle null response metadata * Fix build

* Support uploading with unknown content length * Refactoring

fix merge conflicts

* Multipart API fix merge conflicts * getObject(...) throw UnsupportedOperationException * Use user agent for all requests in MultipartS3Client * MultipartS3AsyncClient javadoc + API_NAME private * use `maximumMemoryUsageInBytes` * fix problem with UserAgent, cleanup * move contextParam keys to S3AsyncClientDecorator * javadoc * more javadoc * Use 4x part size as default apiCallBufferSize

…ltipart

* guard against re-subscription in SplittingPublisher * fix checkstyle * Error msg

…ore the second one (#4260)

…ava-v2 into olapplin/s3-multi-perf-2

…/s3-multi-perf-2

test/s3-benchmarks/.scripts/create_benchmark_files

...-benchmarks/src/main/java/software/amazon/awssdk/s3benchmarks/BaseJavaS3ClientBenchmark.java

...nchmarks/src/main/java/software/amazon/awssdk/s3benchmarks/BaseTransferManagerBenchmark.java

test/s3-benchmarks/.scripts/benchmark

...hmarks/src/main/java/software/amazon/awssdk/s3benchmarks/TransferManagerBenchmarkConfig.java

test/s3-benchmarks/.scripts/create_benchmark_files

...-benchmarks/src/main/java/software/amazon/awssdk/s3benchmarks/BaseJavaS3ClientBenchmark.java

- updated create_benchmark_files script to calculate size

sonarqubecloud · 2023-08-14T19:09:20Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
9 Code Smells

0.0% Coverage
6.1% Duplication

The version of Java (11.0.2) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

zoewangg and others added 30 commits June 16, 2023 15:14

Implement multipart upload in Java-based S3 async client (#4052)

bc80de0

* Implement multipart upload in Java-based S3 async client Co-authored-by: Matthew Miller <[email protected]>

Iterate SdkFields to convert requests (#4177)

910b30f

* Iterate SdkFields to convert requests * Fix flaky test * Rename convertion utils class

Fix null content length in SplittingPublisher (#4173)

d998908

Implement multipart copy in Java-based S3 async client (#4189)

afe5f58

Create split method in AsyncRequestBody to return SplittingPublisher (#…

35e6e4e

…4188) * Create split method in AsyncRequestBody to return SplittingPublisher * Fix Javadoc and build

Merge remote-tracking branch 'origin' into feature/master/s3multipart

8472539

Add more tests with ByteArrayAsyncRequestBody (#4214)

e0b4bfc

Handle null response metadata (#4215)

11c6362

* Handle null response metadata * Fix build

Support streaming with unknown content length (#4226)

3e6e70f

* Support uploading with unknown content length * Refactoring

Multipart API

2e0a841

fix merge conflicts

getObject(...) throw UnsupportedOperationException

19efa8b

Use user agent for all requests in MultipartS3Client

2667219

MultipartS3AsyncClient javadoc + API_NAME private

bcbbf98

use maximumMemoryUsageInBytes

7e82f38

fix problem with UserAgent, cleanup

f197c2e

Create a configuration class for SdkPublisher#split (#4236)

d255b1f

add support for JavaBasedS3Client

915099b

Java based upload

e0b980b

Java copy benchmark

80b9530

Jav based - add apiCallBufferSizeInBytes config from readBufferSizeInMb

242d818

Jav based - checksumAlgo

992f9c2

fix copy benchmark

675aa0d

add support for connection acquire timeout

d91b04a

add support for connection acquire timeout

cca1870

allow to force crt http client

8e63285

allow to force crt http client

98f7974

fix crt http config

2c938c7

Merge remote-tracking branch 'origin/master' into feature/master/s3mu…

32ddd19

…ltipart

Fix test

28c126d

L-Applin and others added 8 commits August 3, 2023 09:04

Guard against re-subscription in SplittingPublisher (#4253)

85a1fd7

* guard against re-subscription in SplittingPublisher * fix checkstyle * Error msg

Fix a race condition where the third upload part request was sent bef…

466e687

…ore the second one (#4260)

maxConcurrency for Javabased with crt http client

43eca15

Merge branch 'feature/master/s3multipart' of github.com:aws/aws-sdk-j…

8794014

…ava-v2 into olapplin/s3-multi-perf-2

fix merge

edd79c7

allow maxConcurrency config with TM and Netty

ef8d290

Merge branch 'master' of github.com:aws/aws-sdk-java-v2 into olapplin…

5ba46a7

…/s3-multi-perf-2

cleanup

1e839e7

L-Applin requested a review from a team as a code owner August 10, 2023 15:44

L-Applin requested a review from millems August 10, 2023 16:19

fix checkstyle

8db6262

millems reviewed Aug 10, 2023

View reviewed changes

pr comment

53ea643

millems reviewed Aug 11, 2023

View reviewed changes

test/s3-benchmarks/.scripts/create_benchmark_files Outdated Show resolved Hide resolved

...-benchmarks/src/main/java/software/amazon/awssdk/s3benchmarks/BaseJavaS3ClientBenchmark.java Show resolved Hide resolved

added version and operation cli arg to README

dd72b7f

- updated create_benchmark_files script to calculate size

millems approved these changes Aug 14, 2023

View reviewed changes

Merge branch 'master' into olapplin/s3-multi-perf-2

589fee8

L-Applin enabled auto-merge (squash) August 14, 2023 17:26

L-Applin merged commit 0558677 into master Aug 14, 2023

L-Applin deleted the olapplin/s3-multi-perf-2 branch August 15, 2025 18:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

S3 Benchmarks - support java-based multipart client #4288

S3 Benchmarks - support java-based multipart client #4288

Uh oh!

L-Applin commented Aug 10, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Aug 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

S3 Benchmarks - support java-based multipart client #4288

S3 Benchmarks - support java-based multipart client #4288

Uh oh!

Conversation

L-Applin commented Aug 10, 2023

Motivation and Context

Modifications

Testing

Screenshots (if appropriate)

Types of changes

Checklist

License

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Aug 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants