Generate a `versions.toml` with the SDK #1311

jdisanti · 2022-04-09T00:12:50Z

Motivation and Context

This is prerequisite work for tracking SDK crate versions independently, as described in the RFC. This PR:

Introduces a crate-hasher tool that builds against Cargo and uses the same code paths that find files for the cargo package subcommand, and then SHA-256 hashes those files in a deterministic way
Introduces a generate-version-manifest subcommand in the publisher tool which examines the generated SDK and runs the crate-hasher against it in order to produce a versions.toml file
Incorporates the new generate-version-manifest subcommand into the Gradle aws:sdk:assemble task so that the generated SDK now has a versions.toml manifest

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

rcoh

as discussed offline:

simplify file listing logic

Otherwise, seems good!

rcoh · 2022-04-11T16:38:56Z

.github/workflows/ci.yml

        path: smithy-rs-base-image
        retention-days: 1

+  # Run the PR bot after acquiring the build Docker image is completed since it will use it.


very hard to parse this sentence

…rsions-toml

github-actions · 2022-04-11T22:27:36Z

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	-2.61%	79582.57	81714.88
Total requests	-2.56%	7168017	7356319
Total errors	NaN%	0	0
Total successes	-2.56%	7168017	7356319
Average latency ms	2.63%	0.78	0.76
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	-14.13%	24.18	28.16
Stdev latency ms	-8.22%	1.34	1.46
Transfer Mb	-2.56%	745.12	764.69
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

github-actions · 2022-04-11T23:58:00Z

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	-4.51%	40028.74	41919.67
Total requests	-4.48%	3606204	3775439
Total errors	NaN%	0	0
Total successes	-4.48%	3606204	3775439
Average latency ms	2.38%	0.86	0.84
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	-11.62%	16.58	18.76
Stdev latency ms	-10.53%	0.68	0.76
Transfer Mb	-4.48%	374.87	392.46
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

github-actions · 2022-04-12T00:47:02Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	-2.23%	36368.29	37195.92
Total requests	-2.21%	3275390	3349403
Total errors	NaN%	0	0
Total successes	-2.21%	3275390	3349403
Average latency ms	2.22%	0.92	0.9
Minimum latency ms	-33.33%	0.02	0.03
Maximum latency ms	-13.44%	20.74	23.96
Stdev latency ms	3.17%	0.65	0.63
Transfer Mb	-2.21%	340.48	348.17
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

github-actions · 2022-04-12T02:38:06Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	3.32%	74597.79	72199.62
Total requests	3.31%	6717024	6501811
Total errors	NaN%	0	0
Total successes	3.31%	6717024	6501811
Average latency ms	36.25%	1.09	0.8
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	-0.20%	24.5	24.55
Stdev latency ms	41.67%	1.87	1.32
Transfer Mb	3.31%	698.24	675.87
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

rcoh · 2022-04-12T17:34:17Z

.github/actions/docker-build/action.yml

  action:
    description: What action to run in the Docker build
    required: true
+  action-arguments:


maybe just arguments? up to you

rcoh · 2022-04-12T17:37:53Z

aws/sdk/build.gradle.kts

+        "--smithy-build",
+        outputDir.resolve("../../smithy-build.json").normalize().absolutePath,
+        "--examples-revision",
+        properties.get("aws.sdk.examples.revision") ?: "missing"


will this be confusing when it happens? should we hard fail if this is missing?

It's definitely not ideal, but the alternative means not being able to run ./gradlew aws:sdk:assemble locally without checking out a copy of aws-doc-sdk-examples

rcoh · 2022-04-12T17:39:50Z

tools/Dockerfile

    rustup component add rustfmt; \
    rustup component add clippy; \
-    rustup install ${rust_nightly_version}; \
+    rustup toolchain install ${rust_nightly_version} --component clippy; \


side note: we might want to be explicit about the toolchains we install. I've run into some surprises from time to time relying on the defaults (and this might also install components we don't need)

https://rust-lang.github.io/rustup/overrides.html#the-toolchain-file may also be handy

I think the --profile minimal on rustup-init is making sure it won't install any undesired components.

rcoh · 2022-04-12T17:43:27Z

tools/ci-build/scripts/generate-aws-sdk

 set -eux

 echo -e "${C_YELLOW}Taking examples from 'awsdocs/aws-doc-sdk-examples'...${C_RESET}"
+examples_revision=$(cd aws-doc-sdk-examples; git rev-parse HEAD)


shellcheck would probably want quotes here?

Surprisingly not in this case...

rcoh · 2022-04-12T17:48:15Z

tools/publisher/src/subcommand/generate_version_manifest.rs

+        }
+
+        let mut model_hash = None;
+        if package.handle.name.starts_with("aws-sdk-") {


how is this different than PackageCategory::AwsSdk?

It isn't. Not sure why I did it that way. Will switch to using the category.

…rsions-toml

github-actions · 2022-04-12T21:36:21Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	-0.32%	79709.79	79968.71
Total requests	-0.32%	7177449	7200797
Total errors	NaN%	0	0
Total successes	-0.32%	7177449	7200797
Average latency ms	0.81%	1.24	1.23
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	20.21%	28.91	24.05
Stdev latency ms	0.00%	2.16	2.16
Transfer Mb	-0.32%	746.1	748.53
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

github-actions · 2022-04-12T22:29:39Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test

A new doc preview is ready to view.

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Measurement	Deviation	Current	Old
Requests/sec	-5.28%	41895.1	44230.73
Total requests	-5.26%	3772698	3982146
Total errors	NaN%	0	0
Total successes	-5.26%	3772698	3982146
Average latency ms	6.58%	0.81	0.76
Minimum latency ms	0.00%	0.02	0.02
Maximum latency ms	0.54%	16.79	16.7
Stdev latency ms	6.90%	0.62	0.58
Transfer Mb	-5.26%	392.17	413.95
Connect errors	NaN%	0	0
Read errors	NaN%	0	0
Write errors	NaN%	0	0
Status errors (not 2xx/3xx)	NaN%	0	0
Timeout errors	NaN%	0	0

jdisanti added 3 commits April 8, 2022 17:07

Implement crate-hasher utility to hash crate source files

abe4b69

Add generate-version-manifest subcommand to publisher tool

e2730db

Incorporate generate-version-manifest into the build system

2e538b0

jdisanti requested a review from a team as a code owner April 9, 2022 00:12

jdisanti added 3 commits April 8, 2022 17:42

Start making the PR bot run through the Docker build image

77c70cd

Use nightly clippy in tools CI when necessary

d1ff360

Fix path

9c26677

rcoh reviewed Apr 11, 2022

View reviewed changes

jdisanti added 9 commits April 11, 2022 10:32

Fix transient failure when acquiring the build image

b84da02

Fix build image Dockerfile

fe2adc2

Simplify crate-hasher tool

b84cc10

Improve comment

f1f16ca

Merge remote-tracking branch 'upstream/main' into jdisanti-initial-ve…

a1f646f

…rsions-toml

Attempt to fix the PR diff bot

c1f60df

Remove compilerVersion option from RequireRustBuildTool

d1f0a05

Fix codegen-diff-revisions.py

db4dd47

Merge remote-tracking branch 'upstream/main' into jdisanti-initial-ve…

a18ad1e

…rsions-toml

rcoh approved these changes Apr 11, 2022

View reviewed changes

jdisanti added 4 commits April 11, 2022 13:43

Attempt to fix PR bot credentials

b51c20b

Remove -S arg for env in AL2 that doesn't support it

ffc2c6f

Fix action arguments

9304f10

Include examples repo revision in versions.toml

d882dad

jdisanti added 2 commits April 11, 2022 15:44

Attempt to fix codegen diff bot message

b6b51ad

Fix clap argument

4a643e1

jdisanti added 3 commits April 11, 2022 17:06

Fix bot message artifact location

1f486af

Fix nightly clippy lint in cargo-api-linter

2d14435

Track examples revision in sync tool

9173cca

jdisanti added 2 commits April 11, 2022 18:31

Fix missing test files

9171904

Make it easier to run Docker CI locally

67e0ada

rcoh approved these changes Apr 12, 2022

View reviewed changes

jdisanti added 2 commits April 12, 2022 13:59

Incorporate feedback

f71a6c0

Merge remote-tracking branch 'upstream/main' into jdisanti-initial-ve…

05ba11a

…rsions-toml

Attempt to fix codegen diff link

c287585

jdisanti merged commit 7245df1 into main Apr 12, 2022

jdisanti deleted the jdisanti-initial-versions-toml branch April 12, 2022 23:55

Generate a versions.toml with the SDK #1311

Generate a versions.toml with the SDK #1311

Uh oh!

Conversation

jdisanti commented Apr 9, 2022

Motivation and Context

Uh oh!

rcoh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 11, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Uh oh!

github-actions bot commented Apr 11, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Uh oh!

github-actions bot commented Apr 12, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Uh oh!

github-actions bot commented Apr 12, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 12, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Uh oh!

github-actions bot commented Apr 12, 2022

Rust Wrk benchmark report:

Duration: 90 sec, Connections: 32, Threads: 2

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Generate a `versions.toml` with the SDK #1311

Generate a `versions.toml` with the SDK #1311