Skip to content

Fix benchmark actions #715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jan 11, 2024
Merged

Fix benchmark actions #715

merged 4 commits into from
Jan 11, 2024

Conversation

@kazimuth
Copy link
Contributor

@kazimuth kazimuth commented Jan 10, 2024 •
edited
Loading

Description of Changes

This fixes two things:

Let other people use the actions runner after we're done with it

Specifically, we chmod the git repo directory to a+rw after we touch it in the callgrind benchmarks action.
This is needed because the callgrind benchmarks run as root in a docker container. Github Actions tries to reuse the resulting directory (and I guess stores it on the host?) afterwards, but now the git repo is owned by root and can't be modified, which causes other benchmarks to break.
This issue was pointed out by @coolreader18.

(Note: we can't just chown the git repo, because we're inside a docker image, so who would we chown to?)

Remove residual references to moved code in the benchmarks & callgrind benchmarks actions.

I pulled out the markdown-report[-callgrind] commands from the benches/summarize bin in my last callgrind PR -- they've moved to another repo -- but there were residual references to them in these ymls, despite the resulting reports not being used. I'm not sure why that didn't cause errors for that PR, but the residual references are gone now.

This issue was pointed out by @cloutiertyler.

@kazimuth kazimuth requested a review from coolreader18 January 10, 2024 20:08
@kazimuth
Copy link
Contributor Author

kazimuth commented Jan 10, 2024

callgrind please

@github-actions
Copy link
Contributor

github-actions bot commented Jan 10, 2024 •
edited
Loading

Benchmarking failed. Please check the workflow run for details.

@kazimuth
Copy link
Contributor Author

kazimuth commented Jan 11, 2024

callgrind please

@kazimuth
Copy link
Contributor Author

kazimuth commented Jan 11, 2024

benchmarks please

@github-actions
Copy link
Contributor

github-actions bot commented Jan 11, 2024 •
edited
Loading

Benchmarking failed. Please check the workflow run for details.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 11, 2024 •
edited
Loading

Benchmarking failed. Please check the workflow run for details.

@kazimuth
Copy link
Contributor Author

kazimuth commented Jan 11, 2024 •
edited
Loading

Wait, it's using the master actions yaml so it's not seeing the changes. I'll try running manually

@github-actions
Copy link
Contributor

github-actions bot commented Jan 11, 2024 •
edited
Loading

Benchmarking failed. Please check the workflow run for details.

@kazimuth
Copy link
Contributor Author

kazimuth commented Jan 11, 2024 •
edited
Loading

Here's the manual runs with the ymls from this branch:
Callgrind: https://github.com/clockworklabs/SpacetimeDB/actions/runs/7493027759
Regular: https://github.com/clockworklabs/SpacetimeDB/actions/runs/7492802811

@kazimuth kazimuth changed the title Clean up after callgrind action Fix benchmark actions Jan 11, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jan 11, 2024 •
edited
Loading

Benchmark results

Benchmark Report

Legend:

  • load: number of rows pre-loaded into the database
  • count: number of rows touched by the transaction
  • index types:
    • unique: a single index on the id column
    • non_unique: no indexes
    • multi_index: non-unique index on every column
  • schemas:
    • person(id: u32, name: String, age: u64)
    • location(id: u32, x: u64, y: u64)

All throughputs are single-threaded.

Empty transaction

db on disk new latency old latency new throughput old throughput
stdb_module 💿 17.7±0.83µs - - -
stdb_module 🧠 17.2±0.40µs - - -
stdb_raw 💿 939.8±1.63ns - - -
stdb_raw 🧠 895.4±1.28ns - - -

Single-row insertions

db on disk schema index type load new latency old latency new throughput old throughput
stdb_module 💿 location multi_index 0 55.5±4.18µs - 17.6 Ktx/sec -
stdb_module 💿 location multi_index 1000 321.6±25.52µs - 3.0 Ktx/sec -
stdb_module 💿 location non_unique 0 43.4±4.03µs - 22.5 Ktx/sec -
stdb_module 💿 location non_unique 1000 211.6±52.18µs - 4.6 Ktx/sec -
stdb_module 💿 location unique 0 48.3±4.70µs - 20.2 Ktx/sec -
stdb_module 💿 location unique 1000 150.1±7.76µs - 6.5 Ktx/sec -
stdb_module 💿 person multi_index 0 66.8±4.30µs - 14.6 Ktx/sec -
stdb_module 💿 person multi_index 1000 340.3±11.47µs - 2.9 Ktx/sec -
stdb_module 💿 person non_unique 0 51.9±11.81µs - 18.8 Ktx/sec -
stdb_module 💿 person non_unique 1000 350.1±21.44µs - 2.8 Ktx/sec -
stdb_module 💿 person unique 0 55.3±6.12µs - 17.7 Ktx/sec -
stdb_module 💿 person unique 1000 322.9±14.49µs - 3.0 Ktx/sec -
stdb_module 🧠 location multi_index 0 39.9±4.46µs - 24.5 Ktx/sec -
stdb_module 🧠 location multi_index 1000 145.6±34.35µs - 6.7 Ktx/sec -
stdb_module 🧠 location non_unique 0 33.0±2.66µs - 29.6 Ktx/sec -
stdb_module 🧠 location non_unique 1000 261.9±8.57µs - 3.7 Ktx/sec -
stdb_module 🧠 location unique 0 36.4±4.14µs - 26.8 Ktx/sec -
stdb_module 🧠 location unique 1000 265.3±4.99µs - 3.7 Ktx/sec -
stdb_module 🧠 person multi_index 0 50.4±6.95µs - 19.4 Ktx/sec -
stdb_module 🧠 person multi_index 1000 160.5±26.32µs - 6.1 Ktx/sec -
stdb_module 🧠 person non_unique 0 34.7±3.04µs - 28.1 Ktx/sec -
stdb_module 🧠 person non_unique 1000 175.5±6.73µs - 5.6 Ktx/sec -
stdb_module 🧠 person unique 0 39.8±4.50µs - 24.5 Ktx/sec -
stdb_module 🧠 person unique 1000 124.8±11.18µs - 7.8 Ktx/sec -
stdb_raw 💿 location multi_index 0 7.2±0.03µs - 134.7 Ktx/sec -
stdb_raw 💿 location multi_index 1000 36.3±262.91µs - 26.9 Ktx/sec -
stdb_raw 💿 location non_unique 0 4.7±0.01µs - 205.8 Ktx/sec -
stdb_raw 💿 location non_unique 1000 6.2±0.15µs - 156.9 Ktx/sec -
stdb_raw 💿 location unique 0 6.2±0.01µs - 157.1 Ktx/sec -
stdb_raw 💿 location unique 1000 8.5±0.15µs - 114.4 Ktx/sec -
stdb_raw 💿 person multi_index 0 11.5±0.68µs - 84.7 Ktx/sec -
stdb_raw 💿 person multi_index 1000 62.8±479.07µs - 15.5 Ktx/sec -
stdb_raw 💿 person non_unique 0 5.9±0.01µs - 166.0 Ktx/sec -
stdb_raw 💿 person non_unique 1000 7.6±0.30µs - 127.8 Ktx/sec -
stdb_raw 💿 person unique 0 8.4±0.03µs - 116.7 Ktx/sec -
stdb_raw 💿 person unique 1000 38.8±277.84µs - 25.2 Ktx/sec -
stdb_raw 🧠 location multi_index 0 4.7±0.01µs - 205.9 Ktx/sec -
stdb_raw 🧠 location multi_index 1000 6.4±0.07µs - 151.4 Ktx/sec -
stdb_raw 🧠 location non_unique 0 2.3±0.00µs - 420.0 Ktx/sec -
stdb_raw 🧠 location non_unique 1000 2.9±0.02µs - 336.5 Ktx/sec -
stdb_raw 🧠 location unique 0 3.6±0.01µs - 268.3 Ktx/sec -
stdb_raw 🧠 location unique 1000 4.9±0.03µs - 198.6 Ktx/sec -
stdb_raw 🧠 person multi_index 0 8.3±0.02µs - 117.1 Ktx/sec -
stdb_raw 🧠 person multi_index 1000 10.7±0.07µs - 91.3 Ktx/sec -
stdb_raw 🧠 person non_unique 0 2.9±0.00µs - 337.0 Ktx/sec -
stdb_raw 🧠 person non_unique 1000 3.6±0.04µs - 269.5 Ktx/sec -
stdb_raw 🧠 person unique 0 5.2±0.01µs - 186.6 Ktx/sec -
stdb_raw 🧠 person unique 1000 6.8±0.04µs - 144.5 Ktx/sec -

Multi-row insertions

db on disk schema index type load count new latency old latency new throughput old throughput
stdb_module 💿 location multi_index 0 100 890.1±2.14µs - 1123 tx/sec -
stdb_module 💿 location multi_index 1000 100 1133.2±8.56µs - 882 tx/sec -
stdb_module 💿 location non_unique 0 100 469.1±1.75µs - 2.1 Ktx/sec -
stdb_module 💿 location non_unique 1000 100 661.9±5.30µs - 1510 tx/sec -
stdb_module 💿 location unique 0 100 669.2±9.09µs - 1494 tx/sec -
stdb_module 💿 location unique 1000 100 916.2±15.98µs - 1091 tx/sec -
stdb_module 💿 person multi_index 0 100 991.0±1.71µs - 1009 tx/sec -
stdb_module 💿 person multi_index 1000 100 1288.4±5.51µs - 776 tx/sec -
stdb_module 💿 person non_unique 0 100 718.1±107.09µs - 1392 tx/sec -
stdb_module 💿 person non_unique 1000 100 1002.1±69.75µs - 997 tx/sec -
stdb_module 💿 person unique 0 100 758.9±33.19µs - 1317 tx/sec -
stdb_module 💿 person unique 1000 100 750.7±6.75µs - 1332 tx/sec -
stdb_module 🧠 location multi_index 0 100 663.3±154.73µs - 1507 tx/sec -
stdb_module 🧠 location multi_index 1000 100 631.0±33.41µs - 1584 tx/sec -
stdb_module 🧠 location non_unique 0 100 382.3±8.71µs - 2.6 Ktx/sec -
stdb_module 🧠 location non_unique 1000 100 511.1±31.32µs - 1956 tx/sec -
stdb_module 🧠 location unique 0 100 446.4±12.76µs - 2.2 Ktx/sec -
stdb_module 🧠 location unique 1000 100 658.2±55.26µs - 1519 tx/sec -
stdb_module 🧠 person multi_index 0 100 945.1±1.86µs - 1058 tx/sec -
stdb_module 🧠 person multi_index 1000 100 1011.3±12.85µs - 988 tx/sec -
stdb_module 🧠 person non_unique 0 100 501.8±44.35µs - 1992 tx/sec -
stdb_module 🧠 person non_unique 1000 100 673.0±19.03µs - 1485 tx/sec -
stdb_module 🧠 person unique 0 100 600.6±10.21µs - 1664 tx/sec -
stdb_module 🧠 person unique 1000 100 706.7±27.02µs - 1415 tx/sec -
stdb_raw 💿 location multi_index 0 100 346.9±9.09µs - 2.8 Ktx/sec -
stdb_raw 💿 location multi_index 1000 100 369.5±1.85µs - 2.6 Ktx/sec -
stdb_raw 💿 location non_unique 0 100 117.4±0.17µs - 8.3 Ktx/sec -
stdb_raw 💿 location non_unique 1000 100 127.2±82.83µs - 7.7 Ktx/sec -
stdb_raw 💿 location unique 0 100 242.3±0.53µs - 4.0 Ktx/sec -
stdb_raw 💿 location unique 1000 100 279.7±189.27µs - 3.5 Ktx/sec -
stdb_raw 💿 person multi_index 0 100 708.7±0.91µs - 1411 tx/sec -
stdb_raw 💿 person multi_index 1000 100 736.5±0.84µs - 1357 tx/sec -
stdb_raw 💿 person non_unique 0 100 217.7±0.22µs - 4.5 Ktx/sec -
stdb_raw 💿 person non_unique 1000 100 239.5±191.12µs - 4.1 Ktx/sec -
stdb_raw 💿 person unique 0 100 431.8±25.18µs - 2.3 Ktx/sec -
stdb_raw 💿 person unique 1000 100 474.3±274.69µs - 2.1 Ktx/sec -
stdb_raw 🧠 location multi_index 0 100 302.3±2.21µs - 3.2 Ktx/sec -
stdb_raw 🧠 location multi_index 1000 100 327.5±0.70µs - 3.0 Ktx/sec -
stdb_raw 🧠 location non_unique 0 100 75.0±0.15µs - 13.0 Ktx/sec -
stdb_raw 🧠 location non_unique 1000 100 76.5±0.11µs - 12.8 Ktx/sec -
stdb_raw 🧠 location unique 0 100 198.7±0.24µs - 4.9 Ktx/sec -
stdb_raw 🧠 location unique 1000 100 217.9±0.25µs - 4.5 Ktx/sec -
stdb_raw 🧠 person multi_index 0 100 616.6±0.90µs - 1621 tx/sec -
stdb_raw 🧠 person multi_index 1000 100 645.6±0.43µs - 1548 tx/sec -
stdb_raw 🧠 person non_unique 0 100 127.4±0.12µs - 7.7 Ktx/sec -
stdb_raw 🧠 person non_unique 1000 100 129.9±0.28µs - 7.5 Ktx/sec -
stdb_raw 🧠 person unique 0 100 337.2±0.43µs - 2.9 Ktx/sec -
stdb_raw 🧠 person unique 1000 100 354.6±0.41µs - 2.8 Ktx/sec -

Full table iterate

db on disk schema index type new latency old latency new throughput old throughput
stdb_module 💿 location unique 53.7±7.55µs - 18.2 Ktx/sec -
stdb_module 💿 person unique 66.5±7.76µs - 14.7 Ktx/sec -
stdb_module 🧠 location unique 50.2±4.90µs - 19.5 Ktx/sec -
stdb_module 🧠 person unique 64.2±11.28µs - 15.2 Ktx/sec -
stdb_raw 💿 location unique 9.1±0.06µs - 107.6 Ktx/sec -
stdb_raw 💿 person unique 9.1±0.02µs - 107.7 Ktx/sec -
stdb_raw 🧠 location unique 9.0±0.01µs - 108.3 Ktx/sec -
stdb_raw 🧠 person unique 9.0±0.02µs - 108.2 Ktx/sec -

Find unique key

db on disk key type load new latency old latency new throughput old throughput
stdb_module 💿 u32 1000 25.6±1.80µs - 38.1 Ktx/sec -
stdb_module 🧠 u32 1000 23.8±1.71µs - 41.1 Ktx/sec -
stdb_raw 💿 u32 1000 2.1±0.01µs - 456.8 Ktx/sec -
stdb_raw 🧠 u32 1000 2.1±0.01µs - 469.8 Ktx/sec -

Filter

db on disk key type index strategy load count new latency old latency new throughput old throughput
stdb_module 💿 string indexed 1000 10 37.0±2.14µs - 26.4 Ktx/sec -
stdb_module 💿 string non_indexed 1000 10 167.0±1.02µs - 5.8 Ktx/sec -
stdb_module 💿 u64 indexed 1000 10 34.0±2.06µs - 28.7 Ktx/sec -
stdb_module 💿 u64 non_indexed 1000 10 140.4±4.80µs - 7.0 Ktx/sec -
stdb_module 🧠 string indexed 1000 10 37.0±2.53µs - 26.4 Ktx/sec -
stdb_module 🧠 string non_indexed 1000 10 166.3±2.19µs - 5.9 Ktx/sec -
stdb_module 🧠 u64 indexed 1000 10 33.1±2.09µs - 29.5 Ktx/sec -
stdb_module 🧠 u64 non_indexed 1000 10 139.0±1.27µs - 7.0 Ktx/sec -
stdb_raw 💿 string indexed 1000 10 4.6±0.01µs - 211.9 Ktx/sec -
stdb_raw 💿 string non_indexed 1000 10 136.3±0.23µs - 7.2 Ktx/sec -
stdb_raw 💿 u64 indexed 1000 10 4.5±0.01µs - 219.0 Ktx/sec -
stdb_raw 💿 u64 non_indexed 1000 10 113.9±0.16µs - 8.6 Ktx/sec -
stdb_raw 🧠 string indexed 1000 10 4.5±0.02µs - 215.2 Ktx/sec -
stdb_raw 🧠 string non_indexed 1000 10 150.8±1.41µs - 6.5 Ktx/sec -
stdb_raw 🧠 u64 indexed 1000 10 4.4±0.02µs - 221.8 Ktx/sec -
stdb_raw 🧠 u64 non_indexed 1000 10 127.5±0.47µs - 7.7 Ktx/sec -

Serialize

schema format count new latency old latency new throughput old throughput
location bsatn 100 1893.0±49.17ns - 50.4 Mtx/sec -
location json 100 3.2±0.04µs - 29.5 Mtx/sec -
location product_value 100 598.8±0.62ns - 159.3 Mtx/sec -
person bsatn 100 2.7±0.03µs - 34.8 Mtx/sec -
person json 100 4.9±0.06µs - 19.6 Mtx/sec -
person product_value 100 668.4±1.45ns - 142.7 Mtx/sec -

Module: invoke with large arguments

arg size new latency old latency new throughput old throughput
64KiB 62.9±5.57µs - - -

Module: print bulk

line count new latency old latency new throughput old throughput
1 23.9±1.52µs - - -
100 200.7±1.80µs - - -
1000 1870.3±65.47µs - - -

Remaining benchmarks

name new latency old latency new throughput old throughput

@github-actions
Copy link
Contributor

github-actions bot commented Jan 11, 2024 •
edited
Loading

Benchmark results # Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module),
and the underlying SpacetimeDB data storage engine (stdb_raw).

Of these, sqlite is the least noisy, stdb_raw is intermediate, and stdb_module is the noisiest. This is due to uncontrollable nondeterminism,
such as thread ordering and filesystem interactions.

Note: there are currently a minimal number of stdb_module benchmarks, pending a fix for an unfortunate interaction between callgrind and sled.
Their benchmark numbers are correct, just expensive to collect.

Legend:

  • preload: number of rows pre-loaded into the database.
  • count: number of rows touched by a transaction.
  • index types:
    • unique: a single, unique index on the id column.
    • non_unique: non-unique, no indexes.
    • btrees: non-unique, separate btree index on every column. (Not a multi-column index!)
  • schemas:
    • person(id: u32, name: String, age: u64) is a small row containing a string.
    • location(id: u32, x: u64, y: u64) is a small row containing some ints.

filter

db data_type index_strategy count preload table_name new instructions old instructions delta instructions
sqlite u64 non_unique 1 100 person 43162 43162 +0.00%
sqlite u64 non_unique 4 100 person 44512 44512 +0.00%
sqlite u64 non_unique 16 100 person 49920 49920 +0.00%
sqlite u64 non_unique 64 100 person 71572 71572 +0.00%
sqlite u64 btrees 1 100 person 10658 10658 +0.00%
sqlite u64 btrees 4 100 person 14247 14247 +0.00%
sqlite u64 btrees 16 100 person 28852 28852 +0.00%
sqlite u64 btrees 64 100 person 86848 86848 +0.00%
sqlite string btrees 1 100 person 11393 11393 +0.00%
sqlite string btrees 4 100 person 15233 15233 +0.00%
sqlite string btrees 16 100 person 30514 30514 +0.00%
sqlite string btrees 64 100 person 91662 91662 +0.00%
stdb_raw u64 non_unique 1 100 person 154857 154931 -0.05%
stdb_raw u64 non_unique 4 100 person 157137 157211 -0.05%
stdb_raw u64 non_unique 16 100 person 166257 166331 -0.04%
stdb_raw u64 non_unique 64 100 person 202651 202725 -0.04%
stdb_raw u64 btrees 1 100 person 18950 19351 -2.07%
stdb_raw u64 btrees 4 100 person 26158 26559 -1.51%
stdb_raw u64 btrees 16 100 person 55513 55914 -0.72%
stdb_raw u64 btrees 64 100 person 171222 170843 +0.22%
stdb_raw string non_unique 1 100 person 176843 176917 -0.04%
stdb_raw string non_unique 4 100 person 179128 179202 -0.04%
stdb_raw string non_unique 16 100 person 188418 188492 -0.04%
stdb_raw string non_unique 64 100 person 225130 225204 -0.03%
stdb_raw string btrees 1 100 person 19997 20398 -1.97%
stdb_raw string btrees 4 100 person 27623 28024 -1.43%
stdb_raw string btrees 16 100 person 56153 56554 -0.71%
stdb_raw string btrees 64 100 person 171315 171673 -0.21%

insert bulk

db index_strategy count preload table_name new instructions old instructions delta instructions
sqlite unique 1 50 person 16782 16782 +0.00%
sqlite unique 25 50 person 90749 90749 +0.00%
sqlite unique 50 50 person 168405 168405 +0.00%
sqlite unique 100 50 person 324174 324174 +0.00%
stdb_raw unique 1 50 person 61331 65944 -7.00%
stdb_raw unique 25 50 person 1296506 1303560 -0.54%
stdb_raw unique 50 50 person 2452911 2485655 -1.32%
stdb_raw unique 100 50 person 4902386 4961166 -1.18%

iterate

db index_strategy count table_name new instructions old instructions delta instructions
sqlite unique 1 person 8565 8565 +0.00%
sqlite unique 25 person 24361 24361 +0.00%
sqlite unique 50 person 40825 40828 -0.01%
sqlite unique 100 person 73748 73748 +0.00%
stdb_raw unique 1 person 12814 12908 -0.73%
stdb_raw unique 25 person 39022 39116 -0.24%
stdb_raw unique 50 person 66360 66430 -0.11%
stdb_raw unique 100 person 120950 121044 -0.08%

serialize_product_value

count format new instructions old instructions delta instructions
50 json 28829 28829 +0.00%
50 bsatn 15989 15989 +0.00%
100 json 56344 56344 +0.00%
100 bsatn 28806 28806 +0.00%

empty transaction

db new instructions old instructions delta instructions
sqlite 3672 3672 +0.00%
stdb_raw 7642 7643 -0.01%
stdb_module 32004 32218 -0.66%

coolreader18
coolreader18 approved these changes Jan 11, 2024
View reviewed changes
Copy link
Collaborator

@coolreader18 coolreader18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@kazimuth kazimuth added this pull request to the merge queue Jan 11, 2024
Merged via the queue into master with commit 11cd68a Jan 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coolreader18 coolreader18 coolreader18 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kazimuth @coolreader18