Callgrind benchmarks

[kazimuth]

Description of Changes

At long last.

[kazimuth]

callgrind please

[cloutiertyler]

I feel under qualified to review this PR, but I don't see any red flags.

[kazimuth]

@cloutiertyler , here's example output:

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module),
and the underlying SpacetimeDB data storage engine (stdb_raw).

Of these, sqlite is the least noisy, stdb_raw is intermediate, and stdb_module is the noisiest. This is due to uncontrollable nondeterminism,
such as thread ordering and filesystem interactions.

Note: there are currently a minimal number of stdb_module benchmarks, pending a fix for an unfortunate interaction between callgrind and sled.
Their benchmark numbers are correct, just expensive to collect.

Legend:

• preload: number of rows pre-loaded into the database.
• count: number of rows touched by a transaction.
• index types:
  • unique: a single, unique index on the id column.
  • non_unique: non-unique, no indexes.
  • btrees: non-unique, separate btree index on every column. (Not a multi-column index!)
• schemas:
  • person(id: u32, name: String, age: u64) is a small row containing a string.
  • location(id: u32, x: u64, y: u64) is a small row containing some ints.

filter

db	data_type	index_strategy	count	preload	table_name	new instructions	old instructions	delta instructions
sqlite	u64	non_unique	1	100	person	43161	43161	+0.00%
sqlite	u64	non_unique	4	100	person	44511	44511	+0.00%
sqlite	u64	non_unique	16	100	person	49919	49919	+0.00%
sqlite	u64	non_unique	64	100	person	71571	71571	+0.00%
sqlite	u64	btrees	1	100	person	10657	10657	+0.00%
sqlite	u64	btrees	4	100	person	14246	14246	+0.00%
sqlite	u64	btrees	16	100	person	28851	28851	+0.00%
sqlite	u64	btrees	64	100	person	86847	86847	+0.00%
sqlite	string	btrees	1	100	person	11378	11378	+0.00%
sqlite	string	btrees	4	100	person	15218	15218	+0.00%
sqlite	string	btrees	16	100	person	30499	30499	+0.00%
sqlite	string	btrees	64	100	person	91647	91647	+0.00%
stdb_raw	u64	non_unique	1	100	person	154548	154548	+0.00%
stdb_raw	u64	non_unique	4	100	person	156822	156822	+0.00%
stdb_raw	u64	non_unique	16	100	person	165918	165918	+0.00%
stdb_raw	u64	non_unique	64	100	person	202216	202216	+0.00%
stdb_raw	u64	btrees	1	100	person	18950	18950	+0.00%
stdb_raw	u64	btrees	4	100	person	26175	26175	+0.00%
stdb_raw	u64	btrees	16	100	person	55600	55600	+0.00%
stdb_raw	u64	btrees	64	100	person	170862	170862	+0.00%
stdb_raw	string	non_unique	1	100	person	176534	176534	+0.00%
stdb_raw	string	non_unique	4	100	person	178813	178813	+0.00%
stdb_raw	string	non_unique	16	100	person	188079	188079	+0.00%
stdb_raw	string	non_unique	64	100	person	224695	224695	+0.00%
stdb_raw	string	btrees	1	100	person	20014	20014	+0.00%
stdb_raw	string	btrees	4	100	person	27658	27657	+0.00%
stdb_raw	string	btrees	16	100	person	56274	56279	-0.01%
stdb_raw	string	btrees	64	100	person	171752	171713	+0.02%
stdb_module	string	btrees	64	100	person	321157	326107	-1.52%

insert bulk

db	index_strategy	count	preload	table_name	new instructions	old instructions	delta instructions
sqlite	unique	1	50	person	16817	16817	+0.00%
sqlite	unique	25	50	person	92152	92152	+0.00%
sqlite	unique	50	50	person	171233	171233	+0.00%
sqlite	unique	100	50	person	329852	329852	+0.00%
stdb_raw	unique	1	50	person	58427	58410	+0.03%
stdb_raw	unique	25	50	person	1308195	1308417	-0.02%
stdb_raw	unique	50	50	person	2490008	2492014	-0.08%
stdb_raw	unique	100	50	person	4966649	4966545	+0.00%
stdb_module	unique	25	50	person	1634353	1642868	-0.52%

iterate

db	index_strategy	count	table_name	new instructions	old instructions	delta instructions
sqlite	unique	1	person	8542	8542	+0.00%
sqlite	unique	25	person	24338	24341	-0.01%
sqlite	unique	50	person	40802	40802	+0.00%
sqlite	unique	100	person	73725	73725	+0.00%
stdb_raw	unique	1	person	12794	12794	+0.00%
stdb_raw	unique	25	person	39026	39026	+0.00%
stdb_raw	unique	50	person	66365	66365	+0.00%
stdb_raw	unique	100	person	121029	121029	+0.00%
stdb_module	unique	25	person	114683	114049	+0.56%

serialize_product_value

count	format	new instructions	old instructions	delta instructions
50	json	29579	29579	+0.00%
50	bsatn	15739	15739	+0.00%
100	json	57844	57844	+0.00%
100	bsatn	28306	28306	+0.00%

empty transaction

db	new instructions	old instructions	delta instructions
sqlite	3657	3657	+0.00%
stdb_raw	7621	7621	+0.00%
stdb_module	31867	31641	+0.71%

[kazimuth]

The delta there is between two runs of the last commit on this branch, fwiw, it's just to show the formatting. You can also see that we still have about 1% variance in the module benchmarks, but it's a lot lower.

I could add other metrics e.g. estimated cycles / cache misses there, but they're a lot higher variance.

[cloutiertyler]

I'd be particularly interested in cache misses even if they're higher variance, if only because optimizing for instructions could incentivize people to optimize that at the expense of cache performance.

[kulakowski]

This overall looks good to me. It's a pretty big PR so I defer to @joshua-spacetime about how to land it. I'll say how I broke this apart in my head and how I feel about each part:

• the CI machinery: seems fine, seems easy to iterate on and land separately, I'd like someone more knowledable than me to say whether it's risky: like, could this interfere with existing CI actions?
• the docker machinery: seems ok, seems linux specific and needs some guard rails for non-linux systems?
• the callgrind machinery: the forks are unfortunate, but I get it. I'd personally want to see some mechanism for knowing we keep them up to date. but I think that about all our dependencies in general, so. I'm personally fine with the flag magic, but someone else should probably agree with me about that.
• the benchmarks themeselves. seems ok but I'd want @joshua-spacetime to be ok with how forward progress there is tracked.
• the serialization machinery: also seems ok for the reason I mentioned, it seems easy for someone else to add on to, and I think incentivizing creation of more benchmarks is what I would optimize for.
• the reporting machinery: again it seems fine, but it's a big chunk of code and I wonder if it can be split out?

I added some places I'd add tracking issues. But really I just want James and Joshua to be on the same page for tracking that work, I don't care if it's issues or something else.

[kazimuth]

I've slimmed this down a bit by ripping out everything related to generating reports (except the script to pack json files for uploading), that now lives in the benchmarks-viewer application: https://github.com/clockworklabs/benchmarks-viewer

None of the stuff here should interfere with other CI.

[cloutiertyler]

These are still failing it would appear.

[kazimuth]

@cloutiertyler this is ready to merge

[kazimuth]

Ah, didn't see Tyler wanted John to look over this, whoops.