Skip to content

bench: ensure that benchmarks "work" under multi-tenancy #83461

@yuzefovich

Description

@yuzefovich

As of #76582, we now use multi-tenant test setup in 50% of the tests, which includes the benchmarks. We probably forgot to consider the impact of that change on running the benchmarks, and there are a couple of things we need to address (quoting Raphael):

  • one is the cosmetic issue of the info messages (when the multi-tenant setup is used), causing the benchstat utility to get confused. This can be reliably addressed in multiple ways, by tweaking the messages (e.g. adding a newline character at the beginning).
  • the other is the question of non-determinism in benchmark outputs (TBD whether this non-determinism actually takes place). Here the situation is that we have automation that tracks benchmark regressions over time. This automation is currently not aware of the multi-tenancy randomization. We need to enhance that.

Separately, we need to make peace with the idea that single tenancy is going away. At some point in the future, we will transition all our bench tests to run on a multi-tenant cluster, and we are expecting no performance regression. If we are seeing a performance regression today, this means that there is multi-tenancy optimization work to do that needs to be planned and done by the appropriate teams.

As a quick fix, #83254 disabled the usage of the multi-tenant setup under bench.

Jira issue: CRDB-17081

Epic CRDB-31933

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-multitenancyRelated to multi-tenancyA-testeng-foundationsC-cleanupTech debt, refactors, loose ends, etc. Solution not expected to significantly change behavior.T-testengTestEng Team

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions