release-23.2: sql: add probabilistic transaction tracing

[dhartunian]

Backport 1/1 commits from #148542.

/cc @cockroachdb/release

---

Previously, tracing transactions in CRDB could be enabled using a single cluster setting: sql.trace.txn.enable_threshold, which, if it was non-zero, would enable tracing on all transactions and only keep and emit the traces for transactions which exceeded the execution time threshold. This has proven to be difficult to use as it reuquires paying tracing cost on the entire workload.

This change modifies the transaction tracing capabilities of CRDB to incorporate a sample rate. A new cluster setting is introduced, named sql.trace.txn.sample_rate which is zero by default. When set to a positive number between 0 and 1, this will set the probability of a transaction having tracing enabled.

For tracing to emit a trace for a transaction, both settings will now need to be set and will need to both trigger for successful output to the SQL_EXEC logging channel. Transactions first pass through the sample rate filter which governs whether a trace is captured, then they will be evaluated through the enable_threshold which governs whether the trace is emitted to the logs.

Epic: None
Resolves: CRDB-51662

Release note (ops change, sql change): In order to selectively capture traces for transactions running in an active workload without haing to capture them via statement diagnostic bundles, customers can now use the sql.trace.txn.sample_rate cluster setting to enable tracing for a fraction of their workload. The sql.trace.txn.enable_threshold will still need to be set to a positive value to provide a filter for how slow a transaction needs to be after being sampled to merit emitting a trace. Traces are emitted to the SQL_EXEC logging channel.

[blathers-crl]

Thanks for opening a backport.

Before merging, please confirm that it falls into one of the following categories (select one):

• Non-production code changes. Includes test-only changes, build system changes, etc.
• Fixes for serious issues. Defined in the policy as correctness, stability, or security issues, data corruption/loss, significant performance regressions, breaking working and widely used functionality, or an inability to detect and debug production issues.
• Other approved changes. These changes must be gated behind a disabled-by-default feature flag unless there is a strong justification not to.

Add a brief release justification to the PR description explaining your selection.

Also, confirm that the change does not break backward compatibility and complies with all aspects of the backport policy.

All backports must be reviewed by the TL and EM for the owning area.

[cockroach-teamcity]

This change is 

[blathers-crl]

❌ PR #149497 does not comply with backport policy

Confidence: high
Explanation: The pull request does not meet the backport policy for CockroachDB regarding critical bug fixes or feature flags. The changes involve the introduction of a new cluster setting (sql.trace.txn.sample_rate) to control transaction tracing based on a sample rate, which is more akin to a new feature enhancement than a critical bug fix. Additionally, this new feature is not gated behind a feature flag that is disabled by default; the setting itself influences operational dynamics but does not strictly adhere to the 'feature flag' mechanism as required under the policy. There is no 'Release justification: .*' line in the PR body that might exempt it under the policy for exceptions. Furthermore, the changes affect production files like pkg/sql/conn_executor_exec.go, pkg/sql/conn_fsm.go, pkg/sql/exec_util.go, pkg/sql/txn_state.go, which are critical to the operation of the database.
Recommendation: Gate the PR behind a feature flag or reconsider the necessity of this backport to the release branch, as it introduces functionality adjustments rather than addressing a critical system flaw.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[dhartunian]

These changes are included in #149607