server: default zone config is not propagated into virtual clusters

[knz]

Found while working on #110001.

Describe the problem

Found while investigating a test failure that was looking at the default zone config (the DefaultZoneConfig field in the system config provider).

The DefaultZoneConfig for secondary tenant services is initialized to the global default. It does not inherit from the KV layer (it should).

This default is used in two places:

• when running ALTER RANGE default CONFIGURE ZONE USING DEFAULT
  if the KV layer has num replicas = 1, then this statement unexpectedly will set the tenant default config to num replicas = 3.

• when using the .DefaultZoneConfig() accessor in ApplicationLaterInterface in test servers.

Expected behavior

ALTER RANGE default CONFIGURE ZONE USING DEFAULT should always use the default tenant zone config as set via the system tenant.

Epic CRDB-26686

Jira issue: CRDB-31213

[knz]

The resolution here is probably:

• a new virtual range name alongside "liveness", "meta" etc in the system tenant, that holds the default zone config for secondary tenants.
• a watcher for changes to it, that pumps the zoneconfig protobuf into the KV side of the tenant connector
• receiving changes to this in the tenant connector
• applying them to the server config tenant-side.

[yuzefovich]

Based on the discussion here, the second place .DefaultZoneConfig() is separate from RANGE default, so I don't think the second point requires any action. The first point about propagating RANGE default zone config from system tenant into virtual clusters is still valid though.

[yuzefovich]

@knz I need a bit more guidance from you. As it turns out, there is also pre-defined RANGE TENANTS (introduced in #86343) which is used as the initial value of RANGE DEFAULT for newly-created virtual clusters. Thus, I'm a bit confused about the desired end state here. I'm my reading of the issue is correct, we would like for virtual clusters' RANGE DEFAULT to be effectively RANGE DEFAULT of the system tenant, and these two would be reconciled via a new watcher. However, how does RANGE TENANTS fit into this? Should it be deprecated?

[knz]

We sketched the direction in this doc.