Better protect experimental APIs from users, especially libraries

[dfawley]

Usage of our experimental APIs only continues to grow. Notable examples include: open-telemetry/opentelemetry-collector#14047, etcd-io/etcd#15145, #8669 (comment)...

It seems clear that documentation alone is not sufficient to dissuade people from widely deploying the use of our experimental APIs.

Providing a mechanism to properly declare experimental functionality is not a priority for the Go team, either: golang/go#34409

Some options:

1. Split experimental into its own module, release it as v0 only.

   • Some downsides here include that it makes our day-to-day development more difficult. Any change that spans modules requires multiple changes and dependency updates.
   • It might not fix the problem at all. People are very accustomed to seeing v0 in their go.mod. All of golang's x/ is v0, for example, but it is very widely used.
   • We already have experimental symbols that are in non-experimental packages. We would need to move them all.

2. Require setting an environment variable, otherwise any experimental functions panic.

   • This can trivially be implemented by a 3-line function that all experimental sites call.
   • We can be really really noisy instead, or do so temporarily for several releases before panicking. E.g. output a warning directly to STDOUT, STDERR and our logger. Ideally this would only occur once, or once every few minutes.
   • As a slight alternative to this option, we could require calling a function in grpc/experimental to enable the use of experimental APIs. We could even check that this is being done directly from func main() or a top-level test function.

3. Stabilize all our experimental APIs ASAP and never have experimental APIs

   • This seems largely intractable.
   • We would want to move some of them to internal rather than stabilize them, since we know we'd like to make changes but are blocked by some usages.
   • We could give up and stabilize some APIs that we're not happy with and then make V2 verisons of them.
   • We might end up with performance problems due to supporting some of the current APIs.