all: Updates for Protocol Version 5.4 and 6.4

[bflad]

terraform-plugin-mux version

v0.11.2

Description

Terraform 1.6 is introducing new functionality which can greatly reduce memory utilization when there are multiple instances of same provider type, by only calling the GetProviderSchema RPC against one provider instance. This can be accomplished by the provider signaling to core that the GetProviderSchema RPC is not required before other RPCs. While this is straightforward to accomplish within a provider with a single provider server implementation, e.g. a single SDK such as terraform-plugin-framework, with terraform-plugin-mux the situation is much more nuanced. Currently, the "routing information" in a mux server for resource-specific RPCs is based on having the GetProviderSchema called so it knows which underlying provider server has each resource type. Injecting a GetProviderSchema call to fill in the routing table when receiving another RPC would defeat the memory usage optimization.

To cover this additional gap, the protocol is also being enhanced with a new GetMetadata RPC, which returns server capabilities and only a listing of each resource type name. A mux server can attempt this new lighter-weight RPC before calling GetProviderSchema, ignoring any GetMetadata RPC unimplemented error as necessary.

Proposal

For an initial functionality pass, each of the mux/translation server implementations will need to be updated to support the new GetMetadata RPC and the existence of the ServerCapabilities.GetProviderSchemaOptional flag. The GetMetadata RPC implementation can be setup similar to GetProviderSchema RPC, looping through each underlying provider server to make the RPC call and returning any errors. The GetProviderSchemaOptional server capability must not be announced at this phase, as each of the resource-specific RPCs will still be dependent on the routing information via GetProviderSchema.

To finish implementing the functionality:

• The GetMetadata RPC should be wired to automatically fall back to the GetProviderSchema RPC without returning any GetMetadata associated errors. While not ideal, the GetProviderSchema information is guaranteed to work and contains the necessary resource type name information.
• The resource-specific RPCs should be wired to populate the routing information, if not already populated, via GetMetadata/GetProviderSchema and the GetProviderSchemaOptional server capability should be enabled.
• Updating the routing information should be protected via mutex to prevent multiple calls to the underlying provider servers.

It is anticipated that providers needing the memory usage improvements will upgrade these Go module dependencies at once:

• terraform-plugin-go (targeting v0.19.0)
• terraform-plugin-framework (targeting v1.4.0)
• terraform-plugin-sdk (targeting v2.29.0)
• terraform-plugin-mux (targeting v0.12.0)

References

• enable schema caching and add get_provider_schema_optional server capability terraform#33486
• Add GetMetadata RPC to provider protocol terraform#33739
• Introduce Protocol Server Capability for Optional GetProviderSchema RPC terraform-plugin-go#310
• Updates for Protocol Versions 5.4 and 6.4 terraform-plugin-framework#828
• Updates for Protocol Version 5.4 terraform-plugin-sdk#1234