Add conformance tests for GatewayConditionReady

[arkodg]

What would you like to be added:
Add an extended conformance test to ensure that when the Gateway Ready Condition is set to True, all traffic sent through this listener must succeed. Here is the condition in the spec.

Relates to #1832 (comment)

Why this is needed:
End users can better understand guarantees provided by implementations.

[gauravkghildiyal]

I'd like to work on this.

I'm sorry if I'm missing something obvious and it would be great if someone can help with the following doubt:

So I'm guessing that the GatewayConditionReady could be set to true even before any corresponding Route resources are programmed into the gateway. If that is the case, is there a way to verify the invariant "all traffic sent through this listener must succeed." without being explicitly aware of the implementation of the listener/gateway? Is there a consensus that when GatewayConditionReady=False|Unknown, we would get a connection timeout/refused (or other equivalent) but when GatewayConditionReady=True, the TCP connection would succeed (but could return some other application layer error)

[howardjohn]

I'd like to work on this.

I'm sorry if I'm missing something obvious and it would be great if someone can help with the following doubt:

So I'm guessing that the GatewayConditionReady could be set to true even before any corresponding Route resources are programmed into the gateway. If that is the case, is there a way to verify the invariant "all traffic sent through this listener must succeed." without being explicitly aware of the implementation of the listener/gateway? Is there a consensus that when GatewayConditionReady=False|Unknown, we would get a connection timeout/refused (or other equivalent) but when GatewayConditionReady=True, the TCP connection would succeed (but could return some other application layer error)

That is an excellent point. Ready=False does not imply traffic must fail IMO, just that we aren't 100% certain it will succeed.

However the interaction with route is troubling I think. It can't mean "all routes are ready", IMO, because we wouldn't have anyway to know if it's an outdated condition. Normally we use observedGeneration but this is cross-resource

[arkodg]

maybe a Ready condition is needed at the route level

gateway-api/apis/v1beta1/shared_types.go

Line 198 in 20be22d

// RouteConditionType is a type of condition for a route.

so a user can ascertain that a route is ready to serve traffic ?

[youngnick]

A Ready condition at Route level would be great, I agree, but I think that in practice, that will be very hard to build. It will mean that every bit of config from every HTTPRoute will need to be tracked in the data plane, so that an implementation can confirm that a particular piece of config that corresponds to that HTTPRoute has been propagated down to the data plane and is ready to pass traffic.

For Envoy data planes, for example, the controller will need to track which filter chain (at least) corresponds to which HTTPRoute, and ensure that ACKs and NACKs flow backwards through the controller, so that the status of the correct HTTPRoute can be updated at the right time. My experience is this is pretty hard to do right, because xDS is eventually consistent, in that even once a specific update has been ACKed, Envoy will need a (small) amount of time to flip things over. Will this be long enough to notice? Probably not, but we have no way of guaranteeing it, because of the way the Kubernetes eventual consistency lines up with the Envoy xDS consistency checks.

[howardjohn]

I feel like all of those concerns apply to "Ready" at Gateway level though? Which is why we aren't implementing it today - its a Hard Problem. I'd be really interested to see if anyone with an Envoy based implementation is able to do it correctly (and handling NxM data plane/control plane replicas)

From the user perspective, what value does Ready provide (over Programmed) if it doesn' include Routes? A gateway without routes is ~useless

[arkodg]

for the platform admin persona

• a Gateway listener Ready status will be useful so they can be sure config like tls is setup
  correctly in the data plane

for the app dev persona

• they might not have access to the Gateway resource
• the xRoute resource is the only resource that links the Gateway to the Backend, so a Ready condition here will signal to the user whether its safe to send traffic or not via this route.

yes, its a really hard problem, but hoping implementations are incentivized to solve it, since its very useful for end users

[howardjohn]

IMO we should either have Ready for both or neither. A partial state isn't particularly useful

[dprotaso]

I second having a Ready on both the Gateway and HTTPRoute - we need it for the reasons @arkodg mentioned.