conformance: Test that webhook validations are performed

[youngnick]

What would you like to be added:
This issue covers adding conformance tests that test that the implementation is implementing the same blocks as are implemented in the webhook. The simplest way to do this is to run the webhook we provide, but the conformance tests should be a bit agnostic about this.

However, I believe that the conformance test should check that invalid objects are never persisted into the cluster by trying to apply them, and checking that the apply fails. That flow will require some sort of webhook, or possibly the extensions to CRD validation api-machinery is considering (which I can't remember the name of right now).

A secondary part of this effort should also clarify some guidelines about what controllers should do with invalid objects - hopefully we can end up with this being as easy as "run a Validate function on the object, and drop the object if it fails" or something.

Why this is needed:
This will make it very clear that the webhook is required for having a conformant implementation.

This issue came up during discussion on #1497, but I think I've mentioned it before, and haven't written it down. Sorry all.

[rainest]

If there is a case where we expect that objects are accepted and reach controllers, how to test that behavior isn't immediately obvious. For example, in the case of #1497, if guidance was:

are invalid and will be rejected by the webhook is installed. Implementations MUST exclude these resources from configuration and add an `Accepted=false` condition with a `HeaderFilterNotValid` reason if the webhook is not running.

We can only test one or the other in a given run. If the webhook is running, the resource simply won't exist, and can't have conditions as such. If it is not running, tests can check conditions, but any test to confirm the webhook behavior (that the resource is rejected at creation time) would then fail. This would basically require two suites of conformance tests for behavior with or without the webhook, correct?

[youngnick]

Sigh, yes, you're right.

I think that if we're going to do this, we should lean in and say "we're not going to mandate anything about what happens if the webhook is not running, sorry".

[shaneutt]

Sigh, yes, you're right.

I think that if we're going to do this, we should lean in and say "we're not going to mandate anything about what happens if the webhook is not running, sorry".

This is inherently a problem with being out of tree, yeah?

If I'm not mistaken at monday's meeting Travis was basically asking us what we recommend as far as getting the webhook deployed on operator clusters, e.g. he was asking the question whether we recommend the webhook be loaded in with the existing (Go) operator. Today the onus is partially on cluster operators to ensure the webhook is installed, but it might be significantly better if implementations were packaging it. I know there's some hesitation to force everyone to run the webhook and I also understand there's some downsides (particularly with multiple implementations), but perhaps we should explore this further and see if we can't put the burden of ensuring the webhook's presence more heavily on the implementation so that cluster operators would have to opt out to start getting into grey territory rather than being in grey territory by default (as this may inevitably get us into trouble)?

[rainest]

Putting responsibility on implementations to run it, either as a separate container or integrated into a Golang controller, seems reasonable--that's basically what this test would enforce.

This is inherently a problem with being out of tree, yeah?

Would the situation change at all post-GA? I know that Ingress has more complex validation than schema enforcement alone, and it seems like we'd be able to add most if not all validation rules in the webhook there.

I don't know if maybe there's an effort to avoid adding additional validations there going forward, but if we would eventually be able to add them there, we'd be able to plan for eventually not worrying about the webhook being installed separately.

[robscott]

I've been thinking about this for some time now, and I tend to lean in a different direction. I think that we need to differentiate between a "conformant environment" and a "conformant implementation". In some instances, these are going to be one and the same, but in others, the controller author only has so much control. I'm not sure how useful it is for our controller-focused conformance tests to require the webhook to be present.

I'd rather test that controllers don't blow up when presented with invalid resources. Most controllers can't control how they're installed, they can simply provide a recommended set of steps. Our docs currently state the following:

CRD and webhook validation is not the final validation i.e. webhook is "nice UX" but not schema enforcement. This validation is intended to provide immediate feedback to users when they provide an invalid configuration. Write code defensively...

I still think the webhook is valuable and improves the UX of Gateway API dramatically, but I'm not convinced that it can/should be relied on or required to be present when running conformance tests.

[markmc]

Just for discussion purposes, I put up #1534

Some thoughts:

• It is definitely highly desirable to have validation like this - even the simple example of immutable GatewayClass.spec.controllerName shows how valuable it is in terms of UX
• If we believe this should usually be achieved by installing our webhook, then this does feel like it's more about a "conformant environment" than a "conformant implementation"
• If we want to check for a "conformant environment", it does seem a bit pointless to test all of the behaviors of the webhook - we can test the webhook before we publish its image!
• In gateway-api: Add opt-in validating webhook istio/istio#41868 the proposal (AIUI) is to use library code from gateway-api to provide some admission validation checks - if we were providing a comprehensive validation library instead of an installable webhook, it might feel a bit more like we're trying to enforce that "conformant implementations" are providing these validations?
• The GatewayClass.spec.controllerName example shows another weirdness with the "conformant implementation" approach - why would we require an implementation to validate objects that it would normally ignore (i.e. because it doesn't recognize the controller name) in order to be conformant? You'd expect we'd only require an implementation to validate objects it is accepting?

HTH

[howardjohn]

I feel like we are testing conformant environment. The environment really is the implementation. If we look at k8s for example, we aren't just testing some api-server binary, we are testing "Kubernetes" which consists of an environment many components. Gateway API is a much smaller environment, but fairly similar.

An environment may get the validation from the built in installable webhook, the Istio library import, or even something else - all that matters is it validates.

controllerName

I think its just a bit of a quirk of mixing global resources and shared resources with multiple implementations, the lines get a bit blurred. For example, you may have 1 validator for the entire cluster (in which case you want cross-class validation), or you may have N implementations doing validations (in which case you might want same-class only validation). Trying to make sure all environments have coverage of all resources is tricky - especially because Kubernetes isn't smart enough to know about "class" in the webhook, so it will actually still need to call all the validators, they could just chose to NOP. But if they are already called, it feels like they might as well validate as well. That does lead to making validation potentially stricter (if I have multiple versions, we are applying both versions of validation at once), but that feels OK to me.

So overall, I think it seems weird especially if we just look at GatewayClass, but it still feels like the best option to me, and its less weird for other fields.

[youngnick]

I think we've definitely had some good points raised here, so here's a new proposal from me.

What I'm really aiming for is this: the webhook currently implements some important checks for various types of object safety. I'd like to verify that a conformant implementation is doing some form of the same checks. It's a much better user experience if the invalid objects never make it into the apiserver (this is what the validation webhook is doing), but I think it's probably okay for implementations to not accept the invalid objects as well (this will handle @robscott's concern about invalid objects somehow making it through).

What I think we should work towards is a set of conformance tests (maybe a separate suite), that test that invalid objects (as defined in the webhook code) are either not accepted (ie, the webhook or some other webhook that performs the same checks is running), or that the objects are Accepted, status: false, with a Reason of FailedValidation (this is the case that the webhook is not running, but the implementation performs the same validations and updates the status accordingly).

For this to work, we'll need to ensure it's easy to import the validation code and either:

• use it in a validation webhook
• validate objects as part of normal implementation functions.

We already include the validation logic in release bundles, but we could make this easier by providing a simplified public API that will be easier to call (maybe a Validate function that takes a Kubenetes object.Object and then typecasts it and runs the type-specific checks?), and moving usage of it in our webhook to that.

Alternatively, we have a pretty clear pattern for validation functions in our current validation package. We could stick with what we have, then just mandate that:

• implementations need to validate resources as part of reconciliation, and not Accept them with a ValidationFailed reason.
• or, implementations can ensure the webhook is running
• or both.

This approach would let the conformance tests validate the important thing - that the restrictions described in the webhook, which aren't easily expressible in field-level validations, are in effect. Whether those restrictions are done in a webhook or in the implementation doesn't seem as important to me as that they happen somewhere.