Incorrect conformance tests and reporting for Mesh?

[LiorLieberman]

What happened:
Looking into our conformance reports and tests for Mesh, I see features being reported as "supported" but suspect we are not even testing them. Taking istio as an example.

1. The last conformance report reports support for many features.
2. If we take HTTPRoutePathRewrite as an example, we can see that the test for this feature only deploys an HTTPRoute with a Gateway as a parentRef and not a Service.
3. In fact, most of the tests for HTTPRoute features are not covering mesh functionalities. There are only a few tests that actually tests mesh and those are the feature names that start with "Mesh"
4. This leads to incorrect reporting - implementations that support N/W HTTPRoute functionalities reports the same support for mesh if they pick the mesh profile.
5. While this case should be ideal, its not 100% the case, and we basically have minimal coverage for Mesh support.

What you expected to happen:
We should test every feature thats applicable to mesh (should be most of the features if we stick to the vision) with Service as a parentRef in addition to Gateway.

Ideally we should address this as a framework change with support to exclude features as opposed to add this functionality to every test. The Service-as-a-parentRef tests should only be running if the implementation indicated that they support the MESH profile.

/area conformance
/cc @howardjohn @robscott @youngnick @mlavacca @kflynn @shaneutt @mikemorris

[LiorLieberman]

/area mesh

[LiorLieberman]

/label release-blocker

I believe some form of this should be a blocker. If we are unable to do the framework changes by 1.3 I'd hope the reporting would at least change for mesh to not report more features than tested.

[k8s-ci-robot]

@LiorLieberman: The label(s) /label release-blocker cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, ci-short, ci-extended, ci-full. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label release-blocker

I believe some form of this should be a blocker. If we are unable to do the framework changes by 1.3 I'd hope the reporting would at least change for mesh to not report more features than tested.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[LiorLieberman]

Probably I am not allowed to use this label?
@shaneutt @robscott

[robscott]

/label release-blocker

[k8s-ci-robot]

@robscott: The label(s) /label release-blocker cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, ci-short, ci-extended, ci-full. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label release-blocker

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.