Skip to content

Commit b783fca

Browse files
shaneuttshaneutt
committed
docs: add implementation ideas for gep 1709
1 parent 69fabd9 commit b783fca

File tree

2 files changed

+307
-3
lines changed

2 files changed

+307
-3
lines changed

‎geps/gep-1709.md‎

Lines changed: 306 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77

88
Add selectable profiles for conformance tests which implementations can
99
subscribe to. Also add the ability to report conformance results back to the
10-
Gateway API project and receive recognition.
10+
Gateway API project and receive recognition. Conformance test reports will
11+
add conformance data to the implementations page for the reporter which can
12+
be linked to with badges from projects and repositories.
1113

1214
## Goals
1315

@@ -24,6 +26,12 @@ Gateway API project and receive recognition.
2426

2527
- We want to avoid adding any infrastructure for the reporting mechanism if
2628
feasible.
29+
- For this iteration we don't want to add configuration files for conformance
30+
tests, instead leaving that to future iterations and working on the raw
31+
machinery here (see [alternatives considered](#alternatives-considered)).
32+
- For this iteration we don't want to add container images for conformance test
33+
runs, instead leaving that to future iterations (see
34+
[alternatives considered](#alternatives-considered).
2735

2836
## Introduction
2937

@@ -37,8 +45,303 @@ which implementations can prove they satisfy and be recognized for.
3745

3846
## API
3947

40-
TODO
48+
The API for conformance profiles will be a pipeline that implementations can
49+
opt into. The workflow is effectively:
50+
51+
1. select a [profile](#profiles)
52+
2. [integrate](#integration) tests in the downstream project
53+
3. [report results and get certified](#certification)
54+
55+
The goal is to make selecting a conformance profile as simple of a process as
56+
feasible and support both the existing command line integration approach (e.g. `go test`) as well
57+
as a [Golang][go] approach using the conformance suite as a library.
58+
59+
[go]:https://go.dev
60+
61+
### Profiles
62+
63+
Initially there will be three profiles:
64+
65+
- `Layer4`
66+
- `Layer7`
67+
- `Mesh`
68+
69+
> **NOTE**: these are simply the initial profiles we're going to start with,
70+
> it's plausible for there to be more in the future.
71+
72+
These named profiles are effectively categories which represent the high level
73+
grouping of tests. When conformance is reported using one of these profiles
74+
extra features can be covered according to support levels:
75+
76+
- `extended`
77+
- `implementation-specific`
78+
79+
> **NOTE**: `implementation-specific` doesn't really have much in the way of
80+
> tests today, but it is something users want to be able to display so it's
81+
> considered for reporting purposes.
82+
83+
The technical implementation of these profiles is very simple: effectively a
84+
"profile" is a static compilation of existing [SupportedFeatures][feat] which
85+
represent the named category. Features that aren't covered under a "core" level
86+
of support are opt-in.
87+
88+
[feat]:https://github.com/kubernetes-sigs/gateway-api/blob/c61097edaa3b1fad29721e787fee4b02c35e3103/conformance/utils/suite/suite.go#L33
89+
90+
### Integration
91+
92+
Integrating the test suite into your implementation can be done using one of
93+
the following methods:
94+
95+
- The [go test][go-test] command line interface which enables projects of any
96+
language to run the test suite on any platform [Golang][go] supports.
97+
- Using the conformance test suite as a [Golang library][lib] within an already
98+
existing test suite.
99+
100+
> **NOTE**: Usage as a library is already an established colloquialism in the
101+
> community, this effort simply intends to make that more official.
102+
103+
Conformance profiles are passed as arguments when running the test suite. For
104+
instance when running via command line:
105+
106+
```console
107+
$ go test ./conformance/... -args -gateway-class=acme -conformance-profile=Layer7
108+
```
109+
110+
Or the equivalent configuration using the Golang library:
111+
112+
```go
113+
cSuite, err := suite.New(suite.Options{
114+
GatewayClassName: "acme",
115+
Profiles: sets.New(Layer7),
116+
// other options
117+
})
118+
require.NoError(t, err, "misconfigured conformance test suite")
119+
cSuite.Setup(t)
120+
121+
for i := 0; i < len(tests.ConformanceTests); i++ {
122+
test := tests.ConformanceTests[i]
123+
test.Run(t, cSuite)
124+
}
125+
```
126+
127+
> **NOTE**: In the `suite.Options` above it's still possible to add `SkipTests`
128+
> but when used in conjunction with `Profile` this will result in a report that
129+
> the profile is not valid for reporting. Implementations in this state may be
130+
> able to report themselves as "in progress", see the
131+
> [certification section](#certification) for details.
132+
133+
Alternatively for an `Extended` conformance profile where not all of the
134+
features are implemented (as described in the [profiles](#profiles) section
135+
above):
136+
137+
```console
138+
$ go test ./conformance/... -args \
139+
-gateway-class=acme \
140+
-conformance-profiles=Layer7,Layer4 \
141+
-unsupported-features=HTTPResponseHeaderModification,HTTPRouteMethodMatching,HTTPRouteQueryParamMatching,
142+
```
143+
144+
Or the equivalent configuration using the Golang library:
145+
146+
```go
147+
cSuite, err := suite.New(suite.Options{
148+
GatewayClassName: "acme",
149+
Profiles: sets.New(
150+
Layer7,
151+
Layer4,
152+
),
153+
UnsupportedFeatures: sets.New(
154+
suite.SupportHTTPResponseHeaderModification,
155+
suite.SupportHTTPRouteMethodMatching,
156+
suite.SupportHTTPRouteQueryParamMatching,
157+
),
158+
// other options
159+
})
160+
require.NoError(t, err, "misconfigured conformance test suite")
161+
cSuite.Setup(t)
162+
163+
for i := 0; i < len(tests.ConformanceTests); i++ {
164+
test := tests.ConformanceTests[i]
165+
test.Run(t, cSuite)
166+
}
167+
```
168+
169+
> **NOTE**: `UnsupportedFeatures` must match support levels. Disabling features
170+
> which are in core will emit a warning and wont be reportable.
171+
172+
> **NOTE**: In the future we may consider expanding the options to configure
173+
> and run the conformance test suite using a container image and/or via a
174+
> configuration file, however it was decided to hold off from doing any of
175+
> those in this iteration to avoid it getting to big (see the
176+
> [alternatives](#alternatives-considered) section for more thoughts).
177+
178+
Some implementations may support more or less extended features than others,
179+
so in some cases it could be cumbersome to have to list ALL features that you
180+
_don't_ support so we optionally and inversely allow `SupportedFeatures` so
181+
you can pick which option makes sense to you, and under the hood the
182+
expressions will compile to the same overall list:
183+
184+
```go
185+
cSuite, err := suite.New(suite.Options{
186+
GatewayClassName: "acme",
187+
Profiles: sets.New(
188+
Layer7,
189+
Layer4,
190+
),
191+
SupportedFeatures: sets.New(
192+
suite.SupportHTTPRouteMethodMatching,
193+
),
194+
// other options
195+
})
196+
197+
> **NOTE**: The `UnsupportedFeatures` and `SupportedFeatures` fields are
198+
> mutually exclusive.
199+
200+
Once an implementation has integrated with the conformance test suite, they can
201+
move on to [certification](#certification) to report the results.
202+
203+
[go-test]:https://go.dev/doc/tutorial/add-a-test
204+
[go]:https://go.dev
205+
[lib]:https://pkg.go.dev/sigs.k8s.io/[email protected]/conformance/utils/suite
206+
207+
### Certification
208+
209+
Implementations will be able to report their conformance testing results using
210+
our [reporting process](#reporting-process). Implementations will be able to
211+
visibly demonstrate their conformance results on their downstream projects and
212+
repositories using our [certification process](#certification-process).
213+
214+
#### Reporting Process
215+
216+
When conformance tests complete a log message will be emitted:
217+
218+
```console
219+
Success! Conformance profiles "Layer7" and "Layer4" passed for version v0.7.0.
220+
The following extended features were flagged as not supported for "Layer7Extended":
221+
- HTTPResponseHeaderModification
222+
- HTTPRouteQueryParamMatching
223+
```
224+
225+
For the above output upstream Gateway API can manually generate or update a report
226+
in the following format:
227+
228+
```json
229+
apiVersion: v1alpha1
230+
kind: ConformanceReport
231+
implementation: acmeorg-acme
232+
reports:
233+
layer4:
234+
releases:
235+
- tag: v0.7.0
236+
- tag: v0.6.2
237+
- tag: v0.6.1
238+
layer7:
239+
releases:
240+
- tag: v0.7.0
241+
extended:
242+
unsupportedFeatures:
243+
- HTTPResponseHeaderModification
244+
- HTTPRouteQueryParamMatching
245+
- tag: v0.6.2
246+
extended:
247+
unsupportedFeatures:
248+
- HTTPResponseHeaderModification
249+
- HTTPRouteQueryParamMatching
250+
- HTTPRouteMethodMatching
251+
- tag: v0.6.1
252+
extended:
253+
unsupportedFeatures:
254+
- HTTPResponseHeaderModification
255+
- HTTPRouteQueryParamMatching
256+
- HTTPRouteMethodMatching
257+
maintainers:
258+
- @kubernetes-sigs/gateway-api-maintainers
259+
```
260+
261+
> **NOTE**: In the above the `implementation` field is a combination of
262+
> `<organization>-<project>`. Organizations can be an open source organization,
263+
> an individual, a company, e.t.c.. Organizations can theoretically have more
264+
> than one `<project>` name and submit separate reports for each of them.
265+
266+
> **NOTE**: The above demonstrates that the `acmeorg-acme` project was already
267+
> passing for versions prior to `v0.7.0`. This is how we will track history of
268+
> conformance for releases across implementations, but at some point might
269+
> require some archival process (which we probably don't have to worry about
270+
> for this iteration). You can also see that there's been progress to add the
271+
> `HTTPRouteMethodMatching` for the latest release.
272+
273+
> **NOTE**: The `maintainers` field indicates the Github usernames or team
274+
> names of those who are responsible for maintaining this file, so they can be
275+
> easily contacted when needed (e.g. for relevant release announcements
276+
> regarding conformance, e.t.c.).
277+
278+
The implementation can compile its `ConformanceReport` and upload it to the
279+
Gateway API project by creating a pull request. The file should be uploaded
280+
to the following sub-directory:
281+
282+
```console
283+
conformance/results/<organization>-<project>.yaml
284+
```
285+
286+
> **NOTE**: For this iteration we're focusing on getting these reports created
287+
> but at first they will probably be created by hand. In future iterations and
288+
> after having some time to experience the system, tooling to handle reporting
289+
> should be created.
290+
291+
This will start the certification process which will result in the
292+
implementation receiving a badge they can use on their project pages and/or
293+
repositories which indicates their conformance levels and link to their
294+
conformance reports in the upstream Gateway API website.
295+
296+
> **NOTE**: No verification process (to prevent intentionally incorrect
297+
> conformance results) will be implemented at this time. We expect that this wont
298+
> be an issue in our community and even if someone were to try and "cheat" on
299+
> the reporting the reputation loss for being caught would make them look very
300+
> bad and would not be worth it.
301+
302+
#### Certification Process
303+
304+
For this initial iteration the raw report data of the `ConformanceReports` will
305+
live in its own directory and _is predominantly meant for machine consumption_.
306+
Report data will be compiled into human-friendly displays which will be added as
307+
headers in the [implementations page][impl] and badges which link to these
308+
displays. We will refer to this process as certification.
309+
310+
Certification starts with the pull request described during the [reporting
311+
process](#reporting-process). Once the `ConformanceReport` is created or
312+
updated a display layer in the implementations page will need to be updated to
313+
point to the new data.
314+
315+
TODO: not sure exactly what the display will look like yet. We will need to
316+
sort this out before we consider this `implementable`.
317+
318+
CI will provide [badges][bdg] to contributors at the end of the process which
319+
link to the implementations page for that specific implementation and can be
320+
easily added via markdown to Git repositories.
321+
322+
[impl]:https://gateway-api.sigs.k8s.io/implementations/
323+
[bdg]:https://shields.io
324+
325+
## Alternatives Considered
326+
327+
### Conformance Test Configuration File
328+
329+
Conformance testing is currently done mainly through command line with
330+
`go test` or via use of the conformance test suite as a library in Golang
331+
projects. We considered whether adding the alternative to provide a
332+
configuration file to the test suite would be nice, but we haven't heard
333+
any specific demand for that from implementors yet so we're leaving the
334+
idea here for later iterations to consider.
335+
336+
### Conformance Test Container Image
337+
338+
Providing a container image which could be fed deployment instructions for an
339+
implementation was considered while working on this GET but it seemed like a
340+
scope all unto itself so we're avoiding it here and perhaps a later iteration
341+
could take a look at that if there are asks in the community.
41342

42343
## References
43344

44-
TODO
345+
- https://github.com/kubernetes-sigs/gateway-api/issues/1709
346+
- https://github.com/kubernetes-sigs/gateway-api/issues/1329
347+

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ nav:
7575
- geps/gep-1426.md
7676
- geps/gep-1324.md
7777
- geps/gep-1282.md
78+
- geps/gep-1709.md
7879
- Experimental:
7980
- geps/gep-1323.md
8081
- geps/gep-1016.md

0 commit comments

Comments
 (0)