Character encoding in discovery

[taleodor]

We should require URL-safe character encoding in discovery, similarly how PURL is doing it - package-url/purl-spec#461

We may want for the PURL PR referenced above to be merged to reuse similar approach with TEA.

[oej]

TEI should be URI safe :-)

I did a lot of tests in code and had no problems embedding PURLs in URI syntax, but don't know how much I can rely on that library. When using a TEI as an argument in a URL, we need to be URL safe. That should be documented in the API spec.

[ppkarwasz]

OpenAPI takes care of most encoding concerns by design. For example, Appendix C of the OpenAPI Specification references both RFC 3986 and RFC 6570, which effectively mandate that Unicode data in URLs must be UTF-8 encoded before percent-encoding.

RFC 3986 further clarifies that URIs differing only by the percent-encoding of characters in the unreserved set (alphanumerics plus -, ., _, ~) are equivalent. This means encoding these characters is optional and has no semantic effect. The real ambiguity lies in how characters from the reserved set (e.g., ?, /, &) are handled.

OpenAPI addresses this with the allowReserved flag:

• By default, reserved characters in parameter values are percent-encoded, ensuring they are not misinterpreted as URL delimiters.
• If allowReserved is set to true, OpenAPI allows reserved characters to appear unencoded in query parameters.

The discussion in the Package URL (purl) spec is somewhat different. There, the goal is to define a canonical byte-level representation so that two PURLs can be compared for exact equivalence. In contrast, OpenAPI assumes that small variations in URL encoding are acceptable and should be resolved by the server.

[taleodor]

Great points, just one note that we encountered this issue in the TEI URN resolution via /.well-known/tea which is not a part of the OpenAPI spec.

Additionally, current examples of PURL (pre-merge of the encoding PR) include non-encoded characters from the reserved set, i.e.

pkg:deb/debian/curl@7.50.3-1?arch=i386&distro=jessie

Which makes it impossible to build proper resolution via TEI URN of type PURL without URL encoding, which in turn may be inconsistent with the final set of rules that PURL will establish and also may lead to double encoding after the PURL's encoding PR is merged.

Because of the above we need clear guidelines here for how to build implementations. For these reasons, for TEA Beta 1 we decided to only implement TEI URN of type UUID in our implementation (since there are no problematic characters in UUIDs) - and I believe everybody would encounter same issues here.

[ppkarwasz]

Thank you for the clarification—now I see the issue as well.

The problem is not with transporting the TEI as a query parameter (which is well defined), but with constructing a valid TEI from its components. We have defined the TEI as follows:

tei     = "urn:tei:" tei-nss ;
tei-nss = type ":" host ":" identifier ;

Here, identifier is a string whose structure depends on the chosen type. However, according to RFC 8141, which defines the syntax of URNs, the tei-nss must conform to the NSS production rule. This imposes constraints on the identifier component: it may only include characters allowed by the unreserved and reserved productions—excluding ?, #, [, and ]. Any other ASCII characters must be percent-encoded using the pct-encoded format.

Package URLs conform to the URI syntax and are therefore nearly usable as identifier values. However, since they may contain the four disallowed characters listed above, those must be percent-encoded to ensure compliance.

✅ Possible Solution

To ensure that the identifier component of a TEI conforms to RFC 8141 constraints, we could use the following encoding strategy:

The identifier must be the ASCII string produced by the percent-encode-after-encoding algorithm from the WHATWG URL Standard, using as percentEncodeSet:

• The C0 percent-encode set, which includes control characters and non-ASCII codepoints, and
• An extended set of characters defined by the percent-encode-set-supplement rule below.

This ensures that all characters not permitted by the NSS production in RFC 8141 are properly encoded.

percent-encode-set-supplement = disallowed-char / "?" / "#" / "[" / "]" / "%" / SP ;
disallowed-char               = DQUOTE / "<" / ">" / "\" / "^" / "`" / "{" / "|" / "}" ;
DQUOTE = %x22 ; "
SP     = %x20 ; space

Note

The WHATWG URL specification does not include the percent sign (%) in any of its default percent-encode sets. However, in the context of TEI identifiers—especially when derived from Package URLs—any literal % must be encoded as %25 to ensure the original identifier can be unambiguously reconstructed after decoding.

📦 Examples

Given the encoding rules described above, the following Package URLs are transformed into valid identifier components of TEIs by applying percent-encoding to disallowed characters.

Input Package URLs

pkg:deb/debian/curl@7.50.3-1?arch=i386&distro=jessie
pkg:npm/%40antora/cli

Resulting TEIs

tei:purl:curl.se:pkg:deb/debian/curl@7.50.3-1%3Farch=i386&distro=jessie
tei:purl:antora.org:pkg:npm/%2540antora/cli

Explanation

In the curl example only the question mark ? need to be percent-encoded.

In the Antora example only the percent % character needs to be encoded.

ABNF Syntax from RFC 3986 and RFC 8141

NSS         = pchar *(pchar / "/") ;
pchar       = unreserved / pct-encoded / sub-delims / ":" / "@" ;
pct-encoded = "%" HEXDIG HEXDIG ;
unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~" ;
reserved    = gen-delims / sub-delims ;
gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@" ;
sub-delims  = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "=" ;

[stevespringett]

We're likely run into different character encoding strategies across all the different types of identifiers, PURL, CPE, GTIN, etc.

One possibly unpopular but viable solution may be to Base64 encode/decode the identifiers.

[taleodor]

Great idea, makes me think that we could also (alternatively) modify discovery logic as following:

1. /.well-known record would contain static information or redirect to the base URL of TEA API
2. We will define a new discovery resolution endpoint inside the API itself
3. We will use POST with identifier string to that API call - which will solve the problem, as special characters won't be an issue for POST

[ppkarwasz]

Great points, just one note that we encountered this issue in the TEI URN resolution via /.well-known/tea which is not a part of the OpenAPI spec.

I just realized that the current description of the discovery mechanism doesn’t match what I recall from the community meetings. I thought we had agreed that clients would make OpenAPI calls directly to the /.well-known/tea endpoint, treating it as if it were the actual service endpoint, and the server would respond with a redirect to the appropriate URL.

Under the current spec, this would look like:

1. Start with a TEI like urn:tei:<irrelevant>:<domain_name>:<some_id>.
2. Make an OpenAPI call to:
   https://<domain_name>/.well-known/tea/products?idType=tei&idValue=<encoded_tei>
3. The server responds with a 302 redirect to:
   https://teapot02.consumer.example.com/tea/v2/products

Let me know if I’m misremembering or if the approach has changed.

[taleodor]

I wasn't too involved in discovery, but what I remember is we agreed to have ./well-known not part of OpenAPI spec. The reason is that ./well-known needs to redirect and point to OpenAPI and if it's itself inside OpenAPI, that would create a chicken-and-egg problem (and I remember you raised this point).

Also, notice from your example that you essentially have idType hardcoded to tei and this is always will be tei - based on our discovery conventions - therefore there is little reason for the idType param to exist on ./well-known - so the current discovery description is actually very similar to what you provided.

All that said, I'm open to have a call on this, but we need @oej as he was most involved with the discovery.

[taleodor]

Resolved by #196 .