WIP: Add a loading mechanism for compat-dependent API sets

[Keno]

This is a first WIP implementation for providing the underlying mechanism for #54905. I haven't fully designed out what I want the final thing to look like, but I've thought about it enough that I think we need some hands-on examples for further design exploration, which is what this PR is designed to support.

The core idea here is that if you specify in your package

[compact]
julia = "0.14"
MyDependency = "0.3,0.4"

Then this mechanism will ask MyDependency to provide you a view of its API that is compatible with its 0.3 version, even if 0.4 is installed. The objective of this mechanism is that downstreams of widely used packages (including Base and standard libraries) can be upgraded more gradually by allowing some portion of the packages in project to use the new 0.4 API set, while packages that have not yet upgraded can continue to use the 0.3 API set. Currently, whenever a widely used package changes an API there's a big frenzied rush to update downstreams, which just isn't sustainable as the ecosystem grows.

In other languages, problems like these are solved by allowing multiple versions of the same package to be loaded. However, in julia, this is not particularly feasible by default because packages may have generic functions that need to be extended and unified and which break if there are multiple copies of such resources.

That said, for simple packages, this mechanism could be used to emulate the multiple-version-loading strategy on an opt-in basis.

The way that this works is that loading gains an additional layer of indirection that is provided the compat specification of the loading package. Packages can opt into this by providing a

module MyDependency
function _get_versioned_api(compat)
# API compat logic, further described below
end
end

function. Where the default is equivalent to
_get_versioned_api(compat) = @__MODULE__.

The loader makes no further assumptions on either the structure of compat or how the package processes it, other than that it must return a Module. This is intentional to allow evolution without impacting the core loading logic (of course Pkg also looks at the structure of compat, as would the _get_versioned_api implementation in Base, so there are some constraints).

That said, the envisioned usage of this is that we create a standard (in the sense of being widely used, not in the sense of it being a stdlib) package to help package authors define the APIs of their packages more precisely and with version information. This package would then create a set of modules under the hood that describe these APIs to the system and would set up _get_versioned_api appropriately (i.e. the mechanism is not intended to be used directly in most cases).

To actually make this useful, we will likely need some additional features in the binding system, in particular:

1. bindings: Add native automatic re-export feature #59859 and a few variants thereon to make the API modueles look more transparent.

2. A mechanism to intercept extension of generic function overloads in order to be able to provide compatibility (Design/PR for this forthcoming).

Note that this PR itself is WIP, it is not yet fully complete and there is also a Pkg.jl side of this required to put compat info into the manifest.

[KristofferC]

Just to check my understanding, this effectively means that what module you get from using Foo is "overloadable" based on what compat string the module loading Foo has?

Would this make it harder for static analyzers, since they AFAIU have to be able to run the _get_versioned_api to know what loading a package means?

Also, I am worried it will also be confusing to humans since you always have to have that compat context available (and run the _get_versioned_api with it) to understand what is going on when reading code. A.Foo.something() and B.Foo.something() can mean something completely different depending on the compat of A and B, which is not available to read anywhere except the manifest. And yes, you can argue that is already somewhat the case since Foo could be different packages, but that is quite rare in practice.

There also needs to be extra registry information here to say that a semantically breaking version bump should not be considered breaking, IIUC?

[Keno]

Just to check my understanding, this effectively means that what module you get from using Foo is "overloadable" based on what compat string the module loading Foo has?

What module the using operates on, yes. But note that this is not technically new since the resolution already depends on the contents of the Project.toml of the loading module.

Would this make it harder for static analyzers, since they AFAIU have to be able to run the _get_versioned_api to know what loading a package means?

Kind of - they could probably just ignore it and resolve the minimum set. Depends on what they're trying to do. We already have various callback mechanisms in loading though, so I don't think this ups the code execution burden for a full loading implementation.

Also, I am worried it will also be confusing to humans since you always have to have that compat context available (and run the _get_versioned_api with it) to understand what is going on when reading code. A.Foo.something() and B.Foo.something() can mean something completely different depending on the compat of A and B, which is not available to read anywhere except the manifest. And yes, you can argue that is already somewhat the case since Foo could be different packages, but that is quite rare in practice.

I think that is true, but as you point out, could already be the case now. I think the alternative to think about here loading multiple versions of the module, which would have the same problem.

There also needs to be extra registry information here to say that a semantically breaking version bump should not be considered breaking, IIUC?

I was not planning to do this. Rather, I was expecting packages to start using minor version bumps for changed APIs and only use major version bumps when removing the compat code.

[mbauman]

Are these compat modules actually Extensions? It feels like the same sort of conditional code loading based on some external state. But it feels slightly backwards in some way that I can't quite put my finger on.

I don't really like that I would now need to know two things in order to understand what using Example means — I need to know both its compat clause and which compat APIs it provides. If it doesn't provide any, I guess I can ignore its compat clause? If it provides some, I need to intersect that with the specified compat. What happens if I know my usage can support both v1 and v2 of a given package (so I have Example = "1,2"), but Example hasn't provided a v1 compat layer? Or what if it does, but I want to explicitly (and conditionally) enable v2 APIs — something I'd historically have done with isdefined introspection?

[Keno]

Are these actually Extensions? It feels like the same sort of conditional code loading based on some external state. But it feels slightly backwards in some way that I can't quite put my finger on.

No, there is no conditional loading.

I don't really like that I would now need to know two things in order to understand what using Example means — I need to know both its compat clause and which compat APIs it provides.

Well, I'd argue the most important thing to know (even now) is which versions of Example you might get. The only way we have for a package author to control that right now is with the compat setting, since any user of your package can create a manifest with one of your dependencies that is any version compatible with your compat setting. In fact the mechanism is supposed to make the mental model easier by letting you basically assume that you're always working against the minimum compatible version in which case everything should work. Of course this depends on people using the mechanism correctly.

What happens if I know my usage can support both v1 and v2 of a given package (so I have Example = "1,2"), but Example hasn't provided a v1 compat layer?

If example does not use this mechanism nothing changes.

Or what if it does, but I want to explicitly (and conditionally) enable v2 APIs — something I'd historically have done with isdefined introspection?

You would not use this mechanism and instead put a versioned import behind a version check (which is outside the scope of this PR).

[mbauman]

Of course this depends on people using the mechanism correctly.

Yeah, that's the part I'm struggling with, I suppose, combined with packages not using the mechanism at all. Because currently, it's entirely the responsibility of the downstream author to ensure that their compats are accurate, often doing so by conditionally updating to support the new APIs or changing the compat. But now the dependency itself can (sometimes!) participate and (perhaps imperfectly!) downgrade its new versions based on the existing compat, but only if the dependency writes some Turing-complete code to handle that compat version for all registered versions in the given compat range.

I know, I know, you're gonna say that's all intentionally out of scope, and I get that this is intentionally very flexible... but this feels wildly flexible.

[Keno]

But now the dependency itself can (sometimes!) participate and (perhaps imperfectly!) downgrade its new versions based on the existing compat, but only if the dependency writes some Turing-complete code to handle that compat version for all registered versions in the given compat range.

Well, the dependency would tag a new version of itself. I don't think this is any different from tagging a new patch version now that unintentionally breaks an API.

but only if the dependency writes some Turing-complete code to handle that compat version for all registered versions in the given compat range.

As I said, it is not the intention that package authors will use this mechanism directly (there are some corner cases where they may want to, e.g. for packages that provide FFI bridges and want to emulate by loading multiple package versions in the other language). There are some syntax proposal for defining API sets in the original issue, but that's a separate concern from the mechanism here. I do think erring on the side of flexibilty in the loader here is the right call. We could of course define some complicated data structure to declare API sets in base and just parse that here, but then we're stuck with it.,

[Keno]

I think as we discuss this there are a couple of things we should separate out.

1. Should we have an API versioning mechanism at all?
2. Should it be tied to compat?
3. If so, is this an appropriate mechanism for it.

I think the discussion will be more helpful if these are separated out.

[Keno]

In offline discussion, there is some skepticism that this mechanism is necessary. In particular, there is a concern that the mechanism being available by default would suggest to package developers that they must use it, as opposed to it being a tool for package developers with large downstream dependency chains who want to maintain backwards compatibility. Thus the plan is not to merge this and instead focus on providing the requisite functionality in a package. Users would have to opt-in to versioned APIs with a macro, but that's likely acceptable and can still read the compat. We can revisit making this automatic in a future julia version if the concept is well accepted.