proto: try to build a pixi backend

[ruben-arts]

Based on the idea #404.

The magic is that we do almost nothing in the backend. We just run a Pixi task in the build script, and use all dependencies directly from the pixi.toml

I don't really see the need for this to be a backend, this was just the easiest way for me to set it up.

Draft because I build this in 1 hour to try something, don't merge!

My test pixi.toml:

[workspace]
channels = ["conda-forge"]
platforms = ["osx-arm64"]
preview = ["pixi-build"]

[tasks]
# The default is to run the `pixi run build` task.
# The user is fully responsible to make sure that it runs that in the right location.
build = {cmd = "touch $PREFIX/build.txt", depends-on = ["configure"]}
configure = "touch $PREFIX/configure.txt"

[dependencies]
self = {path = "."}

[package]
name = "self"
version = "0.1.0"

[package.build.backend]
name = "pixi-build-pixi"
version = "*"

[package.build-dependencies]
cmake = "*"

[package.host-dependencies]
python = "*"

[package.run-dependencies]
py_rattler = "*"

Things I like about this:

• Use Pixi's task runner to create cross-platform commands, keeping the complexity in the pixi.toml.
• Directly usable without the use of pixi-build-rattler-build to do "crazy" setups as long as the commands can be run as a pixi task
• No metadata logic needed (no magic)
• No default input globs so I can specify when to rebuild automatically. (still need a way to force a rebuild without glob changes)

[burdiyan]

The user is fully responsible to make sure that it runs that in the right location.
WDYM by this? Will the task run in the isolated work directory like other build backends, or will it just run in the same directory where the package's pixi.toml file is?

Overall, I really like this — I'm biased ofc :)

[burdiyan]

Overall I think this just needs to be a built-in feature in Pixi — being able to depend on other Pixi packages.

[ruben-arts]

Overall I think this just needs to be a built-in feature in Pixi — being able to depend on other Pixi packages.

I agree, I'm right now not planning to continue this backend. This was just for testing purpose, as all the logic was already prepared to make it work.

[burdiyan]

Any plans to implement this feature directly in Pixi anytime soon? Because I'm about to do some pretty ugly stuff to work around this (more on that below) 😅, so if there was something built in — it would be so much easier.

This has become a big deal-breaker for us to switch our setup from Nix to Pixi, as all the existing build backends don't have good caching and are too aggressive with rebuilds. We have quite a few tools that we don't want to build proper Conda packages for, so we really want to be able to deterministically build/download them via Pixi.

If you're interested, my idea to work around the limitation is the following:

First, I'd have an activation script in the main workspace, that would execute pixi tasks to build or download the necessary tools, using inputs and outputs properties in tasks to make them cacheable. Because of the caching, this should not have major performance drag for the activation.

Because activation scripts are executed before pixi tasks, and before the environment is configured, we need to avoid the infinite recursion, so the script would set an environment variable, such that when it's called recursively it would simply exit. This way the script should be able to call pixi tasks, with all the tools available, and also guaranteed to execute without the developers needing to do an extra "install/configure" call.

Of course this is all very ugly and hard to reason about, but it seems to do the job.

[ruben-arts]

@burdiyan I can't promise that we will do that soon. We are prototyping this proposal next week: prefix-dev/pixi#4721

Please leave your idea there, and the whole idea of [package.tasks], while not refined, sounds like the exact thing you want.