Code loading: add support for "portable scripts".

[KristofferC]

Because one file is much fewer than three.

These are jl files that have project and manifest embedded inside them, similar to how Pluto notebooks have done for quite a while.

The delimiters are #!manifest begin and #!manifest end (analogous for the project data), and the content can either have single-line comments or be inside a multi-line comment.

When the active project is set to a portable script, the project and manifest data will be read from the inlined toml data.

Starting julia with a file (julia file.jl) will set the file as the active project if julia detects that it is a portable script.

The tests and the parser for the inline toml data were written by Claude 🤖 .

---

I still have to finish the Pkg part (edit JuliaLang/Pkg.jl#4479) that writes these files, so this cannot be tested "end-to-end" right now. However, if one creates a portable script manually like this (rename to .jl)
portable_script.txt, we can check that it works via:

❯ JULIA_LOAD_PATH="@" ./julia portable_script.jl
✓ Successfully ran portable script with Plots, CSV, and DataFrames!

❯ JULIA_LOAD_PATH="@" julia +nightly portable_script.jl
ERROR: LoadError: ArgumentError: Package Plots not found in current path.
- Run `import Pkg; Pkg.add("Plots")` to install the Plots package.
Stacktrace:
  [1] macro expansion   

where we can see that the manifest information inside the portable script is used.

Invitation to bike shedding:

• How should the start and end delimiters look
• Should Pkg default to putting them above or below the content.
• Should inline manifest information be needed? If not, what do we do on a Pkg.resolve()? Put it inline or in some temp environment?
• Is it ever ok to use the fle ending (.jl) to have semantic meaning for the code loading?
• Other things?

In addition, I think we need a --instantiate flag or something in Julia that checks if all the packages are available for the scripts and if not, side-loads Pkg to download them. Maybe that should even be automatic for portable scripts. I think this feature has been requested on the Pkg repo. Being able to get a file and run it in "one shot" is quite tempting to support.

---

Previous work:
https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies
https://internals.rust-lang.org/t/pre-rfc-cargo-script-for-everyone/18639

[MasonProtter]

What will happen here if there is only a #!project but no #!manifest? Is that okay? Sometimes when passing around a script you don't want to bundle the full manifest info.

[KristofferC]

Yeah, that's what the

Should inline manifest information be needed? If not, what do we do on a Pkg.resolve()? Put it inline or in some temp environment?

question is about. In order to run the script you at least need to do a resolve it to have manifest data available so you have some concrete versions to load. Maybe julia should do that automatically if it tries to run a portable script without manifest information. But then where should the result be stored? inline or in some hashed directory somewhere?

Also, when running a portable script julia should maybe disable the global env from the load path by default?

[MasonProtter]

Oops sorry, I missed you listing that.

Yeah, what I was picturing was that we could would have a folder for script environments in .julia/environments that stores the "real" project and manifest files. Then we'd just need a hook during Pkg.gc or whatever that when it checks and finds these Project files, it checks if the originating script still exists. If the script is gone, we delete the project/manifest.

Maybe there's a better way to handle that.

[KristofferC]

Yeah, what I was picturing was that we could would have a folder for script environments in .julia/environments that stores the "real" project and manifest files.

Only manifest files, right? If you don't even have an inline project section, then you are just a normal julia file, or?

[MasonProtter]

I was imagining that project info from the script would get copied there, and if there is a manifest it also gets copied. But maybe that's not necessary and you don't need copies, just writing out a manifest if it's not embedded.

[KristofferC]

I was imagining that project info from the script would get copied there

We just read it inline now, so no need for that really.

What should the .julia/environments/scripts/... be keyed on? Some hash based on the absolute path of the script?

[MasonProtter]

In the silly little implementation I played with recently, I was just replacing the path separators in the absolute path with underscores. But there's probably a bunch of reasons why that's a bad idea.

[JanisErdmanis]

Could the key for ~.julia/environments/script/ be a hash of the extracted Project.toml itself?

Also it is worth to add that if manifest were to added in the script itself that would make editing such scripts manually by hand much less pleasant. It doesn’t seem like users would want such a level of reproducibility in practice where compat bounds would suffice.

[KristofferC]

Could the key for ~.julia/environments/script/ be a hash of the extracted Project.toml itself?

I thought about that but then you get collisions if you have different scripts that happen to get the same project content.

It doesn’t seem like users would want such a level of reproducibility in practice where compat bounds would suffice.

I disagree, I think you would want to send someone a file and have them be able to reproduce stuff?

[JanisErdmanis]

Wouldn't interpreting the compat bounds as exact versions with which to resolve produce the same Manifest.toml when executed on the same Julia version?

[KristofferC]

I don't really want to have a discussion here on how the resolver works. The feature can be made to support both inline and outline manifest info, so not sure what there is to discuss more about this.

JuliaLang/Pkg.jl#4479 is a slightly WIP follow-up to this from the pkg side. With that, you can do:

julia> Pkg.activate("portable_script.jl")
  Activating project at `~/julia/FatEnv/portable_script.jl`

(portable_script.jl) pkg> st
  Installing known registries into `~/.julia`
       Added `General` registry to ~/.julia/registries
Status `~/julia/FatEnv/portable_script.jl`
  [a93c6f00] DataFrames v1.8.1
  [7876af07] Example v0.5.5
  [91a5bcdd] Plots v1.41.1

(portable_script.jl) pkg> rm Example
    Updating `~/julia/FatEnv/portable_script.jl`
  [7876af07] - Example v0.5.5
    Updating `~/julia/FatEnv/portable_script.jl`
  [7876af07] - Example v0.5.5

(portable_script.jl) pkg> st
Status `~/julia/FatEnv/portable_script.jl`
  [a93c6f00] DataFrames v1.8.1
  [91a5bcdd] Plots v1.41.1

(portable_script.jl) pkg> precompile
Precompiling packages  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━ 122/157
  ◐ StatsBase

and the file updates during pkg operations.

[tecosaur]

One preference from me: I think it would be nice to put/read the manifest from the bottom, so if you open the script and start looking at it you don't see a few hundred lines of TOML up front.

[KristofferC]

Yeah, I also got to that conclusion. But the project data should still be on top?

[KristofferC]

Regarding storing the manifest inline or not, I think a key in the project section could determine that.

[tecosaur]

Yeah, I also got to that conclusion. But the project data should still be on top?

On one hand, splitting it does seem a bit odd, but having project info at the top does feel right to me somehow.

Regarding storing the manifest inline or not, I think a key in the project section could determine that.

I'm guessing there may be some scripts which are a bit "package-like" in that they have compat bounds on packages, and it's fine for those to be resolved when run. In this way, having the manifest section be optional seems nice.

[KristofferC]

Actually for the outline manifest we can use the manifest = key in the project data. That way you can move around the script without losing track of the manifest (say if it was hashed by path)

[KristofferC]

Ok, so I have made the following changes (mostly in Pkg).

• By default, it starts writing manifest info inline.
• If you put inline_manifest = false in the project section, it puts manifest = "~/.julia/environments/scripts/portable_script_0686334e-56a7-4d0f-b85f-a9bbebe086df/Manifest.toml" in there and copies the existing inline manifest into that file.
• If you put inline_manifest = true, it removes the manifest = entry and copies in the information from that manifest inline and deletes the folder with the external manifest.
• You can then easily "ping pong" between storing manifest data inline or outline without losing any information.

A UUID feels a bit excessive in the path...

[tecosaur]

A UUID feels a bit excessive in the path...

How about <script_name>_<hash>/Manifest.toml? To me that seems like a decent blend of avoiding hash collisions while actually being somewhat informative if you peek at environments/scripts (as a freebee).

julia> let f="path/to/myscript.jl"; first(splitext(basename(f))) * '_' * string(hash(f), base=62) end
"myscript_Hkm5zPmRBLp"

A 64-bit discriminator seems plenty to me, with ~10k scripts there's a ~1 in a trillion chance of a hash collision.

Edit: I remembered case-insensitive filesystems exist. Make that string(hash(f), base=36).

[tecosaur]

Question: What do we expect julia --project=@script myscript.jl to do?

It may not make a lot of sense, but I'm sure that we'll see people try it sooner or later. My instinctive feeling is that doing anything other than using the project/manifest embedded in myscript.jl would be counterintuitive.

[KristofferC]

Yes (and it should also be what it does now). Should add a test :)

[KristofferC]

I dont think hashing only by path is good because you get a collision if you then move the script and create a new one in the same place with the same name. Since the path to the manifest is stored in the script itself it doesn't really need to have anything to do with the path (which is why I just grabbed for a uuid).

[fonsp]

Hey! This looks awesome, very cool to see this getting standardized! Later in the process, I would like to make sure that this can be made compatible with Pluto's format, so that you can activate and run Pluto notebooks in standalone Julia :)

@KristofferC asked me to share some experiences from Pluto (which already has a similar feature by default for notebook files), so here you go:

General reaction

Among Pluto's users (education, Julia end-users), people really appreciate this feature and it makes reproducibility much more accessible. It makes it much easier to share your work reliably with new Julia users, which is a huge win. Pluto enables this by default (when creating a notebook, and when opening a notebook). For this PR I would also suggest enabling it by default when opening a file with embedded project data, or at least showing a hint.

Surprises

Embedding the Project+Manifest also turned out less amazing than I thought. The main unexpected issues are:

Julia versions

In practice, the manifest is only useful with the same Julia version, and it is pretty common to change Julia versions. (E.g. when sharing work with someone else, or when opening old work.) We made https://github.com/JuliaPluto/GracefulPkg.jl to still get some value out of unresolvable environments in Pluto, but for base Julia you might want to address this.

I would be curious to hear your ideas for this issue, and maybe we can share a solution.

Setup times

Package setup times (install+precomp) are long in Julia, and they get longer with every release. Cache reuse is minimal across environments in practice, which means that each new script launch will probably trigger its own lengthy setup process. Recent Julia releases seem more optimised for the "big global env", rather than many specific environments.

This is very relevant in Pluto, where you often share your work, sometimes many notebooks. But if you use this feature less frequently, or only for personal work, then cache hit rates will be high.

[KristofferC]

In practice, the manifest is only useful with the same Julia version, and it is pretty common to change Julia versions. (E.g. when sharing work with someone else, or when opening old work.)

I agree with this. The "competition" has it easier in this case because they often don't use the language itself to launch a script (which immediately ties you to that version of the language), but instead have a launcher one abstraction level above, which can also decide what julia version to use. E.g., in uv run, uv has the choice to go and look at what Python version the script wants, and even go and download it. For julia script.jl, that isn't really possible.

We could have something like juliaup run perhaps but:

• I think it is expanding the responsibilities of juliaup a bit much.
• It is a little bit of a shame to have to download something other than julia to launch scripts like this.

Even for Pkg Apps I feel the pain of the non-flexibility of having the launcher of the app not being able to choose julia version.

(cc @davidanthoff)

[tecosaur]

but instead have a launcher one abstraction level above, which can also decide what julia version to use. E.g., in uv run, uv has the choice to go and look at what Python version the script wants, and even go and download it. For julia script.jl, that isn't really possible.

FWIW, I implemented this feature in Juliaup ~1y ago (in the julia launcher), but it wasn't merged because there weren't enough tests.

[vchuravy]

The actual julia binary (cli in the repository) is very small and is essentially a wrapper around loading libjulia. Years ago I was spit balling with Elliot the idea to put a toml parser in there and use that to choose if to start up with or without the compiler enabled or which system image, which libjulia to load.

Not something that we can do immediately, but it is feasible.

[MasonProtter]

One thought I had was if (by luck) all of the TOML spec can be parsed by the Julia parser and then we could use some @project begin idea to indicate portability (in the same way as @main is used) but stuff like 'abc' does not parse so that's not really workable. Maybe a bad idea anyway.

How about a string macro?

project"""
...
"""

manifest"""
...
"""

?

Could also have a @project begin that doesn't support / needs special handling of 'abc'

[KristofferC]

I'm now thinking that most of the decisions about portable scripts you want to take before executing the file and also if the manifest section is towards the end I'm not sure you will have macro expanded that before the code runs. And you also want the detection to be easy for other tools (like juliaup) so I don't think you get away from raw string processing to parse the inline content.

And at that point, maybe magic comments are OK...

[KristofferC]

I removed the multiline suport and added requirements of project first and manifest last.

But note @Keno, there is no "activation error" because we do not look at the TOML files until we start loading packages so you won't get any error until you actually try do a package operation. We could of course try to eagerly detect this but it wouldn't really be consistent with running files in a normal environment.

[fonsp]

Also, I wonder if a portable script should run with a restrictive load path by default (not have the global environment in there). It isn't very much portable if you rely on the global env...

This is what Pluto does and it works very well 👍

[MasonProtter]

One additional comment I have: if we're giving scripts their own project/manifests, it'd be really nice if we could also have the option to give these scripts their own precompilation images the way that packages get their own images.

When the pkgimages were new, I tested using one for some Benchmark Games scripts and found it'd result in ~20% wall time speedup in one example. It'd be great if one could use the pkgimage machinery without having to go through all the hassle of making a local package.

I'm not asking for this feature to be added in this PR, but I just want to ask if there are design decisions that should be made here that might make implementing "scriptimages" easier.

[KristofferC]

I don't think it should be too hard. I could prototype something. But I'm not sure if most people want to take on that initial latency. There is also some overlap with Pkg apps as well but maybe that is OK.

[tecosaur]

I do wonder if some sort of LRU could be helpful for pkg-related RC (and Juliaup Julia versions, but that's it's own conversation).

[KristofferC]

The discussion about precompiling this spawned a few ideas of an alternative way to implement this and I don't think targeting 1.13 is reasonable when the design space is quite a lot bigger than I originally thought.

[MasonProtter]

But I'm not sure if most people want to take on that initial latency.

It's really just a question of if it's the sort of script one expects to run ~once per machine, or many times per machine. Both are common, but the fact that it's so painful to work around julia's slow TTFX turns a lot of people off of writing "many times per machine" scripts in julia unless those scripts do a lot of work.

There is also some overlap with Pkg apps as well but maybe that is OK.

Now that you mention it, support for Pkg apps that are contained in a single-file would be really awesome 👀, but unless there's a way to make the app local to a certain directory, I don't think it can really replace scripts?

That said, if we made single-file Apps, and made an option for them to be directory local, they could potentially be the preferred way to do scripting workflows.

[KristofferC]

W.r.t compiled scripts, I don't see a way in which the script gets precompiled would still allow julia -i script.jl to access the script globals as they were defined in Main. I think there need to be some module then Main.Script where things are stored under.

[MasonProtter]

I was picturing that if there was a @main entrypoint, we could create a module and execute @main in that module during precompilation or something, but yeah I don't know enough about the details here to know what is viable or not.

[KristofferC]

I punted on precompiled portable scripts for the moment, I'll come back to it. I don't see a nice way of doing it right now without the script author wrapping the script in module.

What I changed in the last commit was that the loading for portable scripts follow the same rules as a package. I think that is correct. Even if it is inconsistent with julia --project=scripts scripts/non_portable_script.jl I think it is the right choice. In interactive work in the REPL you can always include things (just like how you can include package files) and get the normal stacked depot behavior. I also verify that the portable script have the delimiters in the correct place up front like @Keno asked.

[KristofferC]

After testing this locally, I must say that having the ability to put all the toml info at the end of the file would be quite nice. So how about if we have a

#!portable

comment in the beginning that is the definite marker for portability and then the sections can be wherever. That means we still have a concrete marker at the beginning.

[Keno]

I'm still not a fan of wherever. Either put them at the beginning or at the end, but interleaving metadata and code seems terrible.

[KristofferC]

Yeah sure we can do only first or last, just that the user decides which of them?

[Keno]

Yes fine by me (with the portable script marker)

[JeffBezanson]

I think "portable" is a very overloaded term to use for this; it's fine to call it that informally but less clear if part of the actual syntax. Maybe #!hasproject? I'm also not convinced a marker at the beginning is necessary.

[KristofferC]

UV has a # /// script marker (https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies).

We could have

#!script

?

[KristofferC]

I'mma just do

#!script

at the top for now. It is so short you can just enable it manually which is nice.

[KristofferC]

I think "portable" is a very overloaded term to use for this; it's fine to call it that informally but less clear if part of the actual syntax

The first name that came to my mind was "fat script" 😆

[JeffBezanson]

Maybe we can incorporate it into the shbang line? E.g. #!/usr/bin/env julia --script-project ?

[KristofferC]

But Windows though?