TTFX regression with Comonicon CLI on Julia 1.11

[timholy]

Following up on a slack conversation, there is indeed a substantial regression in the responsivity of a CLI application written with Comonicon:

Julia 1.10:

~/.julia/bin$ time ./cli --help
# lots of output
real    0m0.285s
user    0m0.395s
sys     0m0.642s

Julia 1.11-rc1:

real    0m3.681s
user    0m3.656s
sys     0m0.866s

I haven't really participated in the Julia 1.11 release and don't have time to check myself, but since it's a pretty sizable regression I'm slapping a milestone on it. (I am not personally affected by CLI TTFX, though.)

Here are the specifics of what I tested:

tim@diva:/tmp/TTFX/CLI$ tree
.
├── Manifest.toml
├── Project.toml
└── src
    └── CLI.jl

1 directory, 3 files
tim@diva:/tmp/TTFX/CLI$ cat src/CLI.jl
module CLI

using Comonicon

"""
ArgParse example implemented in Comonicon.

# Arguments

- `x`: an argument

# Options

- `--opt1 <arg>`: an option
- `-o, --opt2 <arg>`: another option

# Flags

- `-f, --flag`: a flag
"""
Comonicon.@main function main(x; opt1=1, opt2::Int=2, flag=false)
    println("Parsed args:")
    println("flag=>", flag)
    println("arg=>", x)
    println("opt1=>", opt1)
    println("opt2=>", opt2)
end

end # module CLI

Build with

(CLI) pkg> build

and then test with

~/.julia/bin$ time ./cli --help

[timholy]

Oh, and if anyone is looking into this, note that the master branch of SnoopCompile works on Julia 1.11. See this issue for progress on the release: JuliaDebug/SnoopCompile.jl#382

[IanButterworth]

I tried to replicate this but

(CLI) pkg> st
Project CLI v0.1.0
Status `~/Documents/GitHub/CLI/Project.toml`
  [863f3e99] Comonicon v1.0.8

(CLI) pkg> build

shell> ls ~/.julia/bin
ls: /Users/ian/.julia/bin: No such file or directory

If anyone familiar with Comonicon can help I'd be happy to investigate (@timholy mentioned on slack that he wasn't sure why this didn't build ~/.julia/bin/cli)

[Sbozzolo]

I don't know if this is related, but we are seeing massive latency increases in ClimaAtmos.jl with Julia 1.11-rc1 (CliMA/ClimaAtmos.jl#3186).

I was led to this issue through SnoopCompile. ClimaAtmos.jl does no longer compile in any reasonable time with Julia 1.11-rc1 (or previous betas). I identified that the offending function is get_atmos, a simple function that returns a keyword-defined type. For the most part, this function simply converts string keywords read from YAML files to types, then, it returns a AtmosModel.

I used SnoopCompile (master) to try to get a little more insight. The get_atmos function, as it is, takes too long to compile (on Julia 1.10, this is instantaneous), so I had to remove part of it (as mentioned in the referenced issue). Once I do so, I see with @snoop_inference that:

InferenceTimingNode: 1.171199/288.711156 on Core.Compiler.Timings.ROOT() with 21 direct children

Plotting the inference:

All the time is spent in types.jl line 329. This line simply defines our struct:

Base.@kwdef struct AtmosModel{
    MC,
    MM,
    PM,
    CM,
    CCDPS,
    F,
    S,
    RM,
    LA,
    EXTFORCING,
    EC,
    AT,
    TM,
    EEM,
    EDM,
    ESMF,
    ESDF,
    ENP,
    EVR,
    TCM,
    NOGW,
    OGW,
    HD,
    VD,
    DM,
    SAM,
    VS,
    RS,
    ST,
    IN,
    SM,
    SA,
    NUM,
}
    model_config::MC = nothing
    moisture_model::MM = nothing
    precip_model::PM = nothing
    cloud_model::CM = nothing
    call_cloud_diagnostics_per_stage::CCDPS = nothing
    forcing_type::F = nothing
    subsidence::S = nothing
    radiation_mode::RM = nothing
    ls_adv::LA = nothing
    external_forcing::EXTFORCING = nothing
    edmf_coriolis::EC = nothing
    advection_test::AT = nothing
    tendency_model::TM = nothing
    edmfx_entr_model::EEM = nothing
    edmfx_detr_model::EDM = nothing
    edmfx_sgs_mass_flux::ESMF = nothing
    edmfx_sgs_diffusive_flux::ESDF = nothing
    edmfx_nh_pressure::ENP = nothing
    edmfx_filter::EVR = nothing
    turbconv_model::TCM = nothing
    non_orographic_gravity_wave::NOGW = nothing
    orographic_gravity_wave::OGW = nothing
    hyperdiff::HD = nothing
    vert_diff::VD = nothing
    diff_mode::DM = nothing
    sgs_adv_mode::SAM = nothing
    viscous_sponge::VS = nothing
    rayleigh_sponge::RS = nothing
    sfc_temperature::ST = nothing
    insolation::IN = nothing
    surface_model::SM = nothing
    surface_albedo::SA = nothing
    numerics::NUM = nothing
end

I have not tried without keyword arguments.

I don't think that what we are doing up to this point is particularly complex or unorthodox, and most of the types involved are very simple (mostly singletons and bools, all immutable).

Should I open a new issue for this?

[KristofferC]

Yes, please open a new issue since it is unlikely that it is related to this issue.

[Roger-luo]

Hi you can try the example package here: https://github.com/comonicon/Comonicon.jl/tree/main/example/Hello

and you can install the entry point by running Hello.comonicon_install()

help?> Hello.comonicon_install
  comonicon_install(;kwargs...)

  Install the CLI manually. This will use the default configuration in
  Comonicon.toml, if it exists. For more detailed reference, please refer to
  Comonicon documentation (https://comonicon.org).

you don't have to test that with an entry point, you can test directly with its Julia entry point

help?> Hello.command_main
  No documentation found.

  Hello.command_main is a Function.

  # 2 methods for generic function "command_main" from Hello:
   [1] command_main(ARGS::Vector{String})
       @ ~/Code/Julia/Comonicon/src/codegen/julia.jl:90
   [2] command_main()
       @ ~/Code/Julia/Comonicon/src/codegen/julia.jl:90

[KristofferC]

I get:

# 1.10
hello --help  0.37s user 0.31s system 308% cpu 0.219 total

# 1.11
hello --help  1.01s user 1.58s system 354% cpu 0.731 total

And considering both Pkg and REPL is loaded, this does not seem very surprising.

[timholy]

Hmm, it was much worse for me, I wonder why. Agreed this seems more reasonable.

Is it all TTL? There seems to be a path forward that Jameson and I sketched out for significantly reducing TTL, but it will take some time to explore. Therefore, let me ask a heretical question: if Pkg & REPL are the two culprits and they come up a lot, how important for 1.11 is their exclusion from the sysimg? Clearly we want this eventually, but until juliac is out, I guess I'm unclear on whether the advantages of their exclusion more than compensate for the disadvantages. Something that might be worth considering is "baking them in" for 1.11 (but hopefully that's just a couple of new lines to re-include them, I'm not suggesting we revert any hard work) with the expectation that they'll be excluded in a future release where either the TTL issue has been fixed or the advantages of their exclusion clearly outweigh the disadvantages.

If "putting them back" is not just a couple of lines, then you can ignore this.

[IanButterworth]

@timholy whats your platform? WSL2?

[KristofferC]

Regarding this specific case, it seems "self-inflicted". The Hello app here depends on the full Comonicon.jl which pulls in a lot of dependencies (Pkg, REPL) to both do argument parsing and to add a Hello.comonicon_install command inside the app itself. To me, this would better be structured as having Comonicon install the app (Comonicon.install("examples/Hello")) and having a smaller dependency that handles argument parsing that the app itself would depend on.

[KristofferC]

As of right now, just adding Pkg to the sysimage doesn't work:

LoadError("sysimg.jl", 57, LoadError("/Users/kristoffercarlsson/julia/usr/share/julia/stdlib/v1.12/Pkg/src/Pkg.jl", 3, 
  LoadError("/Users/kristoffercarlsson/julia/usr/share/julia/stdlib/v1.12/Pkg/src/precompile.jl", 101, 
    Base.IOError("could not spawn setenv(`'' a registries/Registry.tar.gz -tgzip

which I guess is from https://github.com/JuliaLang/Pkg.jl/blob/da9f89ffc7df973d273a3952051ad656987bc64b/src/PlatformEngines.jl#L20 and that jlls don't work in the process creating a sysimage since __init__ is disabled in that process.