TOML config support for JBang

[wfouche]

I have been working with Python's PEP 723 inline script metadata format and have come to like it a lot. It is simply a TOML data structure encoded in comments. Adapted to JBang it would allow the JBang configuration for Java / Kotlin / Groovy to be specified as shown in the example below:

///usr/bin/env jbang "$0" "$@" ; exit $?

// ||| jbang
// requires-java = ">=21"
// dependencies = [
//   "org.springframework.boot:spring-boot-starter-web:3.4.4",
// ]
// [java]
//   runtime-options = "-server -Xms2g -Xmx2g -XX:+UseZGC -XX:+ZGenerational"
// |||

import static java.lang.System.*;

public class test {

    public static void main(String... args) {
        out.println("Hello World");
    }
}

What are the benefits of this approach?

0. Standardized configuration format (TOML)
1. Easier to parse the configuration data. The Toml block is extracted and then processed by a Toml library (org.tomlj:tomlj:1.1.1).
2. Extensible configuration format, new options can easily be added.
3. Additional (optional) support could even be provided to externalize the JBang configuration to a file which could then be included in the JBang scriipt. But this will also make it slighly harder to determine if a script should be recompiled (the source did not change, but the config did).

///usr/bin/env jbang "$0" "$@" ; exit $?

// ||| jbang
// jbang-config = "config.toml"
// |||

import static java.lang.System.*;

public class test {

    public static void main(String... args) {
        out.println("Hello World");
    }
}

4. By sharing a configuration file, several related scripts could use the same runtime options that are specified in one place.
5. Editing and syntax highlighting support of TOML configuration files via extensions for VSCode, Intellij, etc.

6. A --config option could be added to JBang to override the configuration of a JBang script to use a different config.toml file.
7. A --config option would also allow files that do not contain any JBang comments to be provided with configuration data at runtime (if required).

// Run with JBang as:  jbang --config config.toml run  test.java
import static java.lang.System.*;

public class test {

    public static void main(String... args) {
        out.println("Hello World");
    }
}

8. Migration support could be provided to rewrite a JBang script file and group the JBang //? config options into a single TOML config comment block or to write the config to a TOML file.
9. If a external config file is used for the JBang config options, then script processing time would be reduced because the script does not have to be scanned for information.

[wfouche]

Java code to parse a TOML-encoded configuration comment block for JBang scripts.

//DEPS org.tomlj:tomlj:1.1.1

import org.tomlj.TomlParseResult;

StringBuffer tomlText = new StringBuffer("");
{
    List<String> lines = Files.readAllLines(Paths.get(scriptFilename));
    boolean found = false;
    for (String line: lines) {
        if (line.startsWith("// ||| jbang")) {
            found = true;
        }
        else if (line.startsWith("// |||")) {
            found = false;
            break;
        } else if (line.startsWith("// ")) {
            if (found) {
                if (tomlText.length() == 0) {
                    tomlText.append(line.substring(3));
                } else {
                    tomlText.append("\n" + line.substring(3));
                }
            }
        }
    }
}
TomlParseResult tpr = Toml.parse(tomlText.toString());

[quintesse]

Personally I don't really find it an improvement in readability :-)

And it would definitely break all backward-compatibility.

[wfouche]

Personally I don't really find it an improvement in readability :-)

Opinions will definitely differ on this. :-)

Config data expressed in a well-known standardized format is easier to comprehend than an invented-on-the-spot alternative. Toml is also easier to process and is extensible which is a huge benefit.

GraalPython with its JBangIntegration.java extension adds //PIP and //PIP_DROP config options. The current configuration paradigm limits extensibility.

Config options are currently (can be) scattered throughout the script. With TOML support all the config information must grouped into the same text block (enhancing readability).

JBang 1.0 could potentially support this functionality.

"Where there's a will, there's a way!"

[quintesse]

I don't think changing the format is an option at this late a stage. JBang might not be 1.0 yet, but throwing away 4+ years of stability in that aspect doesn't seem like a thing our users would appreciate. And supporting both formats would only complicate things.

Also, the current paradigm is completely flexible, the current tags are not fixed and in fact there are several projects out there that add their own.

And currently the tags can be scattered throughout the script, but in actuality nobody does so. (I did suggest in the past to stop parsing tags once the first non-tag line is found, basically assuring the lines must always be kept together, but that suggestion wasn't followed up upon).

[wfouche]

@quintesse, (just to make it clear) we are having two intertwined discussions.

(1) What is the best way to represent configuration options for a tool such as JBang. This is a purely abstract and scientific discussion. If we can rewind the clock and the first few lines of code were to be written today for JBang, then given what we know currently, would we encode dependencies as //DEPS or use the TOML list/array format for dependencies, or pick something even better than TOML? It is nice to have access to a hierarchal namespace (as supported by TOML) which makes it easier to extend the data set that is represented, so I'm of the opinion that using TOML is a better solution than using the current JBang config notation. It is important to have this type of discussion and gain valuable insights. If not to be used in JBang, the knowledge gained might be of use on another project.

(2) This is the where we face reality and weigh our options and in the context of JBang and its development history and user base, does it make any sense to change the way config options are represented? Maybe the ship has sailed on this, and it is just too late. Maybe not. Java 1.0 supported applets and it was an important use case. Applets are no longer supported in the Java language we use today. Software systems change and evolve over time. So will JBang change and evolve in the many years ahead.

[wfouche]

Option [jbang].integrations=false was added to the GraalPy config. Seems more readable to me than //NOINTEGRATIONS.

#!/usr/bin/env python-jvm

# restclient_graalpy.py
#
# /// jbang
# requires-graalpy = "==24.2.1"
# requires-java = ">=21"
# dependencies = [
#   "org.springframework.boot:spring-boot-starter-web:3.4.4"
# ]
# [graalpy]
#   allowAllAccess = true
#   emulateJython = true
# [java]
#   runtime-options = "-XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI -Dpolyglot.engine.WarnInterpreterOnly=false"
# [jbang]
#   integrations = false
# ///

import java.lang.String as String
import org.springframework.web.client.RestClient as RestClient

def restApiCall(uri: str, id: int):
    restClient = RestClient.create()
    rsp = restClient.get().uri(uri, id).retrieve().body(String)
    print(rsp)

def main():
    restApiCall("https://jsonplaceholder.typicode.com/todos/{id}", 1)

main()

[quintesse]

We tend to create flags whose default value is "false" and can be set to "true" to enable an "exceptional/non-default" situation.
That's why it's called "no integrations", because by default integrations are enabled and disabling them is "exceptional".

[maxandersen]

so I've been pondering about this issue for the last few days and wondered if my gut reaction of "no - this is too much" was right or not.

First, I think what you've done for python support in jbang by utilizing PEP 723 makes perfect sense!
Its though quite critical to recognize the reason toml is used there is because of the following from the PEP:

The metadata format is designed to be similar to the layout of data in the pyproject.toml file of a Python project directory, to provide a familiar experience for users who have experience writing Python projects. By using a similar format, we avoid unnecessary inconsistency between packaging tools, a common frustration expressed by users in the recent packaging survey.

i.e.:

• to not introduce yet another format in Python
• because .toml used by a popular project descriptor format in Python

Second, I think if I had kicked of jbang in early days and using this format i'm pretty sure it would been seen as a detractor given toml is not a typical thing in Java world. JBang's comment driven syntax is done that way to simply not interfere with existing tooling and IDE's. a blob of toml does also survive in java code but it does feel alien.

Thirdly, I deliberately did not go for using a tree-based syntax (json,yaml, etc.) for jbang but quite a flat one to limit the amount of different ways I knew we would come up with enabling :) we did use .json for configuration; mainly because it was simplest/smallest depenendency to support.

So I'm not a fan of adding this as a "main" option as it:

• introduces yet another format
• .toml is not commonly used in java
• requires more dependencies

That said - we still haven't found a good format for jbang.build files so I'm not closing the door completely but for just replacing or alternative to //<directive> <value> embedded in java syntax I'm -1.

[wfouche]

@maxandersen , good response and I'm agree with the points you raise.

I just realized today that JBang also support a jbang.properties file that can be used to provide configuration information for scripts. I need to understand jbang.properties a bit better but it looks promising. jbang.build files are new to me, will have look at them as well.

[wfouche]

That said - we still haven't found a good format for jbang.build files so I'm not closing the door completely

Apple's Pkl language is also an interesting option to consider.

https://pkl-lang.org/

[quintesse]

To me that looks like yet another markup language... so perhaps it should be called yaml2 :-)
Joking aside, it's really yet another format, what are the great advantages over all the others that exist?
And it's not only about how good it is, but also: will my users want to go through the trouble of learning it?

[quintesse]

we did use .json for configuration

I'm not sure if you meant to say this or if you're referring to something else, but JBang's config files are Java property files. (The ones supported by java.util.Properties)

Edit: perhaps you're referring to the catalogs? Those are .json files, yes.

[quintesse]

jbang.build files are new to me

They have been a preview feature for some time now, so they will only work if you use the --preview command line option.

It's format is extremely simple: cut all the //-tag lines out of a JBang script and paste them into a file called jbang.build. Then remove the // from the start of each line.
The only other difference is that, because it's not inside a source file, you must specify at least one SOURCE line (for now at least).

The format is the same so as not to have to learn anything new (which is also why I wouldn't look for some other format to use @maxandersen).
And is meant for those cases where you really don't want to add weird comments to your source files.

[wfouche]

Scanning through the JBang documentation I collected all the // names that I could find. The documentaton do not list all the tags that are currently defined (something we can improve on).

Toml is used in Gradle projects these days to specificy versions of artifacts to use; see file gradle/libs.versions.toml

Proposal for jbang.build.toml file layout.

$ jbang run --build-config jbang.build.toml Script.java

# jbang.build.toml
[jbang]
  description = [
  "The best hello world application in the",
  "spiral galaxy that we find ourselves in"
  ]

  dependencies = [
    "io.leego:banana:2.1.0",
    "org.slf4j:slf4j-api:2.0.17",
    "ch.qos.logback:logback-core:1.5.18",
    "ch.qos.logback:logback-classic:1.5.18"
    ]

  repos = [
    "mavencentral,acme=https://maven.acme.local/maven",
    "...."
    ]

  sources = [
    "JavaClass1.java",
    "JavaClass2.java",
    "JavaClass3.java"
    ]

  files = [
    "src/main/resources/config.json",
    "src/main/resources/app.properties"
    ]

  preview = true

  nointegrations = true

  main = "<main-class-name>"

  module = "..."

  [manifest]
    key1 = "value1"
    key2 = "value2"
    key3 = "value3"

[java]
  version = "21+"
  compile-options = "-g"
  runtime-options = "..."
  native-options = "..."
  javaagent = "..."

[kotlin]
  version = "2.1.20"
  compile-options = "-progressive"
  runtime-options = "..."

[groovy]
  version = ""
  compile-options = "--tolerance=5"
  runtime-options = "..."