ESM flag breaks NodeJS child process `fork`

[devversion]

Search Terms

child process, fork, esm, loader, brotli, config, exec argv, exec path

Expected Behavior

Similar to when running with node --loader ts-node/esm or plain node, the main process should be able to fork and execute other NodeJS scripts (not necessarily another TS file).

When vanilla Node executes, the execArgv never contains the script entry-point, so that the fork can set the new script that is intended to be run.

Actual Behavior

Using the builtin NodeJS fork method does not work when running with ts-node --esm because the process.execPath and process.execArgv contain the brotli-base64 encoded config and the custom loader script. This is resulting in the same script being executed all the time, going into an infinite loop based on the script.

Steps to reproduce the problem

1. Create a TS ESM file e.g. .mts
2. Use NodeJS fork and try to load another file, e.g. a plain worker.mjs file
3. Observe that the worker script is never executing, but instead the orignal .mts file is executed again, and so on.

Minimal reproduction

1. Clone https://github.com/devversion/ts-node-esm-fork/tree/main
2. Run yarn test

Specifications

ts-node v10.8.1
node v16.14.2
compiler v4.7.4
Done in 0.35s.

• Linux 64bit

[cspotcode]

Thanks for your analysis of the problem. It sounds like we should pass some amount of configuration to the child process so that our hooks are installed into the child. But we should not be passing our own entrypoint script. This will be tricky because we need to initialize our hooks. Perhaps they can be initialized with `--require`?

…

On Thu, Jun 23, 2022, 9:28 AM Paul Gschwendtner ***@***.***> wrote: Search Terms child process, fork, esm, loader, brotli, config, exec argv, exec path Expected Behavior Similar to when running with node --loader ts-node/esm or plain node, the main process should be able to fork and execute other NodeJS scripts (not necessarily another TS file). When vanilla Node executes, the execArgv never contains the script entry-point, so that the fork can set the new script that is intended to be run. Actual Behavior Using the builtin NodeJS fork method does not work when running with ts-node --esm because the process.execPath and process.execArgv contain the brotli-base64 encoded config and the custom loader script. This is resulting in the same script being executed all the time, going into an infinite loop based on the script. Steps to reproduce the problem 1. Create a TS ESM file e.g. .mts 2. Use NodeJS fork and try to load another file, e.g. a plain worker.mjs file 3. Observe that the worker script is never executing, but instead the orignal .mts file is executed again, and so on. Minimal reproduction 1. Clone https://github.com/devversion/ts-node-esm-fork/tree/main 2. Run yarn test Specifications ts-node v10.8.1 node v16.14.2 compiler v4.7.4 Done in 0.35s. - Linux 64bit — Reply to this email directly, view it on GitHub <#1812>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OG5A5R7JWIRUZIDB3DVQRQ7LANCNFSM5ZULV2RA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[devversion]

Yeah, I haven't looked a lot into the implementation of the child process instantiation and how the bootstrap state works, but it seems like it could also be possible to split up the state into two parts. i.e. the entry-point <-> and the rest of the state. Then the execArgv could be tweaked manually (as it looks like it will be modified anyway) to omit the entry-point part.

[cspotcode]

Now that I think about this again, I think the execArgv is meant to be passed to the child process verbatim, and then argv[1] is meant to be replaced with whatever script you want to run in the child process. But I haven't double-checked that. Might be useful to get copies of all the relevant values logged and posted into this ticket, so we have something concrete to look at.

…

On Thu, Jun 23, 2022, 10:38 AM Paul Gschwendtner ***@***.***> wrote: Yeah, I haven't looked a lot into the implementation of the child process instantiation and how the bootstrap state works, but it seems like it could also be possible to split up the state into two parts. i.e. the entry-point <-> and the rest of the state. Then the execArgv could be tweaked manually (as it looks like it will be modified anyway) to omit the entry-point part. — Reply to this email directly, view it on GitHub <#1812 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC35OGEAHS53KUAVFC4UZDVQRZFNANCNFSM5ZULV2RA> . You are receiving this because you commented.Message ID: ***@***.***>

[devversion]

Yes, that is correct. The problem is that the entry-point is encoded into the execArgv as part of the brotli string.

Main script
Exec path /usr/local/google/home/pgschwendtner/bin/node/bin/node
Exec argv [
  '--require',
  '/usr/local/google/home/pgschwendtner/projects/ts-node-fork-esm/node_modules/ts-node/dist/child/child-require.js',
  '--loader',
  'file:///usr/local/google/home/pgschwendtner/projects/ts-node-fork-esm/node_modules/ts-node/child-loader.mjs',
  '/usr/local/google/home/pgschwendtner/projects/ts-node-fork-esm/node_modules/ts-node/dist/child/child-entrypoint.js',
  '--brotli-base64-config=GxYDIBwFbqzNKwWBK0WRZy2ytwVT0y3l67cmCDeJJuiN9+0BimbR6eo7ddM9pd+taU8NXOdtWhRIF0c6JjBlThcFHaaSxHx3OhV1qJfqeamOrnSXQKghsDYp31UUYFUp2F8O4aObUTXirzDqFc9Bzfh1e2t7d7OoDzS5/Wg71x/34jptFGgxT9Fgj+0kYz3tfhZgaxTu4d3EEH6pxnz0+MgXjiTO/wOJK941tQqAO/CUuoiBiWHoVx44QHLTbyaMu5OFe1BNbR1CT+w7RTZA0wFdr5+C10FjzA7xbBJGFLN0hEMJY9D5zx3Yy1pNn2toXSFzJopFHhAHog6M8dM+K6yddvmyzddyZwJfB0Br4zpmYqP1zg8='
]

The brotli config is the following decompressed:

{
    "shouldUseChildProcess": true,
    "isInChildProcess": false,
    "entrypoint": "HOME/projects/ts-node-fork-esm/node_modules/ts-node/dist/bin.js",
    "parseArgvResult": {
        "argv": [
            "HOME/bin/node/bin/node",
            "HOME/projects/ts-node-fork-esm/node_modules/.bin/ts-node",
            "--esm",
            "--project",
            "tsconfig.json",
            "index.mts"
        ],
        "restArgs": [
            "index.mts"
        ],
        "help": false,
        "version": 0,
        "argsRequire": [],
        "print": false,
        "interactive": false,
        "project": "tsconfig.json",
        "esm": true
    },
    "phase2Result": {
        "executeEval": false,
        "executeEntrypoint": true,
        "executeRepl": false,
        "executeStdin": false,
        "cwd": "HOME/projects/ts-node-fork-esm",
        "scriptPath": "HOME/projects/ts-node-fork-esm/index.mts"
    }
}

[cspotcode]

I see, makes sense. As you say, the solution is to pass the entrypoint as a positional argument outside of the brotli payload, so that fork() and related workflows can overwrite it, and ts-node's bootstrapping logic will accept the overwritten entrypoint and execute it.

We will definitely accept a PR with this fix. I'm available either here or on Discord to offer guidance to whomever wants to tackle this. Our tests are pretty good around this stuff, so I wouldn't be afraid to propose a solution via draft pull request.

[devversion]

@cspotcode I attempted to fix this. When you have a chance, please have a look at #1814.