Node 22.1+ crashes process when a lot of VM processes are created/destroyed in parallel workers

[sheremet-va]

Version

22.1.0

Platform

Darwin MacBook-Air.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:16:51 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8103 arm64

Subsystem

No response

What steps will reproduce the bug?

• Clone https://github.com/vitest-dev/vitest
• Run pnpm install --prefer-offline
• Run pnpm build
• Run cd test/core
• Run pnpm test:vmThreads --run
• See a crash - sometimes it's instant, sometimes it's after a few tests ran

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior? Why is that the expected behavior?

No crash

What do you see instead?

Command failed with exit code 129. with no error.

Additional information

This is most likely an issue with Vitest itself, but I am having a hard time debugging it. The error started happening on 22.1.0, 22.0.0 works as before. The changelog didn't help me.

The issue seems to be with Worker Threads + VM as vmForks pool works fine or at least doesn't crash completely (it uses child_process under the hood instead, but the code for VM is the same) - looks like child_process just exits without crashing the main process, but it also doesn't print anything useful.

[avivkeller]

Do you happen to have a repro without any dependencies?

This is most likely an issue with Vitest itself, but I am having a hard time debugging it.

Have you tried reporting it to vitest?

[sheremet-va]

Do you happen to have a repro without any dependencies?

Unfortunately no. The behaviour seems very unreliable so at least some amount of dependencies is required. It seems like different tests just exit the process on different reruns without any errors printed to stdout/stderr.

Have you tried reporting it to vitest?

I am representing Vitest.

---

I have found NODE_DEBUG=* variable, I'll try to minimize the reproduction.

[joyeecheung]

FYI if it's getting exited deliberately by some user-land code via process.exit(), you can use --trace-exit to see what code is calling that (although for child processes, they need to inherit it i.e. the execArgv etc. should not be replaced with something else).

[avivkeller]

I haven't attempted to reproduce, but given the report, I've optimistically added these labels.

---

@sheremet-va I saw you pushed vitest-dev/vitest#6448, did this resolve the issue?

[sheremet-va]

@sheremet-va I saw you pushed vitest-dev/vitest#6448, did this resolve the issue?

No, it is a separate issue.

FYI if it's getting exited deliberately by some user-land code via process.exit(), you can use --trace-exit to see what code is calling that (although for child processes, they need to inherit it i.e. the execArgv etc. should not be replaced with something else).

I see that it hits some specific module and panics, I don't think it's the process.exit call. Haven't figured out the module itself yet, but it's loaded by vm.runInContext. I will investigate more tomorrow. Just child_process or worker_thread that loads modules normally are not affected.

[sheremet-va]

I wasn't able to create a small reproduction, but I've made it easier to debug: vitest-dev/vitest#6452 You can see in the console the test that failed and it creates a file with stdout logs.

I am still not sure what is happening and why Node crashes with a custom module runner based on VM context. And just a reminder - it started happening in Node 22.1, 22.0 works without an issue.

[AriPerkkio]

How can we debug this further? Below I've tried to narrow down the working-to-crashing versions and attached releases notes for each. There's also core dump if that helps.

• v23.0.0 works, v23.1.0 crashes easily. Crash happens always.
• v22.0.0 works, v22.1.0 crashes easily. Crash happens always.
• v20.16.0 works, v20.17.0 crashes after some time. Crash happens always.
• Note that crash does not happen if we don't run multiple threads or processes parallel.

Is there anything similar in these releases?

• https://nodejs.org/en/blog/release/v23.1.0
• https://nodejs.org/en/blog/release/v22.1.0
• https://nodejs.org/en/blog/release/v20.17.0

This is output of (lldb) bt, but at this point I have no idea what I'm doing. 🙃

Core dump

0x0000000100b0b2d4 node`node::loader::ModuleWrap::SyntheticModuleEvaluationStepsCallback(v8::Local<v8::Context>, v8::Local<v8::Module>) + 136
0x0000000101267698 node`v8::internal::SyntheticModule::Evaluate(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SyntheticModule>) + 112
0x000000010120f698 node`v8::internal::Module::Evaluate(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Module>) + 380
0x0000000101253164 node`v8::internal::SourceTextModule::InnerModuleEvaluation(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>, v8::internal::ZoneForwardList<v8::internal::Handle<v8::internal::SourceTextModule>>*, unsigned int*) + 600
0x0000000101252dec node`v8::internal::SourceTextModule::Evaluate(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SourceTextModule>) + 240
0x000000010120f63c node`v8::internal::Module::Evaluate(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Module>) + 288
0x0000000100d876e4 node`v8::Module::Evaluate(v8::Local<v8::Context>) + 580
0x0000000100b0c60c node`node::loader::ModuleWrap::Evaluate(v8::FunctionCallbackInfo<v8::Value> const&) + 1064
0x0000000101732f98 node`Builtins_CallApiCallbackGeneric + 184
0x0000000107df7080
0x0000000107df7684
0x000000010176d290 node`Builtins_AsyncFunctionAwaitResolveClosure + 80
0x00000001018384d8 node`Builtins_PromiseFulfillReactionJob + 56
0x000000010175d594 node`Builtins_RunMicrotasks + 564
0x000000010172eaf4 node`Builtins_JSRunMicrotasksEntry + 148
0x0000000100ec6d7c node`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 1612
0x0000000100ec7620 node`v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 88
0x0000000100ec7758 node`v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) + 60
0x0000000100ef2304 node`v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) + 356
0x0000000100ef2ae8 node`v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) + 132
0x0000000101732f98 node`Builtins_CallApiCallbackGeneric + 184
0x0000000107da7574
0x000000010172ec0c node`Builtins_JSEntryTrampoline + 172
0x000000010172e8f4 node`Builtins_JSEntry + 148
0x0000000100ec6dbc node`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 1676
0x0000000100ec6700 node`v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 120
0x0000000100d97520 node`v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 532
0x0000000100a75a6c node`node::InternalCallbackScope::Close() + 552
0x0000000100a75cac node`node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context, v8::Local<v8::Value>) + 468
0x0000000100a8ab2c node`node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) + 256
0x0000000100c62210 node`node::StreamBase::CallJSOnreadMethod(long, v8::Local<v8::ArrayBuffer>, unsigned long, node::StreamBase::StreamBaseJSChecks) + 228
0x0000000100c64410 node`node::EmitToJSStreamListener::OnStreamRead(long, uv_buf_t const&) + 380
0x0000000100c68a3c node`node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) + 692
0x0000000100c69144 node`node::LibuvStreamWrap::ReadStart()::$_1::__invoke(uv_stream_s*, long, uv_buf_t const*) + 88
0x000000010171a280 node`uv__stream_io + 1028
0x0000000101721ffc node`uv__io_poll + 1024
0x000000010170f4a4 node`uv_run + 412
0x0000000100a764e4 node`node::SpinEventLoopInternal(node::Environment*) + 256
0x0000000100ba1cf4 node`node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) + 184
0x0000000100ba1a90 node`node::NodeMainInstance::Run() + 124
0x0000000100b1559c node`node::Start(int, char**) + 652
0x00000001863ac274 dyld`start + 2840

[AriPerkkio]

What do you see instead?

Command failed with exit code 129. with no error.

I'm seeing exit code 139 and Segmentation fault: 11 when using threads, not processes. 139 indicates segfault.