Use node version 22 to run wasm based tests

[anutosh491]

Description

Refer #398 (comment)

Type of change

Please tick all options which are relevant.

• Bug fix
• New feature
• Added/removed dependencies
• Required documentation updates

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.77%. Comparing base (190caf6) to head (426be21).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #400   +/-   ##
=======================================
  Coverage   81.77%   81.77%           
=======================================
  Files          20       20           
  Lines         845      845           
  Branches       87       87           
=======================================
  Hits          691      691           
  Misses        154      154           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[anutosh491]

Okay I am just confused here.

The test C++ emscripten job shows the following at the 40 second mark itself.

===============================================================================
[doctest] test cases: 45 | 45 passed | 0 failed | 0 skipped
[doctest] assertions: 72 | 71 passed | 1 failed |
[doctest] Status: SUCCESS!

I'm not sure what is it waiting for there after. I added the exit runtime flag to force the exit once main runs but that didn't help

The emscripten_wasm (osx15-arm, macos-15) jobs times out not doing anything after showing success and just runs till the 4 minute mark. Even other jobs like deploy run for 4 mins even though the tests have already been executed.

I don't expect that to happen through node (atleast doesn't happen locally ... I always build the .js and not the .html through emrun)

@mcbarton as you introduced the testing in the browser thingy, do you know what's happening here ?

The native kernel tests pass under a minute, I see no reason why the wasm kernel tests should take more than a minute in that case !

[mcbarton]

Okay I am just confused here.

The test C++ emscripten job shows the following at the 40 second mark itself.

===============================================================================
[doctest] test cases: 45 | 45 passed | 0 failed | 0 skipped
[doctest] assertions: 72 | 71 passed | 1 failed |
[doctest] Status: SUCCESS!

I'm not sure what is it waiting for there after. I added the exit runtime flag to force the exit once main runs but that didn't help

The emscripten_wasm (osx15-arm, macos-15) jobs times out not doing anything after showing success and just runs till the 4 minute mark. Even other jobs like deploy run for 4 mins even though the tests have already been executed.

I don't expect that to happen through node (atleast doesn't happen locally ... I always build the .js and not the .html through emrun)

@mcbarton as you introduced the testing in the browser thingy, do you know what's happening here ?

The native kernel tests pass under a minute, I see no reason why the wasm kernel tests should take more than a minute in that case !

As far as I can tell it is running the browser tests fine. Its when it uses the node where it stalls (put an echo command such as 'running Emscripten C++ tests in node' just before the node command). This stalling when trying to run the tests in node has happens to me locally when it tries to use emsdk 3.1.73 old version of node. I think there is a strong possibility it is not using the node 24 you install (maybe try a node --version command after you do the echo command). The Emscripten C++ tests can run with the old node emsdk currently uses, but it takes a long time (on the ci runners you have to remove the timeout and it takes about 20 minutes before it starts running the C++ tests in node). With the latest node it runs instantly for me.

[mcbarton]

@anutosh491 To see how long it takes for the Emscripten C++ tests to run with emsdk 3.1.73 version of node see this PR I have open #386 . This should all be fixed when we upgrade to emsdk version 4

[anutosh491]

As far as I can tell it is running the browser tests fine. Its when it uses the node where it stalls (put an echo command such as 'running Emscripten C++ tests in node' just before the node command). This stalling when trying to run the tests in node has happens to me locally when it tries to use emsdk 3.1.73 old version of node. I think there is a strong possibility it is not using the node 24 you install (maybe try a node --version command after you do the echo command). The Emscripten C++ tests can run with the old node emsdk currently uses, but it takes a long time (on the ci runners you have to remove the timeout and it takes about 20 minutes before it starts running the C++ tests in node). With the latest node it runs instantly for me.

Hmm, I still don't understand if you're proposing something to be done here ? Do you have a current way ahead ?

[mcbarton]

As far as I can tell it is running the browser tests fine. Its when it uses the node where it stalls (put an echo command such as 'running Emscripten C++ tests in node' just before the node command). This stalling when trying to run the tests in node has happens to me locally when it tries to use emsdk 3.1.73 old version of node. I think there is a strong possibility it is not using the node 24 you install (maybe try a node --version command after you do the echo command). The Emscripten C++ tests can run with the old node emsdk currently uses, but it takes a long time (on the ci runners you have to remove the timeout and it takes about 20 minutes before it starts running the C++ tests in node). With the latest node it runs instantly for me.

Hmm, I still don't understand if you're proposing something to be done here ?

If you see the new ci output it prints node version v16.20.2 . What is happening is your installing a new node version, but with the changes you've made to the ci its now using the version which comes with emsdk version 3.1.73 which is quite old. Instead of using the action, get emsdk 3.1.73 in Emscripten forge to install node 24 using emsdk install node-24.7.0-64bit and it should be be able to run within the 4 minute timeout. Otherwise remove the time limit and the tests will run in that old version eventually.

[anutosh491]

Not sure that works for me locally

anutosh491@Anutoshs-MacBook-Air emsdk % ./emsdk install node-24.7.0-64bit
error: tool or SDK not found: 'node-24.7.0-64bit'

[mcbarton]

Not sure that works for me locally

anutosh491@Anutoshs-MacBook-Air emsdk % ./emsdk install node-24.7.0-64bit
error: tool or SDK not found: 'node-24.7.0-64bit'

This is working on my Macbook, so unsure why your Macbook doesn't accept it

./emsdk/emsdk install node-24.7.0-64bit
Installing tool 'node-24.7.0-64bit'..

[anutosh491]

Weird, would you like to take over ?

The goal here was to upgrade the node version cause node 20 gives a memory out of bounds error for #398 , whereas there is nothing wrong with the change/test added there. Upgrading to 23/24 should fix that !

[anutosh491]

cc @mcbarton we didn't realize we could simply fetch nodejs24 from conda-forge and address this (makes it a 2 line change and I'll update the readme too)

[mcbarton]

@anutosh491 Is node 22 is not new enough? I would prefer the emscripten forge recipes PR goes in now its been approved, and we use the node from the build environment. If you need node 24, then maybe update the emsdk recipe to be node 24 rather than node 22 like I set it. You could fix 2 tasks at the same time if your willing to approve #386 , since that uses node from tue build environment (the node being whatever you set it to in the emscripten forge PR), and gives xeus-cpp to have a target which automatically runs the c++ tests in node

[anutosh491]

Is node 22 is not new enough?

Anything above node 20 is !

[anutosh491]

@mcbarton I'm all confused after something went in that I'm not aware of. Due to which now I see resolve conflicts. Could you tell me what's the appropriate way to resolve this. I still want to make use of nodejs 22 from conda-forge.

[mcbarton]

We should not be running our tests through a node version added to the xeus-lite-host environment. We should be using the same one which is used during the configure stage of our Emscripten build. Without this your using node 16 through the CMAKE_CROSSCOMPILING_EMULATOR , and then running your tests in node 22. This inconsistency could cause our tests to fail for random reasons which would be hard to debug, and would almost certainly get worse as the node from the build environment gets more and more outdated. What needs to be done is for the node used in the Emscripten forge recipe to be updated to a newer version (to whatever matches the version the upstream emsdk), and a mechanism should be put in place that keeps the MacOS emsdk provided by Emscripten forge upto date with the Linux version.

Although I have no evidence towards this, the emsdk versions not being kept in sync could explain why sometimes in the past we have seen cases where we have had Emscripten jobs fail for one system, but not the other (see https://github.com/compiler-research/xeus-cpp/actions/runs/14780201517 and https://github.com/compiler-research/xeus-cpp/actions/runs/15210235224 for examples where this has happened).

@SylvainCorlay I would like your opinion on this matter.

[anutosh491]

and a mechanism should be put in place that keeps the MacOS emsdk provided by Emscripten forge upto date with the Linux version

I would be in favour of just dropping the macos emscripten build once we switch to emscripten 4-x. I don't think it's adding any value (or ever has).

@DerThorsten's comment emscripten-forge/recipes#1689 (comment) here clearly mentions why. I leave it up to him, if he feels like maintaining a MacOs build. If yes good, if not still good.

When it comes to wasm we just need a good clean build and nothing else. Has been discussed before (emscripten-forge/recipes#1689 (comment))

It just feels like unneccessary annoying maintainance to me at this point. The only point behind this PR was to get #398 in (which was a bug pointed out by a real xeus-cpp user that Johan fixed). So we should have worked towards getting it in.

The CI should aid the developer and not stand against it. This feels like a big battle to get a small bug fix in.

1. If anyone wants to build xeus-cpp-lite on mac locally. They can build the compiler pkg too (should take less than 3 mins). We can mention it very clearly in the Readme and give steps for that too.
2. We do the same. If we want macos CI, we build the compiler pkg ourselves and not wait for emscripten-forge to host it.
3. If not, we can just drop the CI. Nothing lost.

That being said, I really don't want to keep this PR waiting. Node 16 isn't good enough, Node 22 is. Emscripten 4-x would bring it along when that migration happens (but that has time). Till then we need a temporary fix. I'll just prefer to get it from conda-forge till then (no big deal). If a user/contributor wants to run the test, they can do the same.

[anutosh491]

Hey @mcbarton , as Vassil and I discussed on discord (message link)
Node is just a testing framework here. We generally would not care unless it's the same version we get from anywhere (be it emsdk) and unless it contributes somehow to come up with xcpp.wasm

That being said, I've also dropped the osx26 runner for emscripten. We have discussed this in the past on discord. Even if we allow multiple platforms in the CI for emscripten, multiple runners doesn't really contribute here. Our native kernels still use osx15, when we drop those in the future, we can do the same for emscripten and drop osx15 for osx26. But yeah untill then just 1 runner per platform is more than sufficient.

This PR is now ready to go in on my end.

Some other small changes in the readme being I've shifted the install step after the build step which is how it should be logically.

[mcbarton]

Hey @mcbarton , as Vassil and I discussed on discord (message link) Node is just a testing framework here. We generally would not care unless it's the same version we get from anywhere (be it emsdk) and unless it contributes somehow to come up with xcpp.wasm

That being said, I've also dropped the osx26 runner for emscripten. We have discussed this in the past on discord. Even if we allow multiple platforms in the CI for emscripten, multiple runners doesn't really contribute here. Our native kernels still use osx15, when we drop those in the future, we can do the same for emscripten and drop osx15 for osx26. But yeah untill then just 1 runner per platform is more than sufficient.

This PR is now ready to go in on my end.

Some other small changes in the readme being I've shifted the install step after the build step which is how it should be logically.

Hi @anutosh491 I have read through your latest PR changes and have some questions. Maybe I am missing something, but not sure how changing emmake make check-xeus-cpp to make check-xeus-cpp makes sense. Don't we need the emmake command always for Emscripten builds, so that it sets the correct environment variables for the Emscripten build? In this case for the test target wasm, js and html are built with Emscripten.

I don't agree that its logical to move the install step to before the browser tests have been run. I don't understand why you would want to install something into your environment that you haven't fully tested. You even put in your changes Once the test pass, run the install command in the readme after changing it, forgetting the browser tests are part of the testing. This PR should be kept to its original purpose of updating node to a newer version, and not be used as a catch all for multiple things you see as bugs. This particular change will need to be reverted if you want my approval for the PR. If you insist on this change being part of this PR, then you will need to get @vgvassilev to approve directly.

In the ci section on 'Test Emscripten xeus-cpp in browser' you moved the command which created the xeus-lite-host environment. I'm sure what moving where this environment is created has to do with updating node to a newer version, but in the process you have created a bug in the ci. You moved set -e from Test Emscripten xeus-cpp in browser . Without this set -e I am pretty sure that failures in the browser tests will not cause the ci to fail. On the flip side set -e will have no effect on the Jupyter Lite integration stage of the ci. Please revert this change change with regards to set -e at the very least.

On the subject of having just one MacOS job I am not too fussed. I just don't understand why you decided on deleting the more recent MacOS runner. Surely if we are keeping just one then we would want the latest one.

[anutosh491]

Maybe I am missing something, but not sure how changing emmake make check-xeus-cpp to make check-xeus-cpp makes sense. Don't we need the emmake command always for Emscripten builds, so that it sets the correct environment variables for the Emscripten build? In this case for the test target wasm, js and html are built with Emscripten.

emmake is only required when the target invokes the compiler or linker, because it injects Emscripten’s build flags and environment variables for emcc, em++, and emar.

check-xeus-cpp does not compile anything—it simply runs

node test_xeus_cpp.js

That’s a runtime step, not a build step !
Node is just executing the already-built JavaScript/WASM output

[anutosh491]

In the ci section on 'Test Emscripten xeus-cpp in browser' you moved the command which created the xeus-lite-host environment. I'm sure what moving where this environment is created has to do with updating node to a newer version, but in the process you have created a bug in the ci. You moved set -e from Test Emscripten xeus-cpp in browser . Without this set -e I am pretty sure that failures in the browser tests will not cause the ci to fail. On the flip side set -e will have no effect on the Jupyter Lite integration stage of the ci. Please revert this change change with regards to set -e at the very least.

This is a general improvement. The xeus-lite-host env has nothing to do with the browser tests. I just moved it to where it should be !

[mcbarton]

Maybe I am missing something, but not sure how changing emmake make check-xeus-cpp to make check-xeus-cpp makes sense. Don't we need the emmake command always for Emscripten builds, so that it sets the correct environment variables for the Emscripten build? In this case for the test target wasm, js and html are built with Emscripten.

emmake is only required when the target invokes the compiler or linker, because it injects Emscripten’s build flags and environment variables for emcc, em++, and emar.

check-xeus-cpp does not compile anything—it simply runs

node test_xeus_cpp.js

That’s a runtime step, not a build step ! Node is just executing the already-built JavaScript/WASM output

That target may run that command, but it depends on test_xeus_cpp which is a build target (see

xeus-cpp/test/CMakeLists.txt

Line 101 in 190caf6

DEPENDS test_xeus_cpp

)

[mcbarton]

In the ci section on 'Test Emscripten xeus-cpp in browser' you moved the command which created the xeus-lite-host environment. I'm sure what moving where this environment is created has to do with updating node to a newer version, but in the process you have created a bug in the ci. You moved set -e from Test Emscripten xeus-cpp in browser . Without this set -e I am pretty sure that failures in the browser tests will not cause the ci to fail. On the flip side set -e will have no effect on the Jupyter Lite integration stage of the ci. Please revert this change change with regards to set -e at the very least.

This is a general improvement. The xeus-lite-host env has nothing to do with the browser tests. I just moved it to where it should be !

I think you missed my point that moving set -e along with the creation of the environment creates a bug in the ci. Without the set -e where it was, we won't know when the browser tests fail.

[anutosh491]

I think you missed my point that moving set -e along with the creation of the environment creates a bug in the ci. Without the set -e where it was, we won't know when the browser tests fail.

Ahhh okay Sorry, I'll put this back.

[anutosh491]

That target may run that command, but it depends on test_xeus_cpp which is a build target (see

We are just using node as the runtime for the js file here. It does not build anything. I can't explain this any better. You don't need emcmake or emmake untill its a cmake config or a make step !

We were just running node test_xeus_cpp.js before this right ? Did we ever need anything from emscripten then ?
It does not matter !

[mcbarton]

That target may run that command, but it depends on test_xeus_cpp which is a build target (see

We are just using node as the runtime for the js file here. It does not build anything. I can't explain this any better. You don't need emcmake or emmake untill its a cmake config or a make step !

We were just running node test_xeus_cpp.js before this right ? Did we ever need anything from emscripten then ? I does not matter !

Even though I still don't understand how its not needed, I'll accept on this point that you are right since the ci passes.

[anutosh491]

I don't agree that its logical to move the install step to before the browser tests have been run. I don't understand why you would want to install something into your environment that you haven't fully tested. You even put in your changes Once the test pass, run the install command in the readme after changing it, forgetting the browser tests are part of the testing. This PR should be kept to its original purpose of updating node to a newer version, and not be used as a catch all for multiple things you see as bugs. This particular change will need to be reverted if you want my approval for the PR. If you insist on this change being part of this PR, then you will need to get @vgvassilev to approve directly.

Cool, let @vgvassilev take the call here.

1. When I think in terms of the readme, I think in terms of both for the user and for the developer. Using node as a runtime for testing is fair for both the user and the developer (fetching node and 1 line command). But the setup and instructions for the browser tests are really unfair. This ending up in the readme never had my vote in the first place.

• I wouldn't want any user to be breaking their head on this. I would like them to try out what we offer first.
• I bet even from the list of core developers, none of us apart from you would have ever done that setup. I've not for starters. It just looks a bit over the top.
• Okay adding it in the CI is still fair. Maybe magically the browser outperforms node as the runtime. It looks very developer centric to me.

I clearly remember suggesting that we have a build.sh file we offer for this, so that the user just runs the build script and does not have to follow the really intricate setup details.

2. If I see an obvious bug, I'd like to address it then and there itself and not make numerous oneliners PRs and unncessarily populate the commit history.