[feature request] very basic support for MPI debugging multiple processes at the same time

[davydden]

It would be good to have a very basic support of debugging MPI programs.

Outside of VSCode I can attach a debugger to each terminal via

mpirun -np 2 xterm -e lldb -f my_exec

and then do the standard serial debugging in each terminal window.

Alternative is to hack-in into the source a piece of code which prints process ID and waits so that a user can attach a debugger to some/all processes, see https://www.open-mpi.org/faq/?category=debugging#serial-debuggers .

With vscode-cpptools one can already adopt the latter via launch.json and its "processId": "${command:pickProcess}" option.

It would be great, if vscode-cpptools goes one step further by extending the launch config with

    {
      "num_mpi_processes" : "4",
      "mpi_exec": "<path-to-mpirun>"
    },

and in the background simply attach gdb/lldb debugger for each MPI process and perhaps append a number to the "name": "(lldb) Attach", so that one can use the available GUI.

One would, of course, need to manually switch between debuggers for each process in the GUI but this should be fine for up to 4-8 processes, which is often enough to find bugs in MPI programs.

p.s. AFAIK the only other non-commerical project with MPI debugger is Eclipse PTP, but last time I checked it was limited to Linux.

[pieandcakes]

One would, of course, need to manually switch between debuggers for each process in the GUI but this should be fine for up to 4-8 processes, which is often enough to find bugs in MPI programs.

@davydden I don't think the UI would support what you are proposing. Unfortunately, VS Code will only allow one debug instance, so even if we attach to 4 separate processes (which we can't because VS Code won't spawn the debug adapter 4 times) there isn't a UI to allow this change. With the way the debugAdapter is authored, it is a one to one relationship with the debuggee. Your feature request would need to start with VS Code for them to allow simultaneous multi-process debugging.

[davydden]

@pieandcakes
that's unfortunate, but thanks for the prompt and detailed comment, though 👍

EDIT: i guess the only way to hack this is separate attach configs:

    {
      "name": "(lldb) Attach MPI 0",
      "type": "cppdbg",
      "request": "attach",
      "program": "${workspaceFolder}/some.debug",
      "processId": "${command:pickProcess}",
      "MIMode": "lldb"
    },
    {
      "name": "(lldb) Attach MPI 1",
      "type": "cppdbg",
      "request": "attach",
      "program": "${workspaceFolder}/some.debug",
      "processId": "${command:pickProcess}",
      "MIMode": "lldb"
    },
    {
      "name": "(lldb) Attach MPI 2",
      "type": "cppdbg",
      "request": "attach",
      "program": "${workspaceFolder}/some.debug",
      "processId": "${command:pickProcess}",
      "MIMode": "lldb"
    },

[pieandcakes]

@davydden Even then, you can only attach one process at a time so it won't get you too far. Once you have started debugging, you can't really debug again.

[davydden]

Once you have started debugging, you can't really debug again.

I see, thanks for clarifying.

[davydden]

@pieandcakes I was said upstream that the support for multiple processes is already there:

The VS Code debugger supports multiple processes. See https://code.visualstudio.com/updates/v1_20#_node-debugging.
The animated gif shows VS Code debugging a master process and 5 child processes.

do you think it's possible for cpptools to use this functionality and enable debugging with multiple processes?

[davydden]

based on the discussion in vscode issue, I would say one needs to do two things:

1. from cpptools debugger configuration, run a given executable with mpirun -np X my_executable and somehow pause until debuggers are attached (X is number of processes).
2. look for processes with the name my_executable (there should be as many as X) and attach lldb debugger to each one

Personally, I have no idea how one would pause[1] processes to wait until debugger is attached, but apart from this technical issue, this should be doable and everything is there in VSCode to support it.

[1] without injecting any code like

while (0 == i) 
  sleep(5);

[pieandcakes]

@davydden we can take that as a suggested feature request. Can you provide a project with a repro? I don't have experience with mpirun.

[davydden]

@pieandcakes sure, here's a simple example

// compile with:
//   mpic++ -std=c++11 mpi_example.cc -o mpi_example
// run with:
//   mpirun -np 2 mpi_example
// debug manually with different terminals for different processes:
//   mpirun -np 2 xterm -e lldb mpi_example

#include <mpi.h>
#include <stdio.h>
#include <iostream>
#include <unistd.h>

int main(int argc, char** argv) {
    // Initialize the MPI environment
    MPI_Init(NULL, NULL);

    // Get the number of processes
    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    // Get the rank of the process
    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

    for (int p = 0; p < world_size; ++p)
      {
        if (p == world_rank)
          std::cout << "process rank " << p << " out of " << world_size << std::endl;

        // uncomment this if you want all processes to wait and check PID and alike
        /*
        while (p==0)
          sleep(5);
        */

        MPI_Barrier(MPI_COMM_WORLD);
      }

    // Finalize the MPI environment.
    MPI_Finalize();
}

on macOS you can use homebrew or spack to build any MPI providers i.e. OpenMPI (do not confuse with OpenMP, those are different things). On Linux you can get MPI from standard repositories (i.e. sudo apt-get install openmpi-bin openmpi-common libopenmpi-dev should do). I have no experience with building MPI on Windows, though.

Let me know if I could further help somehow, though I have no experience whatsoever with TypeScript / JavaScript used in VSCode.

p.s. this is of course a personal opinion, but I think VS Code could attract a fair amount of computational scientists with this feature as not all universities have licenses for dedicated commercial debuggers, that cost upwards from around 700$. It's probably also not something an individual would buy himself. A good summary of debugger options for MPI is in this stackoverflow post and also in deal.II FAQ wiki.

[WardenGnaw]

Thanks @davydden. We played around with this a bit and I think I could see how we someone could make this work.

Below is a basic outline in case anyone wants to try and take this on. In the interest of full disclosure, this is enough work that we would need to see significant interest in using this from the MPI community before we would take on this work.

At any rate, here is how we think it could work:

1. A VS Code extension that will do something
   1. It opens a unix domain socket (or whatever IPC you want) to listen to messages from a ‘launch helper’ program (see item 2 below).
   2. It kicks off an MPI session by executing something like mpirun -n <num_nodes> /example/path/to/launch/helper <path-to-app>
   3. When the launch helper program reports process ids back, build up a json object with the content that would normally be in a launch.json configuration that tells VS Code to debug using GDB/LLDB, and attach to the specified process ID
   4. Send this launch command to VS Code using vscode.debug.startDebugging(undefined, DebugConfiguration) // see https://code.visualstudio.com/docs/extensionAPI/vscode-api#_debug
   5. In startDebugging.then, tell the launch helper program to run the target app
2. We need a little ‘launch helper’ program that is written in C/C++. I will call this 'MPIDebugLauncher'. It does the following:
   1. When it starts up, it sends a message to the extension with its process id
   2. Wait for the extension to tell the helper that launch is ready (step 1.e)
   3. execv the target app

A few other notes:

1. It might be possible to have GDB launch the process instead of having it to attach if GDB is unhappy with attaching past the execv. But I am not sure if it is just environment variables that need to be propagated from MPI to the target app, or if things will only work if the parent/child process relationship is preserved.
2. When we implemented MPI debugging in full VS many years ago, there were many scalability and races that we needed to deal with. So while this seems like it should work, its hard to say for sure.
3. As an alternative, if the target app processes are all spawned as child processes of mpirun/exec, it might be possible to implement this as just general child process debugging

[davydden]

@WardenGnaw thanks entertaining the idea, appreciate it.

this is enough work that we would need to see significant interest in using this from the MPI community before we would take on this work.

That's perfectly understandable. I will ping a few folks in case the want to add something on the prospect of debugging MPI with GUI in Visual Studio Code @alalazo @bangerth @tjhei @BarrySmith @tgamblin @goxberry @jthies .