Torchscript execution failures (hangs, access violation, Fatal error. Internal CLR fatal error. (0x80131506) )

[dje-dev]

Repeated execution of Torchscript modules leads to consistent failures of one of these varieties, apparently due to memory corruption (within about 30 seconds of runtime).

This has now been narrowed down to a fairly minimal repro. This hangs or gives CLR fatal error within about 20 seconds, repeatedly running a medium sized (26mm parameter) neural network and interleaving with exercising .NET garbage allocation.

This is running latest version of Torchsharp (0.100.3) on Windows 11 with CUDA 12.2. The TorchScript file is too large to easily upload (56MB) and possibly this would happen with any TorchScript module.

    public static void ReplicateFailure()
    {
      var module = torch.jit.load<Tensor, (Tensor, Tensor, Tensor, Tensor)>(@"c:\temp\fail.ts", DeviceType.CUDA, 0).to(ScalarType.Float16);

      for (int bs = 1; bs < 133; bs += 3)
      {
        Tensor input = torch.tensor(new float[bs * 64 * 135], ScalarType.Float16, new Device("cuda:0"), false)
                            .reshape(new long[] { bs, 64, 135 });
        for (int j = 0; j < 100; j++)
        {
          using (var dx = torch.NewDisposeScope())
          {
            module.call(input);
          }
          ExerciseGC(j);
        }
        Console.WriteLine(bs + " done loop this batch size ");
      }
    }

    public static void ExerciseGC(int index)
    {
      Console.WriteLine("in " + index);
      object[] objs = new object[20_000];
      for (int i = 0; i < 20_000; i += 999)
      {
        objs[i] = new byte[i * 10];
      }
      Console.WriteLine("out");
    }

[NiklasGustafsson]

Does it work better if you dispose the input tensor at the bottom of the loop? You're allocating a fair bit of GPU memory, as far as I can tell, and not disposing of it.

[dje-dev]

In the code, the single call to the Torchscript module is immediately enclosed with a NewDisposeScope(), I believe this will release the GPU memory every time thru the loop, correct?

[NiklasGustafsson]

In the code, the single call to the Torchscript module is immediately enclosed with a NewDisposeScope(), I believe this will release the GPU memory every time thru the loop, correct?

It will release the memory produced by the module (i.e. its output) at the bottom of the inner loop, but not the input tensor that is allocated in the outer loop.

Try:

    public static void ReplicateFailure()
    {
      var module = torch.jit.load<Tensor, (Tensor, Tensor, Tensor, Tensor)>(@"c:\temp\fail.ts", DeviceType.CUDA, 0).to(ScalarType.Float16);

      for (int bs = 1; bs < 133; bs += 3)
      {
        Tensor input = torch.tensor(new float[bs * 64 * 135], ScalarType.Float16, new Device("cuda:0"), false)
                            .reshape(new long[] { bs, 64, 135 });
        for (int j = 0; j < 100; j++)
        {
          using (var dx = torch.NewDisposeScope())
          {
            module.call(input);
          }
          ExerciseGC(j);
        }
        Console.WriteLine(bs + " done loop this batch size ");
        input.Dispose();
      }
    }

    public static void ExerciseGC(int index)
    {
      Console.WriteLine("in " + index);
      object[] objs = new object[20_000];
      for (int i = 0; i < 20_000; i += 999)
      {
        objs[i] = new byte[i * 10];
      }
      Console.WriteLine("out");
    }

My general advice is to always, always, always, treat tensors as something that you need to call Dispose() on. Don't ever rely on garbage collection. Of course, using DisposeScope is a convenient way to make that automatic, but then you need to be careful when you pass tensors out of the dynamic scope.

[dje-dev]

Yes, understood. I had missed the outer allocation. Unfortunately the code you provide still hangs within 30 seconds. Is the next thing to try to replicate with a smaller Torchscript file that is accessible to both of us, and see if that can also replicate the problem? I have no problems at all with any of the non-Torchscript features of Torchsharp on my system.

[NiklasGustafsson]

Yes, it's hard to diagnose without more details. Unfortunately, I'm going to be out of the office until August 7th, but maybe we can continue this then?

[dje-dev]

Understood, thanks, I'll try to have data points to share upon your return.

[dje-dev]

Here's a more minimal repro of this problem that makes Torchscript not currently usable for me.

Repeated execution of unit test TestLoadJIT_3 will surface this problem, resulting in CLR errors, access violations and hangs.

For example, the following always fails within a minute or two of execution (on CPU, using explicit release of Tensor memory).

    int xx = 0;
    jit.ScriptModule<Tensor, Tensor> mx = torch.jit.load<Tensor, Tensor>("l1000_100_10.script.dat");
    while (true)
    {
      using var input = torch.ones(1000);
      using var output = mx.call(input);
      Console.WriteLine(xx++);
    }

Here's the tail output of one sample abend (after about 60 seconds):

242146
Fatal error. 242147
242148
Internal CLR error. (0x80131506)
242149
242150

[NiklasGustafsson]

And it works when you run it using PyTorch, presumably?

[dje-dev]

I don't claim to be able to exactly replicate that, since Torchsharp comes with it's own bundled binaries which differ from my Python package.

But yes, the following seems to run indefinitely with no problems on this same system.

import torch

xx = 0
mx = torch.jit.load("l1000_100_10.script.dat")

while True:
    input = torch.ones(1000)
    output = mx(input)
    print(xx)
    xx += 1

[NiklasGustafsson]

Okay, I'll take a look.

I'm not super-optimistic, since TorchSharp doesn't really do much with TorchScript modules -- just invoke the native code. It's a very thin interop layer.

[NiklasGustafsson]

Interesting...

The problem is that when returning the result, the C++ code calls into managed code to allocate a 'TensorOrScalar' struct, which then has to be pinned in order to avoid being moved.

However, in IndexedPinnedArray.cs, line 44, the call to AddrOfPinnedObject() sometimes returns IntPtr.Zero for a valid, allocated handle to a pinned object. This will take some sleuthing to figure out.

[phineasng]

Hi all,

@NiklasGustafsson thanks for this great package.

I was wondering if there is any update on this? Or if in the meantime somebody found out at least an hacky workaround.

[NiklasGustafsson]

I come back to this one every now and then when I'm bored... :-) Still not any closer to figuring out what's going on.

[dje-dev]

Thank you for the update! I was also going to ask, this is definitely an impediment for my work too.

I'm sorry it's proving so much a mystery. Here's one radical thought - maybe you could try upgrading to more recent Pytorch (2.1) and CUDA (12.x) versions. This could have two benefits, it might fix the problem if it was an old bug or at least behave differently and give more clues. Additionally, a package upgrade would be very helpful (especially since the current one does not support modern hardware such as H100, reporting "no kernel found for device" when Torchsharp is run.