More efficient MemoryAllocator

[antonfirsov]

One way to address the concerns in #1590 is to come up with a new MemoryAllocator that meets the following two requirements even under very high load:
(A) Steady allocation patterns, instead of GC fluctuations
(B) Lower amount of memory being retained, at least "after some time"

Some ideas to explore:

1. Allocate native memory over a certain threshold (~1MB) deferring the memory management to the OS
   1.1 Consider pooling unmanaged memory, especially if we implement the next point.
2. Go discontigous, and build all large buffers from fix-sized blocks of memory similarly to RecyclableMemoryStream. It's worth to check how does this work with both pooled arrays and unmanaged memory.
3. Have a threshold on the reatained memory, and release some of the pooled arrays (or pooled unmanaged buffers) when we grow over it.
4. Mentioning an idea I really hate for the sake of completeness: Have a synchronization mechanism around large buffer acquisition, similarly to System.Drawing (WIC?). To do this properly, memory allocation should become an asynchronous operation, otherwise SpinLocks will just add more fuel to the fire.

Point 1. seems to be very simple to prototype. We need an allocator that uses Marshal.AllocHGlobal over some threshold, and an ArrayPool below it, and see how the memory timeline goes with the bee heads MemoryStress benchmark in comparison to ArrayPoolMemoryAllocator.

@saucecontrol any thoughts or further ideas? (Especially on point 2.)

[saucecontrol]

Idea 2 is interesting. I wonder, though, if you can manage discontinous buffers, is it much of a stretch from there to go fully streamed?

For the record, I also hate idea 4, but that may be necessary if you really do have to materialize large images and care at all about 32-bit hosts. The machine I ran my tests on was only a quad core as well. I wonder what would happen on a 16+ core machine... I reckon even in 64-bit you'd start paging.

[antonfirsov]

I wonder, though, if you can manage discontinous buffers, is it much of a stretch from there to go fully streamed?

Implementing the ability to have discontinous buffers was relatively easy, since our processing code usually goes row-by row. We just had upgrade buffer2d.GetRowSpan(y) so it can take the row from different buffers (buffer boundary is always between rows). Going fully-streamed needs much bigger architectural changes, I would rather start with the "decode into target size" first, and see where we can get from there.

Currently we have a very high limit for images being split to different buffers, what I'm considering is to check what happens if we aggressively lower that limit. I'm unable to assess if it would make thing worse or better.

but that may be necessary if you really do have to materialize large images and care at all about 32-bit hosts.

That's why I want to avoid materializing them :) Will open an issue for tracking "decode into target size" stuff tomorrow.

The machine I ran my tests on was only a quad core as well. I wonder what would happen on a 16+ core machine...

I was wondering why do are using the basic Parallel.For overload that scales parallelism with the machine's CPU count? Is this a way of modeling typical server-side multi-threaded stress?

[saucecontrol]

I'm unable to assess if it would make thing worse or better.

I don't think it would make things worse. If I understand right, your current mode of processing is to apply each transform to the entire image before moving on the next? If so, you're kind of trashing the cache at each stage, so it wouldn't matter whether the memory is contiguous or not. I'm not sure it would make things better either, though. You might want to look at your LOH fragmentation under those high memory conditions and see whether being able to stuff a smaller buffer into an LOH hole would gain you anything. Switching to unmanaged buffers may help a bit there, but heap fragmentation can be a problem in unmanaged as well.

Of course being able to aggregate small buffers in place of a single larger one would be of benefit, provided you don't need them at the same time. I assume if that works for RecyclableMemoryStream, it should work for ImageSharp 🤷‍♂️

Is this a way of modeling typical server-side multi-threaded stress?

I must admit I didn't put that much thought into it. But yeah, the point was to make sure there were enough threads running to check whether the libraries were capable of true linear scale-up. As long as there are free processors to hide work on (as Vips does today -- and ImageSharp did before your windowed convolution implementation), you can't be sure. If you think about a high-load web server scenario, you would assume the managed thread pool would fully occupy all processors, so I want to know how a library will perform in that environment -- and to be sure that a large number of image processing operations doesn't make for a self-DoS.

BTW, I suspect the anomalous CPU measurements I got for Vips running under the profiler might have actually come from an oversubscription problem. Could be the CPU overhead of the profiler was just enough to throw the whole system into chaos and drop the speed by several multiples. I plan on doing some followup testing to see if I can nail that down.

[saucecontrol]

Couple of other interesting stress cases you might want to look at.

1. 3400+ frame GIF: http://i.imgur.com/Xv4UKYL.gif
2. Lots of good giant JPEGs available on the ESO site, like: https://www.eso.org/public/usa/images/eso1719a/

[antonfirsov]

The reason I'm thinking about "discontiguous memory by default" is not cache coherence, but the fact that it may enable more efficient pooling, reducing the GC pressure (or OS allocation traffic in case of unmanaged buffers). With large images it is very likely that a the requested buffer will outgrow the pool deferring to GC/OS allocating new giant arrays of non-uniform sizes, which would probably result in heap fragmentation even if we are using unmanaged memory.

I'm thinking of a solution that works as following:

• Under 1MB we use a standard ArrayPool (can be ArrayPool<byte>.Shared)
• Over that, we utilize a custom pool that provides uniform 2MB buffers. We then build the large discontiguous buffers aka. MemoryGroups out of these 2MB subbuffers.
  • This pool might be built of LOH arrays or unmanaged buffers. (Both approaches seem to have pros and cons.)
  • Obviously there is an upper limit for the retained/pooled memory
• If we outgrow the discontiguous pool, we may consider to defer new allocations to unmanaged memory, so at least we don't stress the GC
• There is a logic that periodically shrinks the pool so the solution works well with ImageSharp.Web (which typically places much lower load on the core library after a warmup period, so it doesn't make sense to retain memory in the pools)
  • The periods can be synchronized with actual GC calls with a trick similar to Gen2GcCallback in TlsOverPerCoreLockedStacksArrayPool. For example we may halve the retained memory before each gen2 GC.

@saucecontrol any thoughts on this particular plan? I wish I could pull in a GC expert to assess it ...

[saucecontrol]

That plan looks sound to me, but I'm no GC expert. I do expect that inability to grab a contiguous large block of memory due to LOH (or process heap) fragmentation is at play in the 32-bit OOM scenario, but it would be good to verify that before diving in to the work if possible.

It seems like if you end up doing the work to completely manage the allocations, you may as well go straight to unmanaged memory so that when you say to free something you don't have to then wait for GC to comply and then do its LOH compaction and all that. Plus, you'll probably always have to deal with some allocation requests that will be over whatever pooling threshold you set, and you'll want those to go straight to unmanaged so they are as short-lived as possible.

[antonfirsov]

you may as well go straight to unmanaged memory so that when you say to free something you don't have to then wait for GC to comply and then do its LOH compaction and all that

The problem is that I don't really know when to free up memory. With LOH + the Gen2Callback trick I can at least rely on GC heuristics with this. (Otherwise it feels like implementing my own GC)

[JimBobSquarePants]

Is this useful?

https://github.com/sakno/dotNext/tree/567ac01b54783a2cb9ef167c6fe603dcb42235f5/src/DotNext.Unsafe/Buffers

[antonfirsov]

@JimBobSquarePants it just allocates / returns unmanaged memory without pooling. Maybe it's good enough (or even preferable) approach with unmanaged memory, not sure.

My main concern about unmanaged memory is that I would be hesitant to use it as a default, because of the users who may implicitly depend on not disposing Image. An update would introduce a reliability / security issue for them instead of their existing performance issue.

[saucecontrol]

Instead of holding the raw IntPtr as a member of the MemoryManager<T>, that should hold a SafeHandle wrapping it. Something like:

internal sealed class SafeHGlobalHandle : SafeHandle
{
    private readonly int byteCount;

    public override bool IsInvalid => handle == IntPtr.Zero;

    public SafeHGlobalHandle(int size) : base(IntPtr.Zero, true)
    {
        SetHandle(Marshal.AllocHGlobal(size));
        GC.AddMemoryPressure(byteCount = size);
    }

    protected override bool ReleaseHandle()
    {
        if (IsInvalid)
            return false;

        Marshal.FreeHGlobal(handle);
        GC.RemoveMemoryPressure(byteCount);
        handle = IntPtr.Zero;

        return true;
    }
}

Since SafeHandle has a finalizer, it won't leak, but you can call Dispose to free it immediately if the MemoryManager<T> is Disposed properly.

It might be worth implementing something like that for any allocations larger than your max pool buffer size just because you know those will happen and will have a large GC cost. It's also a lot easier to trial than the segmented buffer idea.

[saucecontrol]

FYI dotnet/runtime#52098 is tracking some possible scenarios to improve in ArrayPool/GC interaction. I think Jan's scenarios 1 and 3 (and maybe 2 as well) are applicable to ImageSharp.

[antonfirsov]

There are so many things I hate about ArrayPool, especially ArrayPool<T>.Shared which is a library-level public shared global state, and as such something that shouldn't exist at all IMO.

All the magic ArrayPool.Shared is doing should be a runtime/GC implementation detail hidden behind standard new T[] allocations. (Edit: OK, it was mostly ranting, this approach would require an API like GC.UnsafeReleaseArray(...), but but AFAIK array pooling is much less common in Java for example, I assume GC does better job there)