Add Span<T> Binary Reader/Writer APIs

[KrzysztofCwalina]

The API allows for reading and writing binary representation of primitve types (bit blitting) from/to spans of bytes. The API is used by SignalR.

A prototype of the API is available in corfxlab: https://github.com/dotnet/corefxlab/tree/master/src/System.Binary

Part of dotnet/corefx#24174

Usage is quite simple:

var span = stackalloc byte[4]; 
span.Write(Int32.MaxValue);
var value = span.Read<int>();

The LittleEndian and BigEndian versions assume/specify specific endianness. The Write/Read versions assume current machine endianness. Try versions return false if the buffer is too small to read/write the specified data type.

API Design:

// System.Memory.dll
namespace System.Buffers.Binary {
    public static class BinaryPrimitives {
        public static short ReadInt16BigEndian(ReadOnlySpan<byte> buffer);
        public static short ReadInt16LittleEndian(ReadOnlySpan<byte> buffer);
        public static int ReadInt32BigEndian(ReadOnlySpan<byte> buffer);
        public static int ReadInt32LittleEndian(ReadOnlySpan<byte> buffer);
        public static long ReadInt64BigEndian(ReadOnlySpan<byte> buffer);
        public static long ReadInt64LittleEndian(ReadOnlySpan<byte> buffer);
        public static T ReadMachineEndian<T>(ReadOnlySpan<byte> buffer) where T : struct;
        public static ushort ReadUInt16BigEndian(ReadOnlySpan<byte> buffer);
        public static ushort ReadUInt16LittleEndian(ReadOnlySpan<byte> buffer);
        public static uint ReadUInt32BigEndian(ReadOnlySpan<byte> buffer);
        public static uint ReadUInt32LittleEndian(ReadOnlySpan<byte> buffer);
        public static ulong ReadUInt64BigEndian(ReadOnlySpan<byte> buffer);
        public static ulong ReadUInt64LittleEndian(ReadOnlySpan<byte> buffer);
        public static byte ReverseEndianness(byte value);
        public static short ReverseEndianness(short value);
        public static int ReverseEndianness(int value);
        public static long ReverseEndianness(long value);
        public static sbyte ReverseEndianness(sbyte value);
        public static ushort ReverseEndianness(ushort value);
        public static uint ReverseEndianness(uint value);
        public static ulong ReverseEndianness(ulong value);
        public static bool TryReadInt16BigEndian(ReadOnlySpan<byte> buffer, out short value);
        public static bool TryReadInt16LittleEndian(ReadOnlySpan<byte> buffer, out short value);
        public static bool TryReadInt32BigEndian(ReadOnlySpan<byte> buffer, out int value);
        public static bool TryReadInt32LittleEndian(ReadOnlySpan<byte> buffer, out int value);
        public static bool TryReadInt64BigEndian(ReadOnlySpan<byte> buffer, out long value);
        public static bool TryReadInt64LittleEndian(ReadOnlySpan<byte> buffer, out long value);
        public static bool TryReadMachineEndian<T>(ReadOnlySpan<byte> buffer, out T value) where T : struct;
        public static bool TryReadUInt16BigEndian(ReadOnlySpan<byte> buffer, out ushort value);
        public static bool TryReadUInt16LittleEndian(ReadOnlySpan<byte> buffer, out ushort value);
        public static bool TryReadUInt32BigEndian(ReadOnlySpan<byte> buffer, out uint value);
        public static bool TryReadUInt32LittleEndian(ReadOnlySpan<byte> buffer, out uint value);
        public static bool TryReadUInt64BigEndian(ReadOnlySpan<byte> buffer, out ulong value);
        public static bool TryReadUInt64LittleEndian(ReadOnlySpan<byte> buffer, out ulong value);
        public static bool TryWriteInt16BigEndian(Span<byte> buffer, short value);
        public static bool TryWriteInt16LittleEndian(Span<byte> buffer, short value);
        public static bool TryWriteInt32BigEndian(Span<byte> buffer, int value);
        public static bool TryWriteInt32LittleEndian(Span<byte> buffer, int value);
        public static bool TryWriteInt64BigEndian(Span<byte> buffer, long value);
        public static bool TryWriteInt64LittleEndian(Span<byte> buffer, long value);
        public static bool TryWriteMachineEndian<T>(Span<byte> buffer, ref T value) where T : struct;
        public static bool TryWriteUInt16BigEndian(Span<byte> buffer, ushort value);
        public static bool TryWriteUInt16LittleEndian(Span<byte> buffer, ushort value);
        public static bool TryWriteUInt32BigEndian(Span<byte> buffer, uint value);
        public static bool TryWriteUInt32LittleEndian(Span<byte> buffer, uint value);
        public static bool TryWriteUInt64BigEndian(Span<byte> buffer, ulong value);
        public static bool TryWriteUInt64LittleEndian(Span<byte> buffer, ulong value);
        public static void WriteInt16BigEndian(Span<byte> buffer, short value);
        public static void WriteInt16LittleEndian(Span<byte> buffer, short value);
        public static void WriteInt32BigEndian(Span<byte> buffer, int value);
        public static void WriteInt32LittleEndian(Span<byte> buffer, int value);
        public static void WriteInt64BigEndian(Span<byte> buffer, long value);
        public static void WriteInt64LittleEndian(Span<byte> buffer, long value);
        public static void WriteMachineEndian<T>(Span<byte> buffer, ref T value) where T : struct;
        public static void WriteUInt16BigEndian(Span<byte> buffer, ushort value);
        public static void WriteUInt16LittleEndian(Span<byte> buffer, ushort value);
        public static void WriteUInt32BigEndian(Span<byte> buffer, uint value);
        public static void WriteUInt32LittleEndian(Span<byte> buffer, uint value);
        public static void WriteUInt64BigEndian(Span<byte> buffer, ulong value);
        public static void WriteUInt64LittleEndian(Span<byte> buffer, ulong value);
    }
}

Original proposals (click to expand)

// System.Memory.dll
namespace System.Buffers.Binary {
    public static class BinaryPrimitives {
        public static short ReadInt16BigEndian(ReadOnlySpan<byte> buffer);
        public static short ReadInt16LittleEndian(ReadOnlySpan<byte> buffer);
        public static int ReadInt32BigEndian(ReadOnlySpan<byte> buffer);
        public static int ReadInt32LittleEndian(ReadOnlySpan<byte> buffer);
        public static long ReadInt64BigEndian(ReadOnlySpan<byte> buffer);
        public static long ReadInt64LittleEndian(ReadOnlySpan<byte> buffer);
        public static T ReadMachineEndian<T>(ReadOnlySpan<byte> buffer) where T : struct;
        public static ushort ReadUInt16BigEndian(ReadOnlySpan<byte> buffer);
        public static ushort ReadUInt16LittleEndian(ReadOnlySpan<byte> buffer);
        public static uint ReadUInt32BigEndian(ReadOnlySpan<byte> buffer);
        public static uint ReadUInt32LittleEndian(ReadOnlySpan<byte> buffer);
        public static ulong ReadUInt64BigEndian(ReadOnlySpan<byte> buffer);
        public static ulong ReadUInt64LittleEndian(ReadOnlySpan<byte> buffer);
        public static byte ReverseEndianness(byte value);
        public static short ReverseEndianness(short value);
        public static int ReverseEndianness(int value);
        public static long ReverseEndianness(long value);
        public static sbyte ReverseEndianness(sbyte value);
        public static ushort ReverseEndianness(ushort value);
        public static uint ReverseEndianness(uint value);
        public static ulong ReverseEndianness(ulong value);
        public static bool TryReadInt16BigEndian(ReadOnlySpan<byte> buffer, out short value);
        public static bool TryReadInt16LittleEndian(ReadOnlySpan<byte> buffer, out short value);
        public static bool TryReadInt32BigEndian(ReadOnlySpan<byte> buffer, out int value);
        public static bool TryReadInt32LittleEndian(ReadOnlySpan<byte> buffer, out int value);
        public static bool TryReadInt64BigEndian(ReadOnlySpan<byte> buffer, out long value);
        public static bool TryReadInt64LittleEndian(ReadOnlySpan<byte> buffer, out long value);
        public static bool TryReadMachineEndian<T>(ReadOnlySpan<byte> buffer, out T value) where T : struct;
        public static bool TryReadUInt16BigEndian(ReadOnlySpan<byte> buffer, out ushort value);
        public static bool TryReadUInt16LittleEndian(ReadOnlySpan<byte> buffer, out ushort value);
        public static bool TryReadUInt32BigEndian(ReadOnlySpan<byte> buffer, out uint value);
        public static bool TryReadUInt32LittleEndian(ReadOnlySpan<byte> buffer, out uint value);
        public static bool TryReadUInt64BigEndian(ReadOnlySpan<byte> buffer, out ulong value);
        public static bool TryReadUInt64LittleEndian(ReadOnlySpan<byte> buffer, out ulong value);
        public static bool TryWriteInt16BigEndian(Span<byte> buffer, short value);
        public static bool TryWriteInt16LittleEndian(Span<byte> buffer, short value);
        public static bool TryWriteInt32BigEndian(Span<byte> buffer, int value);
        public static bool TryWriteInt32LittleEndian(Span<byte> buffer, int value);
        public static bool TryWriteInt64BigEndian(Span<byte> buffer, long value);
        public static bool TryWriteInt64LittleEndian(Span<byte> buffer, long value);
        public static bool TryWriteMachineEndian<T>(Span<byte> buffer, ref T value) where T : struct;
        public static bool TryWriteUInt16BigEndian(Span<byte> buffer, ushort value);
        public static bool TryWriteUInt16LittleEndian(Span<byte> buffer, ushort value);
        public static bool TryWriteUInt32BigEndian(Span<byte> buffer, uint value);
        public static bool TryWriteUInt32LittleEndian(Span<byte> buffer, uint value);
        public static bool TryWriteUInt64BigEndian(Span<byte> buffer, ulong value);
        public static bool TryWriteUInt64LittleEndian(Span<byte> buffer, ulong value);
        public static void WriteInt16BigEndian(Span<byte> buffer, short value);
        public static void WriteInt16LittleEndian(Span<byte> buffer, short value);
        public static void WriteInt32BigEndian(Span<byte> buffer, int value);
        public static void WriteInt32LittleEndian(Span<byte> buffer, int value);
        public static void WriteInt64BigEndian(Span<byte> buffer, long value);
        public static void WriteInt64LittleEndian(Span<byte> buffer, long value);
        public static void WriteMachineEndian<T>(Span<byte> buffer, ref T value) where T : struct;
        public static void WriteUInt16BigEndian(Span<byte> buffer, ushort value);
        public static void WriteUInt16LittleEndian(Span<byte> buffer, ushort value);
        public static void WriteUInt32BigEndian(Span<byte> buffer, uint value);
        public static void WriteUInt32LittleEndian(Span<byte> buffer, uint value);
        public static void WriteUInt64BigEndian(Span<byte> buffer, ulong value);
        public static void WriteUInt64LittleEndian(Span<byte> buffer, ulong value);
    }
}

[weshaggard]

Api review feedback:

public static bool TryReadBigEndiann(this ReadOnlySpan buffer, out T value) where T : struct;

TryReadBigEndiann -> TryReadBigEndian

• The T should be constrained to blittable types, currently the compiler doesn't support that so it will throw in cases it cannot support.
• BigEndian/LittleEndian the T should be constrained to our single value primitive types, perhaps just use an overload for each of those instead of a T.
• Consider making the Read/Write Apis use the default endianness and just have a flip endianness method for just the single value primitives.
• How do these relate to BitConverter APIs?
• We need a better name then Binary for this type. Perhaps SpanBinaryExtensions.
• Will anyone ever call LittleEndian APIs over the Read/Write APIs? As they would essentially be the same until we run on a BigEndian architecture.

[stephentoub]

Additional questions:

• We've been calling things TryWriteBytes, but here it's just TryWrite. Is there a reason for the difference? Should we rename TryWriteBytes -> TryWrite or TryWrite -> TryWriteBytes?
• What is the behavior of the non-Try Read/Write... do they throw in cases where the Try would return false?

[svick]

I'm concerned that a common mistake will be to try to use this API to write multiple values to a single Span<byte> like this:

var span = stackalloc byte[8]; 
span.Write(Int32.MinValue);
span.Write(Int32.MaxValue);

Similar code works for other Write methods in the framework (including BinaryWriter, Stream and TextWriter). Is there a way to change the API so that this mistake is avoided? (Maybe by using a different name?)

Also, how should that code be written correctly? Do I have to manually use Slice()?

var span = stackalloc byte[8]; 
span.Slice(0).Write(Int32.MinValue);
span.Slice(4).Write(Int32.MaxValue);

(The first Slice is unnecessary, but I added it for symmetry.)

[jkotas]

System.Buffers.dll

Why System.Buffers and not System.Memory where Span and friends live?

just use an overload for each of those instead of a T.

+1

[ahsonkhan]

Why System.Buffers and not System.Memory where Span and friends live?

This was brought up in the review. The rationale is that we only have the span/memory primitive type and APIs/extension methods closely related to them in System.Memory (IndexOf, AsBytes, etc). Operations like Read/Write, transformation APIs like text encode, and text parse/format would live in System.Buffers.

Also, how should that code be written correctly? Do I have to manually use Slice()?

Yes.

What is the behavior of the non-Try Read/Write... do they throw in cases where the Try would return false?

Yes. Are there any concerns with that?

We've been calling things TryWriteBytes, but here it's just TryWrite. Is there a reason for the difference? Should we rename TryWriteBytes -> TryWrite or TryWrite -> TryWriteBytes?

One approach could be to remain consistent with TryRead and since we don't have TryReadBytes, change TryWriteBytes elsewhere to TryWrite and keep this as TryWrite as well.

[jkotas]

The rationale is that we only have the span/memory primitive type and APIs/extension methods closely related to them in System.Memory

Well, it is more complex than for .NET Core: Span/Memory primitive types are in System.Runtime, SpanExtension are in System.Memory, and now you are proposing to add extension methods to read/write bytes from Span to System.Buffers that has nothing to do with Span so far. It feels like we are trying to overthink the layering again. It has been proven that we do not know how to do that well. We should preffer larger assemblies that have all related stuff lumped together.

[davidfowl]

I think we should remove the generics and instead have the type being written in the method name e.g.

span.ReadInt32()
span.TryReadInt32()

It's too easy to mess up the generic method. Here's some MQTT parser code using these APIs:

https://github.com/aspnet/SignalR/blob/6151af7e2610d422bba2097c8326de7e5d545803/samples/SocketsSample/EndPoints/MqttEndPoint.cs#L48-L59

[ahsonkhan]

add extension methods to read/write bytes from Span to System.Buffers that has nothing to do with Span so far

We need a better name then Binary for this type. Perhaps SpanBinaryExtensions.

We could change them so they are no longer extension methods on Span (and take Span as a parameter, similar to the other System.Text.Primitive APIs for example bool TryParseBoolean(ReadOnlySpan<byte> text, out bool value)). However, in that case, a short name like Binary would be preferable over something like SpanBinaryExtensions.
Used like:

var span = stackalloc byte[4]; 
Binary.Write(span, Int32.MaxValue);
var value = Binary.Read<int>(span);

[davidfowl]

Also this problem that @svick brings up is another big usability issue with these APIs and span:

Also, how should that code be written correctly? Do I have to manually use Slice()?

Since the early days of pipelines and span @benaadams, @Drawaes, @mgravell and I have been discussing this scenario. We should also think about a mutable ref like struct that moves forward as the data is written.

var writer = new BinaryWriter(span);
writer.WriteInt32(10); // writes and moves to span + sizeof(int32)
writer.WriteInt64(20); 

[Drawaes]

Funny over the last couple of days I have been writing a fast PE/IL rewriter and while the code is dogs breakfast due to me deep diving the topic I have had to write exactly those methods again.

https://github.com/Drawaes/dotnetXperiments/blob/master/PEQuick/PEQuick/MetaData/MetaDataReader.cs#L60

A struct wrapper for a span. And then extension methods that read from (to be replaced with your methods of course mine are a hack) the span. Then slice the span and return it. Then the struct wrapper uses this to keep its internal span up to date.

Seeing as every app I write that uses spans heavily ends up with this same pattern it seems a good candidate for a framework item.

[stephentoub]

We'd discussed a number of similar concerns yesterday as well:

• Generically reading T is problematic, especially when talking about reading big/little endian, as that's likely not what's going to do the intended operation for anything but int/long/etc., e.g. Guid would be incorrect.
• Especially as extension methods, Write/Read look like they advance but don't.
• We have these basic read/write operations for span on BitConverter, and these operations essentially duplicate them; I prefer it when these separate packages provide the more advanced functionality rather than duplicating what's in the core.

I'd be happy with just putting an API like this in System.Memory.dll or wherever:

namespace System.Buffers
{
    public ref struct SpanReader
    {
        public SpanReader(ReadOnlySpan<byte> span);

        public int Length { get; }
        public int Position { get; set; }
        public ReadOnlySpan<byte> Remaining { get; }

        public bool TryReadInt32LittleEndian(out int result);
        public bool TryReadInt32BigEndian(out int result);

        public bool TryReadInt64LittleEndian(out long result);
        public bool TryReadInt64BigEndian(out long result);

        ... // repeated for each primitive type we care about
    }

    public ref struct SpanWriter
    {
        public SpanWriter(Span<byte> span);

        public int Length { get; }
        public int Position { get; set; }
        public Span<byte> Remaining { get; }

        public bool TryWriteInt32LittleEndian(int result);
        public bool TryWriteInt32BigEndian(int result);

        public bool TryWriteInt64LittleEndian(long result);
        public bool TryWriteInt64BigEndian(long result);

        ... // repeated for each primitive type we care about
    }
}

We wouldn't have just the unadorned TryReadInt32 as it's duplicative: if you don't care about endianness because you're just reading/writing on the same system, then you can use either the little or big methods, as long as you're consistent, and if you do care about endianness, then you can use the one you need. I didn't include the non-Try methods above, because if you're writing this kind of code, it seems you do care about the potential expense of exceptions, in which case you either want to use the Try methods to avoid exceptions, or you know for certain that the operation will succeed, in which case you can just use the Try variant; that said, if folks feel strongly about having the non-Try, I don't think it's a big deal, it just seems overkill to me to have both.

This avoids concerns about arbitrary Ts and endianness (plus needing primitive checks), it avoids concerns about duplication with BitConverter as it's clearly providing more advanced functionality than just the primitive read/write BitConverter does, it avoids concerns about not advancing because it advances, it fits well with the general .NET notion of XxReader/Writer, etc.