JsonSerializer polymorphic serialization and deserialization support

[Symbai]

EDIT see #30083 (comment) for a finalized proposal.

Original proposal by @Symbai (click to view)

public class FooBase
public class FooA : FooBase
public class FooB : FooBase
List<FooBase> SerializationObject = new List<FooBase> { new FooA(), new FooB() }

Serialized with Newtonsoft.JSON

JsonSerializerSettings Settings = new JsonSerializerSettings
{
    SerializationBinder = new KnownTypesBinder
    {
        KnownTypes = new List<Type> { typeof(FooA), typeof(FooB) }
    },
    TypeNameHandling = TypeNameHandling.Objects
};
List<FooBase> FooBases = new List<FooBase>() { new FooA(), new FooB() };
var json = JsonConvert.SerializeObject(FooBases, Settings);

will be:
[{"$type":"FooA","NameA":"FooA","Name":"FooBase"},{"$type":"FooB","NameB":"FooB","Name":"FooBase"}]

When using System.Text.Json.JsonSerializer to deserialize the json, FooA and FooB are both type of FooBase. Is there a way that JsonSerializer supports inheirited classes? How can I make sure the type will be the same and not the base class it inherits from?

[Symbai]

Figured it's somehow possible using JsonConverter, which is for some reason completely undocumented https://docs.microsoft.com/en-us/dotnet/api/system.text.json?view=netcore-3.0 but I have no idea how to deserialize without copying each property by hand like it's shown in this example https://github.com/dotnet/corefx/issues/36639. Copying properties by hand seems extremely odd as classes may have lots of them and maintainability isn't given in such a case anymore. Is there a way I can simply deserialize into a specific type in the converter?

Or am I missing something and there is a simple solution like in Newtonsoft.Json ?

public class FooAConverter : System.Text.Json.Serialization.JsonConverter<FooBase>
{
    public override bool CanConvert(Type type)
    {
        return typeof(FooBase).IsAssignableFrom(type);
    }
    public override FooBase Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        reader.Read();
        var typeProperty = reader.GetString();
        if (typeProperty != "$type")
            throw new NotSupportedException();
        reader.Read();
        var typeValue = reader.GetString();
        FooBase value;
        switch (typeValue)
        {
            case "FooA":
                value = new FooA(); //<-- return deserialized FooA?
                break;
            case "FooB":
                value = new FooB(); //<-- return deserialized FooB?
                break;
            default:
                throw new System.Text.Json.JsonException();
        }
        while (reader.Read())
        {
            //How to deserialize without copying every property at this point?
            if (reader.TokenType == JsonTokenType.EndObject)
            {
                return value;
            }
        }
        throw new NotSupportedException();
    }

    public override void Write(Utf8JsonWriter writer, FooBase value, JsonSerializerOptions options)    {    }
}

[ahsonkhan]

cc @steveharter

[Timovzl]

It's pretty awkward that JsonConverter doesn't give you a way to recurse back into the "regular" flow (i.e. "not handled by this particular converter") for deserializing subcomponents.

This seems to make the whole design only useful for very simple cases, with few properties and no nested custom types.

[steveharter]

Figured it's somehow possible using JsonConverter, which is for some reason completely undocumented https://docs.microsoft.com/en-us/dotnet/api/system.text.json?view=netcore-3.0

It's in the sub-namespace Serialization: https://docs.microsoft.com/en-us/dotnet/api/system.text.json.serialization?view=netcore-3.0

[steveharter]

I have no idea how to deserialize without copying each property by hand like it's shown in this example

and

t's pretty awkward that JsonConverter doesn't give you a way to recurse back into the "regular" flow (i.e. "not handled by this particular converter") for deserializing subcomponents

Yes that is known - the existing converters are bare-bones converters that let you do anything you want, but also require you do essentially do everything for a given type (but you can recurse to handle other types). They were primarily intended for data-type converters, not for objects and collections.

We are working on making this better in 5.0 timeframe by providing object- and collection- based specific converters that would make this much easier, as well as improve performance in certain cases.

[Timovzl]

but you can recurse to handle other types

@steveharter That seems like it could help. Can you give an example of how to do this?

[ANahr]

We are using custom converters like the following for inherited class support. I'm pretty sure deserialization is abysmal from a perf point of view, but it works and doesn't need much code and serialization works as it should. So as a Workaround you can use somthing like this:

public class ParameterTransferObjectConverter : JsonConverter<ParameterTransferObject>
{
    public override ParameterTransferObject Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        using (var doc = JsonDocument.ParseValue(ref reader))
        {
            ParamType typeDiscriminator = (ParamType)doc.RootElement.GetProperty(@"parameterType").GetInt32();
            var type = ParameterManager.GetTransferObjectType(typeDiscriminator);
            // Enhance: doc.RootElement.GetRawText() has likely bad perf characteristics
            return (ParameterTransferObject)JsonSerializer.Deserialize(doc.RootElement.GetRawText(), type, options);
        }
    }

    public override void Write(Utf8JsonWriter writer, ParameterTransferObject value, JsonSerializerOptions options)
    {
        var type = ParameterManager.GetTransferObjectType(value.ParameterType);
        JsonSerializer.Serialize(writer, value, type, options);
    }
}

[lfr]

Thanks @ANahr, indeed performance must be abysmal in comparison, but this allowed me to at least get deserialization to work till System.Text.Json matures, in my opinion it's a mistake for Microsoft to encourage people to use it, it should still be flagged as preview

[layomia]

Marking this as the issue to track polymorphic deserialization for 5.0.
Serialization is tracked in https://github.com/dotnet/corefx/issues/38650.

[layomia]

From @Milkitic in dotnet/corefx#41305

For some reasons that Json for configuration use, I did a test on how the System.Text.Json handles the interface deserialization. I read the API and can't find anything about this function. First I tried IEnumerable<T>. Here is the code:

using System.Collections.Generic;
using System.Text.Json;

namespace dotnet30test
{
    class Program2
    {
        static void Main(string[] args)
        {
            var obj = new TestSerializationObj2
            {
                Names = new HashSet<string> { "1", "2" }
            };
            var sysContent = JsonSerializer.Serialize(obj, new JsonSerializerOptions { WriteIndented = true, });
            var sysNewObj = JsonSerializer.Deserialize<TestSerializationObj>(sysContent);
        }
    }

    public class TestSerializationObj2
    {
        public IEnumerable<string> Names { get; set; }
    }
}

The Names property is forced to convert from Hashset<string> to List<string>, but it works successfully indeedly.
And then I try the custom interface, it throws JsonException. What I think that needed is something like Json.NET's auto type name handling. Here is the code:

using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using NewtonJson = Newtonsoft.Json;
using SysJson = System.Text.Json;

namespace dotnet30test
{
    class Program
    {
        static void Main(string[] args)
        {
            var obj = new TestSerializationObj
            {
                CustomObj = new DefaultCustom()
            };

            var newtonContent = NewtonJson.JsonConvert.SerializeObject(obj,
                new NewtonJson.JsonSerializerSettings
                {
                    TypeNameHandling = NewtonJson.TypeNameHandling.Auto,
                    Formatting = NewtonJson.Formatting.Indented
                });
            var newtonNewObj = NewtonJson.JsonConvert.DeserializeObject<TestSerializationObj>(newtonContent,
                new NewtonJson.JsonSerializerSettings { TypeNameHandling = NewtonJson.TypeNameHandling.Auto }); // works

            var sysContent = SysJson.JsonSerializer.Serialize(obj, new SysJson.JsonSerializerOptions { WriteIndented = true, });
            var sysNewObj = SysJson.JsonSerializer.Deserialize<TestSerializationObj>(sysContent); // throws exception
        }
    }


    public sealed class TestSerializationObj
    {
        public Dictionary<string, int> Dictionary { get; set; } = new Dictionary<string, int>() { ["a"] = 0, ["b"] = 1 };
        public ICustom CustomObj { get; set; }
    }

    public class DefaultCustom : ICustom
    {
        public string Name { get; set; } = "Default Implementation";
        public int Id { get; set; } = 300;
    }

    public interface ICustom
    {
        int Id { get; set; }
    }
}

Thanks!

[mcatanzariti]

Please try this library I wrote as an extension to System.Text.Json to offer polymorphism: https://github.com/dahomey-technologies/Dahomey.Json