proposal: encoding/json: add omitnil option

[blixt]

Note: This proposal already has as a patch from 2015 by @bakineggs, but it appears to have fallen between the cracks.

I have the following case:

type Join struct {
    ChannelId string      `json:"channel_id"`
    Accounts  []Ident     `json:"accounts,omitempty"`
    History   []TextEntry `json:"history,omitempty"`
}

This struct is used for message passing and the slices are only relevant (and set to non-nil) in some cases. However, since encoding/json does not differentiate between a nil slice and an empty slice, there will be legitimate cases where a field is excluded when it's not expected to (e.g., the History slice is set, but empty).

I reiterate the proposal by Dan in his patch referred above to support an omitnil option which allows this differentiation for slices and maps.

Note for hypothetical Go 2.0: This is already how omitempty works for pointers to Go's basic types (e.g., (*int)(nil) is omitted while pointer to 0 is not). For Go 2.0 the behavior of omitempty could change to omit both nil and 0 when specified, and then only nil would be omitted when omitnil is specified.

[robyoder]

Yes please! This bit me today. We like to use omitempty to minimize the fields we need to include in a response. But we do need to differentiate between a nil slice and an empty slice, so we had to remove omitempty for slices and always include the field, so now it's an array or null. 🙁

[rsc]

Partial dup of #11939, which we may get to someday. Mostly JSON is done but better handling of zeros is on the long-term wish list.

[rsc]

Closing in favor of #11939, which now mentions this one.

[robyoder]

@rsc how is this even a partial dupe of that issue? Its focus is defining additional zero values to be counted as "empty" by omitempty. This issue is focused on the fact that there is no way to differentiate between nil and empty slices with omitempty. How does #11939 address that?

[blixt]

@rsc That proposal you referred seems to make omitempty exclude even more values from being encoded. This proposal is for doing the opposite – keep values that are today hidden from the output!

To clarify this proposal, here's an example. I want to be able to omit the Contents property for the Black Box (as it works today), but keep it for the Empty Box:

package main

import "encoding/json"
import "fmt"

type Box struct {
	Label    string
	Contents []string `json:",omitempty"`
}

func main() {
	var d []byte
	d, _ = json.Marshal(Box{Label: "Black Box"})
	fmt.Println(string(d))
	d, _ = json.Marshal(Box{Label: "Empty Box", Contents: []string{}})
	fmt.Println(string(d))
	d, _ = json.Marshal(Box{Label: "Banana Box", Contents: []string{"Banana", "Another banana"}})
	fmt.Println(string(d))
}

Here's the output I want, after changing omitempty to omitnil:

{"Label":"Black Box"}
{"Label":"Empty Box","Contents":[]}
{"Label":"Banana Box","Contents":["Banana","Another banana"]}

Actual output:

{"Label":"Black Box"}
{"Label":"Empty Box"}
{"Label":"Banana Box","Contents":["Banana","Another banana"]}

[blixt]

Further clarification: The patch mentioned above adds support to encoding/json to differentiate on []string(nil) vs. []string{} (through a new keyword omitnil so backwards compatibility is not broken), allowing nil slices to be excluded without excluding empty slices.

[ianlancetaylor]

I guess I'll reopen this, but to be honest it sounds like a bad idea to me. Distinguishing between a nil slice and an empty slice is sufficiently confusing that it is generally a mistake. I don't think that the additional complexity in the already very complex encoding/json package is worth it.

[robyoder]

@ianlancetaylor well, the encoder distinguishes. It encodes an empty slice as [] and a nil slice as null. But including a null property in the JSON is kinda pointless. Including [] is not.

[bakineggs]

In go, you shouldn't need to distinguish between a nil slice and an empty slice because they're generally treated the same. In JSON and many languages in which people parse JSON, null arrays and empty arrays are treated differently.

Ideally, the consumers of your JSON will be able to handle output that has either a null array, an empty array, or a missing key. In the real world, customers often demand that you supply output exactly how they want it. Product managers insist that engineers do the things that customers who pay boatloads of money want, so that's why I had to write the omitnil patch and run a product I built for a previous employer on a fork of go that includes this patch (and the poor souls who inherited that product have to continue maintaining that fork). It appears that my situation was not unique and others have to deal with similar demands, so it would be helpful if they didn't also have to maintain their own forks of go.

[rsc]

On hold for #11939.