Optionally exclude defaults and `None`

[redruin1]

I have a seemingly simple problem: I have an attrs class that I want to serialize to a Python dictionary, but I want to be able to prune outputs if they're None or equal to their default values, e.g:

@attrs.define
class Example:
    a: int
    b: int | None
    c: int = 123
    
    def to_dict(self, exclude_none: bool, exclude_defaults: bool) -> dict:
        ... # TODO

e = Example(a=100, b=None)
e.to_dict(exclude_none=False, exclude_defaults=False) # {'a': 100, 'b': None, 'c': 123}
e.to_dict(exclude_none=True, exclude_defaults=False) # {'a': 100, 'c': 123}
e.to_dict(exclude_none=False, exclude_defaults=True) # {'a': 100, 'b': None}
e.to_dict(exclude_none=True, exclude_defaults=True) # {'a': 100}

omit_if_default seems to do half of what I want already, and I was hopeful that I could do something like:

converter = cattrs.Converter(omit_if_default=True)

class Example:
    ...
    def to_dict(self, exclude_none: bool, exclude_defaults: bool) -> dict:
        converter.omit_if_default = exclude_defaults
        return converter.unstructure(self)

But after taking a closer look, it seems the unstructure function is only generated once with the current value of omit_if_default, and then cached for reuse (for performance reasons I imagine). Does this imply that the "correct" way to do this is to create 4 separate hooks for every possible class I want to unstructure, and then select between them based on the passed in arguments?

If push comes to shove I can just generate the maximum dict and then manually prune myself:

converter = cattrs.Converter(omit_if_default=False)

class Example:
    ...
    def to_dict(self, exclude_none: bool, exclude_defaults: bool) -> dict:
        defaults = {field.name: field.default for field in attrs.fields(type(self))}
        return {
            k: v for k, v in converter.unstructure(self, type(self)).items()
            if not ((v is None and exclude_none) or (getattr(self, k) == defaults[k] and exclude_defaults))
        }

But this doesn't support removing None/defaults recursively through the structure, and it feels like I'm circumventing cattrs instead of properly using it.

[Tinche]

Hm. Since you need to recursively apply the rules, I would probably go with 4 converters, and have a dictionary with all 4 combinations. You can copy an existing container 3 times if you have some existing rules you'd like to keep.

Two converters with omit_if_default, two without, two with additional hooks for attrs and dataclasses that filter out none, and two without.

Going to close the issue now to keep the issue tracker clear, but we can continue the discussion. Let me know if you need more guidance.

[redruin1]

Thanks for the reply. I kind of figured I would probably have to do something like that - I'm using a similar kind of framework in order to select converters targeting different versions, so I suppose I can just adapt that system to generate the new converters on-demand.

When I was experimenting with how I could modify cattrs' behavior to circumvent this, I realized that from a user's perspective the ability to pass in a custom callable instead of just a static boolean could be a very ergonomic solution:

def equals_default(self, attribute, value) -> bool:
    """
    This is essentially equivalent to `_cattrs_omit_if_default`. 
    Returns `True` if this attribute should be omitted.
    """
    if isinstance(attribute.default, attrs.Factory):
        if attribute.default.takes_self:
            return value == attribute.default(self)
        else:
            return value == attribute.default()
    else:
        return value == attribute.default

converter.register_unstructure_hook(
    Example,
    cattrs.gen.make_dict_unstructure_fn(
        Example,
        converter,
        _cattrs_omit_if=equals_default, # default behavior is exclude defaults
        always_include = cattrs.gen.override(
            omit=False,
        ),
        always_exclude = cattrs.gen.override(
            omit=True,
        ),
        only_omit_if_none = cattrs.gen.override(
            omit_if=lambda s, a, v: v is None
        )
    )
)

This would allow you to get rid of the specific omit_if_default argument and make it entirely customizable to the users needs; so all I would have to do would be to simply create a set of custom functions and then pass them in when creating the converters that I want. After taking a cursory look at the generation functions, this doesn't seem impossible to implement either by simply replacing the hard-coded logic with a call to the passed in function, but I suspect it might incur a tiny performance hit from the additional indirection on each attribute set.

[redruin1]

Since this is more of a discussion thread, I'll tack this issue onto here rather than cluttering up the issue tracker as you said. I'm looking to create structuring/unstructuring functions that can serialize and deserialize dicts which formats (slightly) differ in structure from the attrs classes that I want to make.

# Suppose I'm given a dict like this (where the format is predefined and cannot be changed)
button_dict = {
    "control_behavior": { 
        "enabled": True # Here, the information I want is stored inside of "control_behavior"
    }
}

# However, I want my attrs class to be structured like this, 
# since "control_behavior" is essentially an implementation detail to the users:
@attrs.define
class Button:
    enabled: bool

I could add control_behavior to Button and implement enabled as a property, but it doesn't really "hide" the backend, especially since on different versions control_behavior might not exist, while enabled always will. Implementing the structuring/unstructuring functions manually are of course options, but are against the principles of cattrs. Essentially, I just want to specify that input_dict["control_behavior"]["enabled"] should point to Button.enabled when structuring, and vice-versa when unstructuring, whereas currently in cattrs generators it is a hardcoded 1-to-1 relationship between dict keys and class attributes.

Could this limitation be relaxed somewhat? I actually managed to cobble a cursed solution for structuring by adding an additional location option to override:

cattrs.gen.make_dict_structure_fn(
      Button,
      converter,
      enabled=cattrs.gen.override(location="['control_behavior']") # defaults to ""
  )

And by replacing all instances of "o[{kn}]" to be "o{location}[{kn}]" in make_dict_structure_fn, which miraculously happens to totally work without issue:

print(converter.structure(button_dict, Button))
# Button(enabled=True)

What are your thoughts on this? While the above works, I think a better solution would be specifying as a single string or tuple of strings, each one representing a key access value:

enabled=cattrs.gen.override(
    location=("control_behavior", "enabled")
)

This could even superceed rename, since it encapsulates it's behavior, just more flexibly:

attribute=cattrs.gen.override(
    # location="attribute" # by default, accessed key is the same name
)
attribute2=cattrs.gen.override(
    location="different_name" # equivalent to `rename="different_name"`
)
attribute3=cattrs.gen.override(
    location=("nested", "different_name") # different structure and rename functionality, all in one
)

Adapting unstructuring would be less straightforward, but I think it could be handled by writing multiple dictionaries using the same logic in a sequence of root-most to leaf-most:

def unstructure_Button(inst)
    res = {
        ... # handle root-most keys
    }
    res["control_behavior"] = {
       "enabled": inst.enabled
    }
    res["control_behavior"]["another_nested_object"] = {
        ... # and so on...
    }
    ...
    # post init statements...
    ...
    return res