Special case TypedDict.get

[randolf-scholz]

Makes TypedDict.get smarter by producing a list of overloads corresponding to the key-value pairs.

from typing import TypedDict, Required

class Demo(TypedDict, total=False):
    x: Required[int]
    y: str

def test(d: Demo) -> None:
    reveal_type(d.get)

On 1.17.1:

note: Revealed type is "Overload(
    def (builtins.str) -> builtins.object,
    def [_T] (builtins.str, default: builtins.object) -> builtins.object,
)"

With this PR:

note: Revealed type is "Overload(
    def (Literal['x'], builtins.object =) -> builtins.int,
    def (Literal['y']) -> builtins.str | None,
    def (Literal['y'], builtins.str) -> builtins.str,
    def [T] (Literal['y'], T`-1 =) -> builtins.str | T`-1,
    def (builtins.str, builtins.object =) -> builtins.object
)"

I think in principle we should be able to combine the last two y overloads like pyright does it, but this likely needs some changes in generic callable inference.

Moreover, we now get the non-optional type when doing TypedDict.get(required_key):

from typing import TypedDict, Required

class Demo(TypedDict, total=False):
    x: Required[int]
    y: str

def test(d: Demo) -> None:
    reveal_type(d.get("x"))  # master: int | None, PR: int
    reveal_type(d.get("y"))  # master: str | None, PR: str | None

We get slightly different behavior with List/Set/Dict expressions:

from typing import TypedDict

class C(TypedDict):
    a: int

class Demo(TypedDict, total=False):
    x: list[int]
    y: C

def test(d: Demo):
    reveal_type(d.get("x", ["x"]))
    reveal_type(d.get("y", {}))   

master: https://mypy-play.net/?mypy=master&python=3.12&gist=b3cac3b9355a53f8a5a1ad3fcdd8bbba

main.py:11: note: Revealed type is "builtins.list[builtins.int] | builtins.list[builtins.str]"
main.py:12: note: Revealed type is "TypedDict('__main__.C', {'a'?: builtins.int})"

1.17.1: https://mypy-play.net/?mypy=latest&python=3.12&gist=dfb9f9df42e01a143f8f14d644347cc6

main.py:11: note: Revealed type is "builtins.list[builtins.int]"
main.py:11: error: List item 0 has incompatible type "str"; expected "int"  [list-item]
main.py:12: note: Revealed type is "TypedDict('__main__.C', {'a'?: builtins.int})"

This PR:

note: Revealed type is "builtins.list[builtins.int] | builtins.list[builtins.str]"
note: Revealed type is "TypedDict('tmp.C', {'a': builtins.int}) | builtins.dict[Never, Never]"

Since in the example above, C is total, so an empty dict is not compatible with C. Therefore, the behavior on both master and 1.17.1 is incorrect in this example.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

operator (https://github.com/canonical/operator)
- ops/model.py:1149: error: Incompatible types in assignment (expression has type "str | None", variable has type "str")  [assignment]
- ops/model.py:1223: error: Incompatible types in assignment (expression has type "str | None", variable has type "str")  [assignment]
- ops/_private/harness.py:2280: error: Value expression in dictionary comprehension has incompatible type "str | int | float | None"; expected type "str | int | float"  [misc]

paasta (https://github.com/yelp/paasta)
+ paasta_tools/kubernetes_tools.py:1801: error: Need type annotation for "crypto_keys"  [var-annotated]

core (https://github.com/home-assistant/core)
+ homeassistant/data_entry_flow.py:645: error: Unused "type: ignore" comment  [unused-ignore]
+ homeassistant/data_entry_flow.py:650: error: Unused "type: ignore" comment  [unused-ignore]
- homeassistant/components/evohome/storage.py:98: error: Argument 1 to "fromisoformat" of "date" has incompatible type "object"; expected "str"  [arg-type]
+ homeassistant/components/fritz/coordinator.py:397: error: Statement is unreachable  [unreachable]
+ homeassistant/components/fritz/coordinator.py:414: error: Incompatible types in assignment (expression has type "bool | None", variable has type "bool")  [assignment]

steam.py (https://github.com/Gobot1234/steam.py)
- steam/http.py:62: error: Incompatible types in assignment (expression has type "int", variable has type "Language")  [assignment]
+ steam/http.py:62: error: Incompatible types in assignment (expression has type "object", variable has type "Language")  [assignment]
- steam/http.py:349: error: Value of "last_assetid" has incompatible type "str | int"; expected "str"  [typeddict-item]
- steam/user.py:125: error: Argument "game_name" to "CMsgClientPersonaStateFriend" has incompatible type "str | int"; expected "str"  [arg-type]
- steam/state.py:206: error: Incompatible types in assignment (expression has type "int", variable has type "PersonaState")  [assignment]
+ steam/state.py:206: error: Incompatible types in assignment (expression has type "object", variable has type "PersonaState")  [assignment]
- steam/state.py:207: error: Incompatible types in assignment (expression has type "int", variable has type "UIMode")  [assignment]
+ steam/state.py:207: error: Incompatible types in assignment (expression has type "object", variable has type "UIMode")  [assignment]
- steam/state.py:208: error: Incompatible types in assignment (expression has type "int", variable has type "PersonaStateFlag")  [assignment]
+ steam/state.py:208: error: Incompatible types in assignment (expression has type "object", variable has type "PersonaStateFlag")  [assignment]

discord.py (https://github.com/Rapptz/discord.py)
+ discord/automod.py:176: error: No overload variant of "get" of "dict" matches argument type "str"  [call-overload]
+ discord/automod.py:176: note: Possible overload variants:
+ discord/automod.py:176: note:     def get(self, Never, None = ..., /) -> None
+ discord/automod.py:176: note:     def get(self, Never, Never, /) -> Never
+ discord/automod.py:176: note:     def [_T] get(self, Never, _T, /) -> _T
- discord/scheduled_event.py:139: error: Incompatible types in assignment (expression has type "object", variable has type "str | None")  [assignment]
- discord/scheduled_event.py:145: error: Incompatible types in assignment (expression has type "object", variable has type "str | None")  [assignment]
- discord/scheduled_event.py:146: error: Incompatible types in assignment (expression has type "object", variable has type "int")  [assignment]
- discord/scheduled_event.py:150: error: Argument 1 to "store_user" of "ConnectionState" has incompatible type "object"; expected "User | PartialUser"  [arg-type]
- discord/scheduled_event.py:155: error: No overload variant of "parse_time" matches argument type "object"  [call-overload]
- discord/scheduled_event.py:155: note: Possible overload variants:
- discord/scheduled_event.py:155: note:     def parse_time(timestamp: None) -> None
- discord/scheduled_event.py:155: note:     def parse_time(timestamp: str) -> datetime
- discord/scheduled_event.py:155: note:     def parse_time(timestamp: str | None) -> datetime | None
- discord/scheduled_event.py:159: error: Argument 1 to "_unroll_metadata" of "ScheduledEvent" has incompatible type "object"; expected "EntityMetadata | None"  [arg-type]
- discord/poll.py:449: error: Item "None" of "PollMedia | None" has no attribute "get"  [union-attr]
- discord/poll.py:459: error: Argument "question" to "Poll" has incompatible type "str | Any | None"; expected "PollMedia | str"  [arg-type]
- discord/app_commands/models.py:224: error: No overload variant of "int" matches argument type "object"  [call-overload]
- discord/app_commands/models.py:224: note: Possible overload variants:
- discord/app_commands/models.py:224: note:     def __new__(cls, str | Buffer | SupportsInt | SupportsIndex | SupportsTrunc = ..., /) -> int
- discord/app_commands/models.py:224: note:     def __new__(cls, str | bytes | bytearray, /, base: SupportsIndex) -> int
- discord/app_commands/models.py:231: error: Incompatible types in assignment (expression has type "object", variable has type "bool")  [assignment]
- discord/app_commands/models.py:237: error: Argument 1 to "_from_value" of "ArrayFlags" has incompatible type "object"; expected "Sequence[int]"  [arg-type]
- discord/app_commands/models.py:243: error: Argument 1 to "_from_value" of "ArrayFlags" has incompatible type "object"; expected "Sequence[int]"  [arg-type]
- discord/app_commands/models.py:245: error: Incompatible types in assignment (expression has type "object", variable has type "bool")  [assignment]
- discord/app_commands/models.py:246: error: Argument 1 to "_to_locale_dict" has incompatible type "object"; expected "dict[str, str]"  [arg-type]
- discord/app_commands/models.py:247: error: Argument 1 to "_to_locale_dict" has incompatible type "object"; expected "dict[str, str]"  [arg-type]
- discord/app_commands/models.py:1059: error: Argument 1 to "_to_locale_dict" has incompatible type "object"; expected "dict[str, str]"  [arg-type]
- discord/app_commands/models.py:1060: error: Argument 1 to "_to_locale_dict" has incompatible type "object"; expected "dict[str, str]"  [arg-type]
- discord/app_commands/models.py:1158: error: Argument 1 to "_to_locale_dict" has incompatible type "object"; expected "dict[str, str]"  [arg-type]
- discord/app_commands/models.py:1159: error: Argument 1 to "_to_locale_dict" has incompatible type "object"; expected "dict[str, str]"  [arg-type]
- discord/message.py:869: error: "object" has no attribute "items"  [attr-defined]
- discord/interactions.py:247: error: Need type annotation for "raw_channel"  [var-annotated]
- discord/guild.py:244: error: Incompatible types in assignment (expression has type "int | None", variable has type "int")  [assignment]
- discord/guild.py:245: error: Incompatible types in assignment (expression has type "int | None", variable has type "int")  [assignment]

archinstall (https://github.com/archlinux/archinstall)
+ archinstall/lib/models/users.py:204: error: Left operand of "or" is always false  [redundant-expr]
+ archinstall/lib/models/authentication.py:48: error: Statement is unreachable  [unreachable]

[randolf-scholz]

• ops/model.py:1149: false positive on master (T | None despite key being required)
• ops/model.py:1223: false positive on master (T | None despite key being required)
• ops/_private/harness.py:2280 false positive on master (T | None despite key being required)
• paasta_tools/kubernetes_tools.py:1801: due to changed dict-expression behavior.
• homeassistant/components/evohome/storage.py:98 false positive on master (T | None despite key being required)
• homeassistant/components/fritz/coordinator.py due to inferring T rather than T | None for required key.
• steam errors seem mostly related to using some hacky custom enum types.
• archinstall/lib/models/users.py:204 false negative on master (T | None despite key being required)
• discord/automod.py:176: fails because data.get('metadata', {}).get('custom_message'), but with the current version data.get('metadata', {}) is some TypedDict or dict[Never, Never]

For the last one, possibly it would be better to infer dict[Any, Any] or dict[str, Never] rather than dict[Never, Never] to allow chaining get.