Honor return type of `__new__`

[JukkaL]

Currently the return type of A(...) is implicitly A even if A.__new__ return something else. If the return type is Any, maybe A(...) should have type Any, and if it's a subclass, we should also use that. Currently this is a little difficult to implement due to implementation limitations (we derive the type object identity from the return type of a type object). Maybe we should introduce a new attribute to Callable for this purpose.

[JukkaL]

This is follow-up to #982.

[gvanrossum]

Honestly I don't think that A() should ever return something that's not an instance of A (or a subclass), even though technically __new__() can return anything it likes. Somehow I worry that if __new__() were defined as returning Any for some class and from that we'd infer that A() returned Any we would weaken the type checking -- I'd much rather get an error if the declared return type of __new__() isn't A or a subclass of A.

Have you seen any real code that violates this rule?

[JukkaL]

No, I haven't seen real code that uses this, but I've seen tutorials that mention that this is possible -- they always mention that you probably shouldn't do that.

Okay, what about this:

• If the return type is Any (either implicit or explicit), we don't check return statements within __new__ but the type of A() is still A.
• If the return type of __new__ is not A or a subclass of A (and not Any), reject it as a type error.
• If the return type B is a proper subclass of A, A() has type B.

[gvanrossum]

SGTM

[ilevkivskyi]

@JukkaL On one hand, constructor returning an instance of different class is unacceptable in static typed languages. But on other hand, this feature is sometimes quite useful in Python. Even typing.py uses this for Union and for NamedTuple (since 3.6b1).

In general, it would be nice if mypy minimizes the number of false positives like flagging this code:

class C:
    def __new__(cls):
        return 0

C() + 1

with Unsupported operand types for + ("C" and "int"). Maybe there could be an option to allow this.

[davidroeca]

One thought: the return type of __new__, if explicitly defined and different from the class name itself, could implicitly be the class's supertype.

class ZeroLiteral:  # supertype: int
    def __new__(cls) -> int:
        return 0

This would also be a general alternative to python/typing#415 and could likely handle #3062

[chadrik]

We make use of __new__ to promote base classes to leaf classes in a few places. I don't love it, but it'll take some serious effort to untangle it, so I'm gonna +1 this for the case of base class promoting to subclasses.

Ours works something like this:

from typing import Union

class Base(object):
    def __init__(self, name):
        self.name = name

    def __new__(cls, name):
        # type: (str) -> Union[A, B]
        if cls is Base:
            if name == 'A':
                return A(name)
            else:
                return B(name)
        else:
            # A or B
            return object.__new__(cls)

class A(Base):
    pass

class B(Base):
    pass

b = Base('B')
reveal_type(b)

The revealed type is Base, but I expect it to Union[A, B].

Also, this seems related to #3307.

[nierob]

Another use case that I had is to have async constructor, poor man's __ainit__:

class WorkItem(metaclass=ABCMeta):
    async def __new__(cls, *args, **kwargs):
        """ Override __new__ to support our custom `async` constructors. """
        o = super().__new__(cls)
        await o.ainit(*args, **kwargs)
        return o

That actually fails in a different place, as in this case new returns the right thing, but

class BuildItem(WorkItem):    ...
toolsetItem = await BuildItem(...)

fails with:

error: Incompatible types in "await" (actual type "BuildItem", expected type "Awaitable[Any]")