Pytest holds on to fixtures too long causing a memory leak

[matham]

Pytest seems to hold on to the object yielded, even after the test is done, sometimes, even until all the tests are done.

The following code:

import pytest
import gc
import weakref


class SomethingInteresting(object):

	def check_app(self):
		assert True
		print('app checked')


@pytest.fixture
def my_app():
	app = SomethingInteresting()
	yield app

	app = weakref.ref(app)
	gc.collect()
	print('app is', app())
	if app() is not None:
		assert False


def test_my_app(my_app):
	my_app.check_app()

Results in this:

pytest /g/Python/libs/Playground/src/playground15.py
============================================================================================================= test session starts ============================================================================================================= platform win32 -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: G:\Python\libs\Playground
collected 1 item

..\Playground\src\playground15.py .E                                                                                                                                                                                                     [100%]

=================================================================================================================== ERRORS ==================================================================================================================== ______________________________________________________________________________________________________ ERROR at teardown of test_my_app _______________________________________________________________________________________________________

    @pytest.fixture
    def my_app():
        app = SomethingInteresting()
        yield app

        app = weakref.ref(app)
        gc.collect()
        print('app is', app())
        if app() is not None:
>               assert False
E     assert False

..\Playground\src\playground15.py:22: AssertionError
------------------------------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------------------------------- app checked
---------------------------------------------------------------------------------------------------------- Captured stdout teardown ----------------------------------------------------------------------------------------------------------- app is <src.playground15.SomethingInteresting object at 0x0000021F5F295908>
====================================================================================================== 1 passed, 1 error in 0.09 seconds ======================================================================================================

I tried to investigate using objgraph, and various pytest modules (e.g. warning) were blamed depending on the exact code. When I tested the app after the fixture has fully returned at the next test, then the app seemed to have been released.

However, in a more complex app that I have, pytest seemed to hold on to the app fixture instances forever, untill all the test has finished. I'm attaching what objgraph shows for this case that Terminal reporter is holding on to it.

Tested on Windows10, py3.7.

pytest                        4.6.3
pytest-cov                    2.7.1
pytest-trio                   0.5.2

[blueyed]

Just some quick ideas: is this a regression maybe? In that case git-bisect could be useful.
In either case trying the latest pytest might be good - but maybe you are stuck on Python 2?

[matham]

I updated to pytest 5.0.1 and the result was the same (assert False was hit)

[matham]

Is there a pytest version I should give a try to see if this is a regression? I can downgrade and test to see if a bisect would help at all.

[blueyed]

bisect

Likely not then - just thought that you've might have noticed this yourself as a regression.
But might be worth a try still, if you can easily reproduce it (and as long as older pytest versions work for you / your code, of course :))

[asottile]

I'm not seeing a leak here, or at least the memory for that is freed once the fixture is used up

The way fixtures work is they are (potentially) reused so a fixture may be called once and used multiple times. What pytest does is store the fixture's yielded / returned value in the FixtureDef object which usually outlives the fixture's function frame itself

when that fixture is no longer needed, the FixtureDef's finalization is called which includes tearing down the cached result

pytest/src/_pytest/fixtures.py

Lines 860 to 861 in 13c4b7d

	if hasattr(self, "cached_result"):
	del self.cached_result

so I think this is not a bug

[asottile]

here is, for example, a test demoing that the object in fact does not leak:

import pytest
import gc
import weakref


class C(object):

    def check_app(self):
        assert True
        print('app checked')


@pytest.fixture
def my_app():
    app = C()
    yield app

    app = weakref.ref(app)


def test_my_app(my_app):
    my_app.check_app()


def test2():
    obj = next(iter(o for o in gc.get_objects() if isinstance(o, C)), None)
    assert obj is None

$ pytest t.py -vs
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /tmp/y/venv/bin/python3.7
cachedir: .pytest_cache
rootdir: /tmp/y
collected 2 items                                                              

t.py::test_my_app app checked
PASSED
t.py::test2 PASSED

=========================== 2 passed in 0.01 seconds ===========================

[matham]

But what about the objgraph image I uploaded? Admittedly, it I don't have example code as I saw it in a larger app. That image was generated in a test subsequent to the test that used the fixture. I.e. the fixture was already done, but terminal reporter seems to have held on to that until all the tests were sone. I saw similar behavior when there are warnings in the logs

I'll try to replicate this.

[matham]

Also, for the example code I posted, is there some post fixture hook I can use to make sure the fixture has been garbage collected? Rather than having to wait for the next test. E.g. if I wanted to make sure the fixture is not being leaked.

[asottile]

I tried reading the objgraph but have no idea what I'm looking at and can't ^F -- could you post an svg instead?

[matham]

Sorry I'm on a phone now so I cannot crop it, and anyway, the only relevant thing afaict in the giant graph is that the object at the root that ultimately keeps all these other objects alive, which ultimately keeps the app object alive, is a TerminalReporter object.

In other words it seems like TerminalReporter keeps the app fixture alive until all the tests are all done. Is it possible those logging object are keeping a ref to the fixtures?

I'll see if I can replicate this with a minimal example.

[matham]

Ok, I have reproducible example:

import pytest
import gc
import weakref


class SomethingInteresting(object):

    def check_app(self):
        assert True
        print('app checked')

    async def sleep(self):
        pass


apps = []


@pytest.fixture
async def my_app():
    app = SomethingInteresting()
    app.sleep()
    yield app

    app = weakref.ref(app)
    apps.append(app)
    gc.collect()

    print('app is', app())
    if len(apps) >= 2 and apps[0]() is not None:
        assert False


async def test_my_app(my_app):
    my_app.check_app()


async def test_my_app2(my_app):
    my_app.check_app()

As you can see, I save a ref to the current app fixture in apps, and then check in the subsequent test if the last fixture is still alive. If I remove the line app.sleep(), everything passes, otherwise, I get the warning it (as I should) was never awaited on and the assert False is triggered:

============================================================================================================= test session starts ============================================================================================================= platform win32 -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: G:\Python\libs\ceed, inifile: pytest.ini
plugins: asyncio-0.10.0, cov-2.7.1, trio-0.5.2
collected 2 items

ceed\tests\playground15.py ..E                                                                                                                                                                                                           [100%]

=================================================================================================================== ERRORS ==================================================================================================================== ______________________________________________________________________________________________________ ERROR at teardown of test_my_app2 ______________________________________________________________________________________________________

    def finalizer():
        """Yield again, to finalize."""
        async def async_finalizer():
            try:
                await gen_obj.__anext__()
            except StopAsyncIteration:
                pass
            else:
                msg = "Async generator fixture didn't stop."
                msg += "Yield only once."
                raise ValueError(msg)

>       loop.run_until_complete(async_finalizer())

e:\python\python37\lib\site-packages\pytest_asyncio\plugin.py:93:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ C:\Users\Matthew Einhorn\AppData\Local\Programs\Python\Python37\Lib\asyncio\base_events.py:584: in run_until_complete
    return future.result()
e:\python\python37\lib\site-packages\pytest_asyncio\plugin.py:85: in async_finalizer
    await gen_obj.__anext__()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    @pytest.fixture
    async def my_app():
        app = SomethingInteresting()
        app.sleep()
        yield app

        app = weakref.ref(app)
        apps.append(app)
        gc.collect()

        print('app is', app())
        if len(apps) >= 2 and apps[0]() is not None:
>           assert False
E           assert False

ceed\tests\playground15.py:31: AssertionError
------------------------------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------------------------------- app checked
---------------------------------------------------------------------------------------------------------- Captured stdout teardown ----------------------------------------------------------------------------------------------------------- app is <playground15.SomethingInteresting object at 0x000001E33C6D6588>
============================================================================================================== warnings summary =============================================================================================================== ceed/tests/playground15.py::test_my_app
ceed/tests/playground15.py::test_my_app2
  G:\Python\libs\ceed\ceed\tests\playground15.py:22: RuntimeWarning: coroutine 'SomethingInteresting.sleep' was never awaited
    app.sleep()

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================================================================ 2 passed, 2 warnings, 1 error in 0.21 seconds ================================================================================================

So essentially, when there's a warning, the fixture never dies. The same seemed to happen if app.sleep() was dropped in test_my_app instead.

Using pytest-trio instead of pytest-asyncio had the same issue. Here's the objgraph for the app at the time the assertion failed for whatever it's worth showing that pytest (tree root) is keeping the app object alive (final leaf at the bottom):

[asottile]

at least in that case:

• cpython constructs a warning which points at the source coroutine: https://github.com/python/cpython/blob/cfe172dc6bd0a02d36db31ffabcc38f9320a4510/Lib/warnings.py#L510
• https://github.com/python/cpython/blob/cfe172dc6bd0a02d36db31ffabcc38f9320a4510/Lib/warnings.py#L398
• the warnings are held to be displayed at the end of the run

not sure there's anything to do here 🤷‍♂

the only ~slighty improvement would be to tear down the PytestPDB global at the end of the run, but even then I don't think that helps you since the cpython warning system is the one keeping your frame alive so long

[blueyed]

As for the warnings: could be collect/stringify them earlier in pytest?
Or could/should Python not keep the ref in the first place, but format it already (earlier)?

It might also be that that's the case already, and __pytestPDB / pytestPDB is the problem after all? (this is via the import for set_trace)

[asottile]

it's not just pytest internally, they're captured and exposed through this hook: https://pytest.readthedocs.io/en/latest/reference.html#_pytest.hookspec.pytest_warning_captured

since this hook is historic=True (presumably because warnings can occur during startup before plugins are registered) it records all calls indefinitely