Proposal: control test distribution in pytest-xdist #17
Description
Here's a proposal on how to control how tests are distributed among slaves in pytest-xdist in the --dist=load mode, feature that has been requested by a lot of people in pytest-dev/pytest#175.
Note: the names here are just an initial thought, suggestions are more than welcome.
Scenarios
People have requested a number of scenarios. Here I will summarize the more popular ones and show a proposed API to support each one.
Overview
pytest-xdist in --dist=load will split tests over all slaves, where each test will run in exactly one slave distributing the test load across multiple processes and/or remotes.
Serial execution of tests of the same Class
The most popular scenario by far, people would like to somehow mark a class or module so all tests from that class/module would execute in the same slave.
import pytest
@pytest.mark.xdist_same_slave
class TestFoo:
def test_1(self):
pass
def test_2(self):
pass
@pytest.mark.xdist_same_slave
class TestBar:
def test_3(self):
pass
def test_4(self):
passThe @pytest.mark.xdist_same_slave decorator when applied to a class will instruct xdist to run all tests of that class in the same slave. Which slave will be assigned to run tests from each class will be determined at runtime in a round-robin fashion, so as test items from classes with this decorator ar scheduled for execution, they will be assigned to execute on the same node, but different classes will have their tests assigned to different nodes. In the example above with -n 2 pytest-xdist will executes test from TestFoo in slave 0 and tests from TestBar in slave 1. The actual slave numbers are not guaranteed to be the same between runs and will be determined at runtime, with xdist doing its best to distribute tests to available slaves while keeping the constraint of binding marked tests to run in the same slave.
As tests execute in the same slave, their execution will be effectively serial.
Serial execution of tests of the same Module
Similar to the above, but will serialize all tests in a module, both contained in test classes or free functions:
import pytest
pytestmark = pytest.mark.xdist_same_slave
class Test:
def test_1(self):
pass
def test_2(self):
pass
def test_a():
passSerial execution of tests which use a fixture
import xdist, pytest
@xdist.xdist_same_slave
@pytest.fixture(scope='session')
def db_session():
return DBSession()
@pytest.yield_fixture
def db(db_session):
db_session.begin_transaction()
yield
db_session.end_transaction()This will make all tests which use db execute in the same slave (which one exactly to be determined at runtime) because db depends on db_session, which is bound to execute in a single slave. This will effectively setup the database in a single slave once, and all tests which use db will run in that slave.
Unfortunately we can't use pytest.mark with fixtures, so we have to rely on a normal decorator.
Opinions and suggestions are more than welcome!