Allow different test output for different Python versions

[cdce8p]

Followup to #10381 (comment)

Turns out the test suite does have an option to specify different output messages for different Python version already. It just wasn't usable as the output message check would subsequently fail.

This could be a good option where ever only the output messages between version changed / a pinning py-version was previously necessary. This does not replace different test cases for added syntax or if the config files are different. E.g. a lot of the typing extension test cases look quite similar but use runtime-typing yes and no.

[Pierre-Sassoulas]

This looks amazing. Especially the doc ! The code is a little too terse and hard to tests imo, We had a lot of issue in the past with testutils not doing what we expected it to do. Then we trust the CI and bad things happens. I would use the following checker to test the changes:

class TestutilVersionChecker(BaseChecker):
    name = 'version-checker'
    msgs = {
        "E9009": ("Running on Python 3.9",),
        "E9010": ("Running on Python 3.10",),
        "E9011": ("Running on Python 3.11",),
        "E9012": ("Running on Python 3.12",),
        "E9013": ("Running on Python 3.13",),
        "E9014": ("Running on Python 3.14",),
    }

    def visit_module(self, node):
        major, minor = sys.version_info[:2]
        error_code = f"E90{minor:02d}"
        self.add_message(error_code, node=node)

Also I wonder how multiple messages on the same line with some conditional on the interpreter are going to be handled # <3.14:[undefined-variable] # <3.13:[syntax-error] ? (the reason why I put the interpreter info just after the message name in something = 1 # [unused-variable|>=3.13,missing-module-docstring])

[cdce8p]

Also I wonder how multiple messages on the same line with some conditional on the interpreter are going to be handled # <3.14:[undefined-variable] # <3.13:[syntax-error] ? (the reason why I put the interpreter info just after the message name in something = 1 # [unused-variable|>=3.13,missing-module-docstring])

Not sure that's allowed with the current test syntax, though haven't tested it. What does work though is to prefix a message with an offset. E.g. to separate two messages on the same line, we could use something like this:

# +2:<3.13: [syntax-error]
# +1:<3.14: [undefined-variable]
var: x = 2  # some code which raised pylint warnings

In theory, I'd be open to explore other syntax forms. Just used the existing one for the initial implementation as it seems to work good enough and has been (partially) supported for some time now. The pattern is defined here

pylint/pylint/testutils/constants.py

Lines 25 to 31 in 1257474

	_EXPECTED_RE = re.compile(
	r"\s*#\s*(?:(?P<line>[+-]?[0-9]+):)?" # pylint: disable=consider-using-f-string
	r"(?:(?P<op>[><=]+) *(?P<version>[0-9.]+):)?"
	r"\s*\[(?P<msgs>{msg}(?:,\s*{msg})*)]".format(**_MESSAGE)
	)
	_OPERATORS = {">": operator.gt, "<": operator.lt, ">=": operator.ge, "<=": operator.le}

The code is a little too terse and hard to tests imo, We had a lot of issue in the past with testutils not doing what we expected it to do. Then we trust the CI and bad things happens. I would use the following checker to test the changes:

I agree. Thanks for the example. Will add some tests for it.

[Pierre-Sassoulas]

What does work though is to prefix a message with an offset. E.g. to separate two messages on the same line, we could use something like this:

# +2:<3.13: [syntax-error]
# +1:<3.14: [undefined-variable]
var: x = 2  # some code which raised pylint warnings

Ha, right, not super elegant but it's not like its going to happen all the time anyway. Let's document it, and keep it as it is. Maybe we can use this for the doc:

# +2:>3.13: [something-else]
# +1:>=3.13: [undefined-variable]
var: x = 2  # <3.13: [syntax-error]

[jacobtylerwalls]

Thank you!

[cdce8p]

Rebased the PR to include the PyPy fix already in main. Only the last two commits are "new":
https://github.com/pylint-dev/pylint/pull/10382/files/d0454ae38ec268b38c12f884e953188f689267ba..e2528113d2de537d90aefd032e961740815bd6d4

Only added tests to make sure the output file matching works as expected. Not sure we test the message regex anywhere expect for the tests themselves. Though the TestutilVersionChecker idea would likely work, not sure that's really needed, especially since it already worked and we're likely to add more uses to the functional tests themselves in the future.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.90%. Comparing base (5fce790) to head (6a530b9).
⚠️ Report is 302 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10382      +/-   ##
==========================================
+ Coverage   95.84%   95.90%   +0.06%     
==========================================
  Files         174      176       +2     
  Lines       18995    19135     +140     
==========================================
+ Hits        18206    18352     +146     
+ Misses        789      783       -6     

Files with missing lines	Coverage Δ
pylint/testutils/functional/test_file.py	96.82% <100.00%> (+0.52%)	⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🤖 According to the primer, this change has no effect on the checked open source code. 🤖🎉

This comment was generated for commit 6a530b9

[cdce8p]

Got a bit distracted last week. Anything to add here @Pierre-Sassoulas? Otherwise I'd go ahead and merge it.

[Pierre-Sassoulas]

Great change, let's merge !