Typechecking job failing: `error: the configured Python interpreter version (3.14) is newer than PyO3's maximum supported version (3.13)`

[MadLittleMods]

I don't know why it's trying to use Python 3.14 as I don't see it configured anywhere in our CI.

Relevant error in the Typechecking job:

error: the configured Python interpreter version (3.14) is newer than PyO3's maximum supported version (3.13)

More logs:

❌ https://github.com/element-hq/synapse/actions/runs/18957150720/job/54136538943?pr=19116#step:5:641

    | error: failed to run custom build command for `pyo3-ffi v0.24.1`
    | 
    | Caused by:
    |   process didn't exit successfully: `/tmp/tmpmx5s4dqz/pydantic_core-2.33.2/target/release/build/pyo3-ffi-fc7839d0f716f7d0/build-script-build` (exit status: 1)
    |   --- stdout
    |   cargo:rustc-check-cfg=cfg(Py_LIMITED_API)
    |   cargo:rustc-check-cfg=cfg(Py_GIL_DISABLED)
    |   cargo:rustc-check-cfg=cfg(PyPy)
    |   cargo:rustc-check-cfg=cfg(GraalPy)
    |   cargo:rustc-check-cfg=cfg(py_sys_config, values("Py_DEBUG", "Py_REF_DEBUG", "Py_TRACE_REFS", "COUNT_ALLOCS"))
    |   cargo:rustc-check-cfg=cfg(pyo3_disable_reference_pool)
    |   cargo:rustc-check-cfg=cfg(pyo3_leak_on_drop_without_reference_pool)
    |   cargo:rustc-check-cfg=cfg(Py_3_7)
    |   cargo:rustc-check-cfg=cfg(Py_3_8)
    |   cargo:rustc-check-cfg=cfg(Py_3_9)
    |   cargo:rustc-check-cfg=cfg(Py_3_10)
    |   cargo:rustc-check-cfg=cfg(Py_3_11)
    |   cargo:rustc-check-cfg=cfg(Py_3_12)
    |   cargo:rustc-check-cfg=cfg(Py_3_13)
    |   cargo:rustc-check-cfg=cfg(Py_3_14)
    |   cargo:rerun-if-env-changed=PYO3_CROSS
    |   cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
    |   cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
    |   cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
    |   cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
    |   cargo:rerun-if-env-changed=PYO3_USE_ABI3_FORWARD_COMPATIBILITY
    | 
    |   --- stderr
    |   error: the configured Python interpreter version (3.14) is newer than PyO3's maximum supported version (3.13)
    |   = help: please check if an updated version of PyO3 is available. Current version: 0.24.1
    |   = help: set PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 to suppress this check and build anyway using the stable ABI
    | warning: build failed, waiting for other jobs to finish...
    | 💥 maturin failed
    |   Caused by: Failed to build a native library through cargo
    |   Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_BUILD_EXTENSION_MODULE="1" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.14-64bit" PYO3_PYTHON="/tmp/tmpbrzhi_g3/.venv/bin/python" PYTHON_SYS_EXECUTABLE="/tmp/tmpbrzhi_g3/.venv/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/tmpmx5s4dqz/pydantic_core-2.33.2/Cargo.toml" "--release" "--lib" "--crate-type" "cdylib"`
    | Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/tmpbrzhi_g3/.venv/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

[anoadragon453]

This would be because we're not specifying a particular python_version to the matrix-org/setup-python-poetry action:

https://github.com/element-hq/synapse/blob/develop/.github/workflows/tests.yml#L162-L172

Thus we'll (eventually) get the latest stable Python, and as the codebase doesn't currently support Python 3.14, we get build failures. Python 3.14 support is currently being worked on in #19055

In the meantime we should pin our Python versions in CI to avoid CI breaking on develop without warning.

[anoadragon453]

Actually, the above is only part of the story. We should support Python 3.14 already, since PyO3 0.25.0+ supports it, and we specify PyO3 0.25.1 in our Cargo.toml:

synapse/rust/Cargo.toml

Lines 33 to 38 in 3ccc518

	pyo3 = { version = "0.25.1", features = [
	"macros",
	"anyhow",
	"abi3",
	"abi3-py310",
	] }

which is reflected in our Cargo.lock:

synapse/Cargo.lock

Lines 815 to 831 in 3ccc518

	[[package]]
	name = "pyo3"
	version = "0.25.1"
	source = "registry+https://github.com/rust-lang/crates.io-index"
	checksum = "8970a78afe0628a3e3430376fc5fd76b6b45c4d43360ffd6cdd40bdde72b682a"
	dependencies = [
	"anyhow",
	"indoc",
	"libc",
	"memoffset",
	"once_cell",
	"portable-atomic",
	"pyo3-build-config",
	"pyo3-ffi",
	"pyo3-macros",
	"unindent",
	]

So why, oh why, is this CI job failing with:

help: please check if an updated version of PyO3 is available. Current version: 0.24.1

[anoadragon453]

There is a Rust cache that's being utilised in the linked run:

Restored from cache key "v0-rust-lint-mypy-Linux-x64-0c6f0f96-7f8eb0c4" full match: true.

I've deleted that cache entry and re-run the CI job. Let's see what happens: https://github.com/element-hq/synapse/actions/runs/18957150720/job/54194762215?pr=19116

[anoadragon453]

...and that retry used Python 3.13 instead: https://github.com/element-hq/synapse/actions/runs/18957150720/job/54194762215?pr=19116#step:5:41

wtf?

Edit: Aha, we got a different, randomly-assigned GitHub runner version for each run. That would explain why sometimes we're getting Python 3.14 (which we don't pin, and which fails) vs. Python 3.13.

The Python 3.14 runner (first run linked above):

Runner Image
  Image: ubuntu-24.04
  Version: 20251030.96.2
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20251030.96/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20251030.96

The Python 3.13 runner (second run):

Runner Image
  Image: ubuntu-24.04
  Version: 20250929.60.1
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250929.60/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250929.60

The release notes of the 3.14 runner confirm the Python 3.14 update.

[anoadragon453]

A recent typechecking job passed on Python 3.14: https://github.com/element-hq/synapse/actions/runs/19040990305/job/54377870833?pr=19116

This branch includes #19055 (Python 3.14). So I think things are now fixed (potentially due to rust caches, which included an old PyO3, got busted by that PR).