ignore user-installed packages when running python in sanitycheck #21362

Flamefire · 2024-09-10T11:09:38Z

Running a plain python -c ... or python -m ... in ECs not using PythonPackage might pick up packages installed in $HOME/.local/lib/python*.
As we haven't set $PYTHONNOUSERSITE in those ECs/EasyBlocks we need to pass -s to python to the same effect.
This avoids unexpected failures or success caused by that user environment.

SebastianAchilles · 2024-09-10T13:14:04Z

@boegelbot please test @ jsc-zen3
EB_ARGS="--sanity-check-only --filter-ecs=OptiX-7.2.0.eb,2019a,2020a,2020b,2021a,2021b,2022a,2022b,8.2.0,8.3.0,9.3.0,10.2.0,10.3.0"

SebastianAchilles · 2024-09-10T13:16:23Z

@boegelbot please test @ generoso
EB_ARGS="--sanity-check-only --filter-ecs=OptiX-7.2.0.eb,Gurobi-8.1.1.eb,Optax-0.1.7-foss-2022a-CUDA-11.7.0.eb,OR-Tools-7.1-foss-2019a-Python-3.7.2.eb,ParaView-5.11.1-foss-2022b-CUDA-12.2.0.eb"

boegelbot · 2024-09-10T13:20:05Z

@SebastianAchilles: Request for testing this PR well received on login1

PR test command 'EB_PR=21362 EB_ARGS="--sanity-check-only --filter-ecs=OptiX-7.2.0.eb,Gurobi-8.1.1.eb,Optax-0.1.7-foss-2022a-CUDA-11.7.0.eb,OR-Tools-7.1-foss-2019a-Python-3.7.2.eb,ParaView-5.11.1-foss-2022b-CUDA-12.2.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_21362 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 14230

Test results coming soon (I hope)...

- notification for comment with ID 2340745407 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-09-10T13:58:15Z

Test report by @boegelbot
FAILED
Build succeeded for 36 out of 44 (42 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/3691c9de94b5a64eff637e2ff22bcae1 for a full test report.

SebastianAchilles · 2024-09-11T08:27:08Z

@boegelbot please test @ jsc-zen3
EB_ARGS="--sanity-check-only OTF2-3.0.3-GCCcore-12.3.0.eb OTF2-3.0.3-GCCcore-12.2.0.eb OTF2-3.0-GCCcore-11.3.0.eb OTF2-3.0.2-GCCcore-11.3.0.eb OTF2-3.0.3-GCCcore-13.2.0.eb longestrunsubsequence-1.0.1-GCCcore-10.3.0.eb LIBSVM-Python-3.30-foss-2022a.eb PICI-LIGGGHTS-3.8.1-foss-2022a.eb OTF2-3.0.2-GCCcore-11.2.0.eb scikit-bio-0.6.0-foss-2023a.eb cfgrib-0.9.14.0-foss-2023a.eb librosa-0.10.1-foss-2023a.eb ParaView-5.9.1-foss-2021a-mpi.eb ParaView-5.10.1-foss-2022a-mpi.eb ParaView-5.11.1-foss-2022a-mpi.eb ParaView-5.11.2-foss-2023a.eb CLANS-2.0.8-foss-2023a.eb pyMBE-0.8.0-foss-2023b.eb Gubbins-3.3.5-foss-2022b.eb ParaView-5.11.1-foss-2022b.eb ParaView-5.12.0-foss-2023b-Qt5.eb ParaView-5.12.0-foss-2023b.eb ParaView-5.9.1-foss-2021b-mpi.eb scCODA-0.1.9-foss-2023a.eb"

boegelbot · 2024-09-11T08:30:09Z

@SebastianAchilles: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=21362 EB_ARGS="--sanity-check-only OTF2-3.0.3-GCCcore-12.3.0.eb OTF2-3.0.3-GCCcore-12.2.0.eb OTF2-3.0-GCCcore-11.3.0.eb OTF2-3.0.2-GCCcore-11.3.0.eb OTF2-3.0.3-GCCcore-13.2.0.eb longestrunsubsequence-1.0.1-GCCcore-10.3.0.eb LIBSVM-Python-3.30-foss-2022a.eb PICI-LIGGGHTS-3.8.1-foss-2022a.eb OTF2-3.0.2-GCCcore-11.2.0.eb scikit-bio-0.6.0-foss-2023a.eb cfgrib-0.9.14.0-foss-2023a.eb librosa-0.10.1-foss-2023a.eb ParaView-5.9.1-foss-2021a-mpi.eb ParaView-5.10.1-foss-2022a-mpi.eb ParaView-5.11.1-foss-2022a-mpi.eb ParaView-5.11.2-foss-2023a.eb CLANS-2.0.8-foss-2023a.eb pyMBE-0.8.0-foss-2023b.eb Gubbins-3.3.5-foss-2022b.eb ParaView-5.11.1-foss-2022b.eb ParaView-5.12.0-foss-2023b-Qt5.eb ParaView-5.12.0-foss-2023b.eb ParaView-5.9.1-foss-2021b-mpi.eb scCODA-0.1.9-foss-2023a.eb" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_21362 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 4845

Test results coming soon (I hope)...

- notification for comment with ID 2342989130 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-09-11T08:48:01Z

Test report by @boegelbot
FAILED
Build succeeded for 23 out of 24 (24 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/20bc638a19f7288e6e2d332a5cdc1c8d for a full test report.

boegel · 2024-09-11T17:27:26Z

@boegelbot please test @ generoso
EB_ARGS="LIBSVM-Python-3.30-foss-2022a.eb"

boegel · 2024-09-11T17:29:07Z

@Flamefire We should also add a check in the easyconfigs test suite to avoid that python -c without -s gets introduced again, or we can keep playing whack-a-mole for this going forward...

Are you up for looking into that in this PR?

boegelbot · 2024-09-11T17:30:08Z

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=21362 EB_ARGS="LIBSVM-Python-3.30-foss-2022a.eb" EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_21362 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

exit code: 0
output:

Submitted batch job 14246

Test results coming soon (I hope)...

- notification for comment with ID 2344275155 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2024-09-11T17:31:44Z

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
cns3 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/6614f3253fdeef9380ed3e8eda832196 for a full test report.

Running a plain `python -c ...` or `python -m ...` in ECs not using `PythonPackage` might pick up packages installed in `$HOME/.local/lib/python*`. As we haven't set `$PYTHONNOUSERSITE` in those ECs/EasyBlocks we need to pass `-s` to `python` to the same effect. This avoids unexpected failures or success caused by that user environment.

Flamefire · 2024-09-12T08:18:19Z

@Flamefire We should also add a check in the easyconfigs test suite to avoid that python -c without -s gets introduced again, or we can keep playing whack-a-mole for this going forward...

Are you up for looking into that in this PR?

Done, although it is not trivial.

Flamefire · 2024-09-23T19:54:43Z

Test report by @Flamefire
FAILED
Build succeeded for 39 out of 43 (43 easyconfigs in total)
i7010 - Linux Rocky Linux 8.9 (Green Obsidian), x86_64, AMD EPYC 7702 64-Core Processor (zen2), Python 3.8.17
See https://gist.github.com/Flamefire/41b812d34923f1cc119ad10977090782 for a full test report.

Failures:

HPDBSCAN: Issue with numpy includes: add patch to fix compilation of HPDBSCAN 20210826 #21467
OR-Tools: Missing license for dependency
Paraview-intel: Likely some intel compiler issue. Still looking for the actual error message in make

boegel · 2024-09-25T15:52:28Z

test/easyconfigs/easyconfigs.py

+                # Make sure the user packages in $HOME/.local/lib/python*/ are ignored when running python commands
+                # For the EasyBlocks above this is handled automatically by setting $PYTHONNOUSERSITE
+                # Detect any code or module invocation (-m or -c), `python cc` and `python <filepath>`
+                python_re = re.compile(r'\bpython (-c|-m|cc|[^ ]*\w+.py) ')


Suggested in EasyBuild conf call:

just define $PYTHONNOUSERSITE globally in framework;

or introduce a template to run python -c -s commands

And the other 2 variables too? Do we have anything like that already, if so where?

Enforcing that template might be tricky. And we can only have it python -s as we need -m, -c, cc and maybe others which makes using the template more typing.

What other two variables do you mean exactly?

We have similar stuff, though not for Python yet, see for example setting of $CUDA_CACHE_DISABLE in easyblock.py

There is also $PIP_DISABLE_PIP_VERSION_CHECK and $XDG_CACHE_HOME and later $PIP_REQUIRE_VIRTUALENV, see https://github.com/easybuilders/easybuild-easyblocks/pull/3460/files#diff-254aa63061aafed567ef5cc25be7b809dc9a1e4e41b5de85c2e607ea95cf33cbR123

$CUDA_CACHE_DISABLE is set in the prepare_step which IIRC isn't run for sanity-check-only. I think we can set $PYTHONNOUSERSITE and $PIP_DISABLE_PIP_VERSION_CHECK without issues but I'm not sure where. $PIP_REQUIRE_VIRTUALENV would need to be set globally and after loading modules (including fake modules)

Flamefire · 2024-10-01T02:34:25Z

Test report by @Flamefire
SUCCESS / FAILED
Build succeeded for 42 out of 43 (43 easyconfigs in total)
i7123 - Linux Rocky Linux 8.9 (Green Obsidian), x86_64, AMD EPYC 7702 64-Core Processor (zen2), Python 3.8.17
See https://gist.github.com/Flamefire/87e95cc3897f06afb4eaf6b6487e30a5 for a full test report.

OR-Tools fails due to license issues, so OK

Flamefire · 2025-05-08T07:10:50Z

@boegel @SebastianAchilles Can this be merged? Just found another instance of this issue being introduced by an updated EC

akesandgren

LGTM

akesandgren · 2025-10-15T14:10:45Z

Going in, thanks @Flamefire!

Flamefire mentioned this pull request Sep 10, 2024

use proper Python dependency for OTF2 #21325

Merged

SebastianAchilles added this to the release after 4.9.2 milestone Sep 10, 2024

SebastianAchilles added the enhancement label Sep 10, 2024

boegel modified the milestones: 4.9.3, release after 4.9.3 Sep 11, 2024

Flamefire added 2 commits September 12, 2024 09:58

Add test for presence of python without -s

238f3de

Flamefire force-pushed the python-sanity-no-user branch from d8aa881 to 238f3de Compare September 12, 2024 08:17

Flamefire and others added 4 commits September 12, 2024 12:47

Fix python cmd in PICI-LIGGGHTS-3.8.1-foss-2022a.eb

fdffac1

Include line number in error

703e791

Merge branch 'easybuilders:develop' into python-sanity-no-user

35898b5

Merge branch 'develop' into python-sanity-no-user

8fab322

Flamefire force-pushed the python-sanity-no-user branch from 506d07a to 8fab322 Compare September 23, 2024 08:23

boegel reviewed Sep 25, 2024

View reviewed changes

boegel modified the milestones: release after 4.9.4, release after 5.0.0 Mar 18, 2025

Flamefire added 2 commits March 20, 2025 16:50

Merge branch 'develop' into python-sanity-no-user

9e9c737

Merge branch 'develop' into python-sanity-no-user

d564d17

github-actions bot added the change label Mar 20, 2025

boegel modified the milestones: next release (5.1.0), 5.x May 23, 2025

Merge branch 'develop' into python-sanity-no-user

0974f91

akesandgren approved these changes Oct 15, 2025

View reviewed changes

akesandgren modified the milestones: 5.x, next release (5.2.0?) Oct 15, 2025

akesandgren merged commit 6e8ce8d into easybuilders:develop Oct 15, 2025
8 checks passed

Flamefire deleted the python-sanity-no-user branch October 15, 2025 14:12

ignore user-installed packages when running python in sanitycheck #21362

ignore user-installed packages when running python in sanitycheck #21362

Uh oh!

Conversation

Flamefire commented Sep 10, 2024

Uh oh!

SebastianAchilles commented Sep 10, 2024

Uh oh!

SebastianAchilles commented Sep 10, 2024

Uh oh!

boegelbot commented Sep 10, 2024

Uh oh!

boegelbot commented Sep 10, 2024

Uh oh!

SebastianAchilles commented Sep 11, 2024

Uh oh!

boegelbot commented Sep 11, 2024

Uh oh!

boegelbot commented Sep 11, 2024

Uh oh!

boegel commented Sep 11, 2024

Uh oh!

boegel commented Sep 11, 2024

Uh oh!

boegelbot commented Sep 11, 2024

Uh oh!

boegelbot commented Sep 11, 2024

Uh oh!

Flamefire commented Sep 12, 2024

Uh oh!

Flamefire commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

Flamefire Sep 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

boegel Oct 8, 2024

Choose a reason for hiding this comment

Uh oh!

Flamefire Oct 9, 2024

Choose a reason for hiding this comment

Uh oh!

Flamefire commented Oct 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Flamefire commented May 8, 2025

Uh oh!

akesandgren left a comment

Choose a reason for hiding this comment

Uh oh!

akesandgren commented Oct 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Flamefire commented Sep 23, 2024 •

edited

Loading

Flamefire Sep 25, 2024 •

edited

Loading

Flamefire commented Oct 1, 2024 •

edited

Loading