Skip to content

Avoid failure in Python package installation and sanity check when $PIP_REQUIRE_VIRTUALENV is set #3460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
May 23, 2025
Merged

Avoid failure in Python package installation and sanity check when $PIP_REQUIRE_VIRTUALENV is set #3460

merged 2 commits into from
May 23, 2025

Conversation

@Flamefire
Copy link
Contributor

@Flamefire Flamefire commented Sep 24, 2024

We already have it for the installation if set by users but if a module sets it (e.g Python) the sanity check will fail

@Flamefire Flamefire force-pushed the vrtenv branch 2 times, most recently from 8b6f66a to 03c35b4 Compare September 24, 2024 14:52
@boegel boegel added the bug fix label Sep 25, 2024
@boegel boegel added this to the 4.x milestone Sep 25, 2024
boegel
boegel requested changes Sep 25, 2024
View reviewed changes
easybuild/easyblocks/generic/pythonbundle.py Outdated
# because the environment is reset to the initial environment right before loading the module
env.setvar('PYTHONNOUSERSITE', '1', verbose=False)
# Set (again) in case the module changes it
env.setvar('PIP_REQUIRE_VIRTUALENV', 'false', verbose=False)
Copy link
Member

@boegel boegel Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Flamefire Shouldn't we introduce a function that can be called from both pythonbundle and pythonpackage?

Copy link
Contributor Author

@Flamefire Flamefire Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean one that sets PYTHONNOUSERSITE, PIP_REQUIRE_VIRTUALENV and PIP_DISABLE_PIP_VERSION_CHECK?

Given that we already need it in quite some places it makes sense, yes. Maybe named setup_pip_env with a verbose param? I'd make that default to False and only set it to True in PythonPackage.__init__ to reduce log spam. Even there it gets likely duplicated a lot for bundles but completely omitting it feels wrong.

Copy link
Member

@boegel boegel Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good

Copy link
Contributor Author

@Flamefire Flamefire Sep 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, implemented it in the Python EasyBlock file to avoid cyclic includes of that and PythonPackage. Those variables are also required/useful for TensorFlow & friends, hence I made them a constant.

To avoid the log spam the actual set is only executed if any of the variables need changing.

@Flamefire Flamefire force-pushed the vrtenv branch 2 times, most recently from 66b3699 to 7c6bfc9 Compare September 26, 2024 14:31
@boegel boegel changed the title Avoid failure in Python sanity check when PIP_REQUIRE_VIRTUALENV is set Avoid failure in Python sanity check when $PIP_REQUIRE_VIRTUALENV is set Oct 9, 2024
@boegel
Copy link
Member

boegel commented Oct 9, 2024

@boegelbot please test @ jsc-zen3-a100
EB_ARGS="tensorflow-compression-2.11.0-foss-2022a-CUDA-11.7.0.eb TensorFlow-2.15.1-foss-2023a.eb --installpath /tmp/$USER/pr3460"

@boegelbot
Copy link

boegelbot commented Oct 9, 2024

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3460 EB_ARGS="tensorflow-compression-2.11.0-foss-2022a-CUDA-11.7.0.eb TensorFlow-2.15.1-foss-2023a.eb --installpath /tmp/$USER/pr3460" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3460 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 5038

Test results coming soon (I hope)...

- notification for comment with ID 2401832507 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

boegelbot commented Oct 9, 2024

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS tensorflow-compression-2.11.0-foss-2022a-CUDA-11.7.0.eb
  • SUCCESS TensorFlow-2.15.1-foss-2023a.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/4b82c97538296dae8d67d812cc0e9f72 for a full test report.

@Flamefire Flamefire changed the title Avoid failure in Python sanity check when $PIP_REQUIRE_VIRTUALENV is set Avoid failure in Python package installation and sanity check when $PIP_REQUIRE_VIRTUALENV is set Oct 10, 2024
@Flamefire
Copy link
Contributor Author

Flamefire commented Feb 18, 2025

@boegel ping on that. Just ran into this failure while testing EB 5 ECs.

@Flamefire
Copy link
Contributor Author

Flamefire commented Mar 21, 2025

Rebased for 5.0x

@Flamefire
Copy link
Contributor Author

Flamefire commented Apr 10, 2025

Can this be included in 5.0.1? I had to patch our 5.0.0 installation again after the update. And test reports looked fine too so I don't expect any downsides.

@boegel boegel modified the milestones: 5.x, 5.0.1 Apr 23, 2025
@boegel
Copy link
Member

boegel commented Apr 23, 2025

Can this be included in 5.0.1? I had to patch our 5.0.0 installation again after the update. And test reports looked fine too so I don't expect any downsides.

I'll see what I can do to get this merged ASAP

@boegel
Copy link
Member

boegel commented May 23, 2025

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS Python-2.7.18-GCCcore-12.3.0.eb
  • SUCCESS Python-3.12.3-GCCcore-13.3.0.eb
  • SUCCESS matplotlib-3.9.2-gfbf-2024a.eb
  • SUCCESS numexpr-2.9.0-foss-2023a.eb
  • SUCCESS TensorFlow-2.13.0-foss-2023a.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total)
node3502.doduo.os - Linux RHEL 9.4, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.9.18
See https://gist.github.com/boegel/63fade637b46537fb613f90197c33773 for a full test report.

@Flamefire
Copy link
Contributor Author

Flamefire commented May 23, 2025

Rebased without changes to resolve merge conflict

boegel
boegel approved these changes May 23, 2025
View reviewed changes
Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel boegel merged commit 68bf2fd into easybuilders:develop May 23, 2025
17 checks passed
@Flamefire Flamefire deleted the vrtenv branch May 24, 2025 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@boegel boegel boegel approved these changes

Assignees

No one assigned

Labels

bug fix

Projects

None yet

Milestone

5.1.0

Development

Successfully merging this pull request may close these issues.

3 participants

@Flamefire @boegel @boegelbot