Skip to content

Commit e733cf4

Browse files
RovanionRovanion
authored
Merge pull request #1 from lexming/PR4248
Update for PR 4248
2 parents 4f8ddb8 + 8c76561 commit e733cf4

File tree

99 files changed

+3847
-1217
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

99 files changed

+3847
-1217
lines changed

‎.github/workflows/container_tests.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ jobs:
7474
ls dist
7575
export PREFIX=/tmp/$USER/$GITHUB_SHA
7676
pip install --prefix $PREFIX dist/easybuild-framework*tar.gz
77-
pip install --prefix $PREFIX https://github.com/easybuilders/easybuild-easyblocks/archive/develop.tar.gz
77+
pip install --prefix $PREFIX https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
7878
7979
- name: run test
8080
run: |
@@ -95,7 +95,7 @@ jobs:
9595
echo '%_dbpath %{_var}/lib/rpm' >> $HOME/.rpmmacros
9696
# build CentOS 7 container image for bzip2 1.0.8 using EasyBuild;
9797
# see https://docs.easybuild.io/en/latest/Containers.html
98-
curl -OL https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/develop/easybuild/easyconfigs/b/bzip2/bzip2-1.0.8.eb
98+
curl -OL https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/5.0.x/easybuild/easyconfigs/b/bzip2/bzip2-1.0.8.eb
9999
export EASYBUILD_CONTAINERPATH=$PWD
100100
export EASYBUILD_CONTAINER_CONFIG='bootstrap=docker,from=ghcr.io/easybuilders/centos-7.9-python3-amd64'
101101
eb bzip2-1.0.8.eb --containerize --experimental --container-build-image

‎.github/workflows/container_tests_apptainer.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ jobs:
6666
ls dist
6767
export PREFIX=/tmp/$USER/$GITHUB_SHA
6868
pip install --prefix $PREFIX dist/easybuild-framework*tar.gz
69-
pip install --prefix $PREFIX https://github.com/easybuilders/easybuild-easyblocks/archive/develop.tar.gz
69+
pip install --prefix $PREFIX https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
7070
7171
- name: run test
7272
run: |
@@ -87,7 +87,7 @@ jobs:
8787
echo '%_dbpath %{_var}/lib/rpm' >> $HOME/.rpmmacros
8888
# build CentOS 7 container image for bzip2 1.0.8 using EasyBuild;
8989
# see https://docs.easybuild.io/en/latest/Containers.html
90-
curl -OL https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/develop/easybuild/easyconfigs/b/bzip2/bzip2-1.0.8.eb
90+
curl -OL https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/5.0.x/easybuild/easyconfigs/b/bzip2/bzip2-1.0.8.eb
9191
export EASYBUILD_CONTAINERPATH=$PWD
9292
export EASYBUILD_CONTAINER_CONFIG='bootstrap=docker,from=ghcr.io/easybuilders/centos-7.9-python3-amd64'
9393
export EASYBUILD_CONTAINER_TYPE='apptainer'

‎.github/workflows/end2end.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
name: End-to-end test of EasyBuild in different distros
2+
on: [push, pull_request]
3+
jobs:
4+
build_publish:
5+
name: End-to-end test
6+
runs-on: ubuntu-latest
7+
strategy:
8+
matrix:
9+
container:
10+
- centos-8.5
11+
- fedora-36
12+
- opensuse-15.4
13+
- rockylinux-8.8
14+
- rockylinux-9.2
15+
- ubuntu-20.04
16+
- ubuntu-22.04
17+
fail-fast: false
18+
container:
19+
image: ghcr.io/easybuilders/${{ matrix.container }}-amd64
20+
steps:
21+
- name: Check out the repo
22+
uses: actions/checkout@v3
23+
24+
- name: download and unpack easyblocks and easyconfigs repositories
25+
run: |
26+
cd $HOME
27+
for pkg in easyblocks easyconfigs; do
28+
curl -OL https://github.com/easybuilders/easybuild-${pkg}/archive/5.0.x.tar.gz
29+
tar xfz 5.0.x.tar.gz
30+
rm -f 5.0.x.tar.gz
31+
done
32+
33+
- name: Set up environment
34+
shell: bash
35+
run: |
36+
# collect environment variables to be set in subsequent steps in script that can be sourced
37+
echo "export PATH=$PWD:$PATH" > /tmp/eb_env
38+
echo "export PYTHONPATH=$PWD:$HOME/easybuild-easyblocks-5.0.x:$HOME/easybuild-easyconfigs-5.0.x" >> /tmp/eb_env
39+
40+
- name: Run commands to check test environment
41+
shell: bash
42+
run: |
43+
cmds=(
44+
"whoami"
45+
"pwd"
46+
"env | sort"
47+
"eb --version"
48+
"eb --show-system-info"
49+
"eb --check-eb-deps"
50+
"eb --show-config"
51+
)
52+
for cmd in "${cmds[@]}"; do
53+
echo ">>> $cmd"
54+
sudo -u easybuild bash -l -c "source /tmp/eb_env; $cmd"
55+
done
56+
57+
- name: End-to-end test of installing bzip2 with EasyBuild
58+
shell: bash
59+
run: |
60+
sudo -u easybuild bash -l -c "source /tmp/eb_env; eb bzip2-1.0.8.eb --trace --robot"

‎.github/workflows/unit_tests.yml‎

Lines changed: 33 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,7 @@ jobs:
1313
setup:
1414
runs-on: ubuntu-20.04
1515
outputs:
16-
lmod7: Lmod-7.8.22
1716
lmod8: Lmod-8.7.6
18-
modulesTcl: modules-tcl-1.147
19-
modules3: modules-3.2.10
2017
modules4: modules-4.1.4
2118
steps:
2219
- run: "true"
@@ -29,49 +26,25 @@ jobs:
2926
modules_tool:
3027
# use variables defined by 'setup' job above, see also
3128
# https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#needs-context
32-
- ${{needs.setup.outputs.lmod7}}
3329
- ${{needs.setup.outputs.lmod8}}
34-
- ${{needs.setup.outputs.modulesTcl}}
35-
- ${{needs.setup.outputs.modules3}}
3630
- ${{needs.setup.outputs.modules4}}
37-
module_syntax: [Lua, Tcl]
3831
lc_all: [""]
39-
# don't test with Lua module syntax (only supported in Lmod)
40-
exclude:
41-
- modules_tool: ${{needs.setup.outputs.modulesTcl}}
42-
module_syntax: Lua
43-
- modules_tool: ${{needs.setup.outputs.modules3}}
44-
module_syntax: Lua
45-
- modules_tool: ${{needs.setup.outputs.modules4}}
46-
module_syntax: Lua
4732
include:
4833
# Test different Python 3 versions with Lmod 8.x (with both Lua and Tcl module syntax)
4934
- python: 3.7
5035
modules_tool: ${{needs.setup.outputs.lmod8}}
51-
module_syntax: Lua
5236
- python: 3.8
5337
modules_tool: ${{needs.setup.outputs.lmod8}}
54-
module_syntax: Lua
55-
- python: 3.8
56-
modules_tool: ${{needs.setup.outputs.lmod8}}
57-
module_syntax: Tcl
5838
- python: 3.9
5939
modules_tool: ${{needs.setup.outputs.lmod8}}
60-
module_syntax: Lua
6140
- python: '3.10'
6241
modules_tool: ${{needs.setup.outputs.lmod8}}
63-
module_syntax: Lua
64-
- python: '3.11'
65-
modules_tool: ${{needs.setup.outputs.lmod8}}
66-
module_syntax: Lua
6742
- python: '3.11'
6843
modules_tool: ${{needs.setup.outputs.lmod8}}
69-
module_syntax: Tcl
7044
# There may be encoding errors in Python 3 which are hidden when an UTF-8 encoding is set
7145
# Hence run the tests (again) with LC_ALL=C and Python 3.6 (or any < 3.7)
7246
- python: 3.6
7347
modules_tool: ${{needs.setup.outputs.lmod8}}
74-
module_syntax: Lua
7548
lc_all: C
7649
fail-fast: false
7750
steps:
@@ -122,16 +95,11 @@ jobs:
12295
# and are only run after the PR gets merged
12396
GITHUB_TOKEN: ${{secrets.CI_UNIT_TESTS_GITHUB_TOKEN}}
12497
run: |
125-
# don't install GitHub token when testing with Lmod 7.x or non-Lmod module tools,
126-
# and only when testing with Lua as module syntax,
127-
# to avoid hitting GitHub rate limit;
98+
# only install GitHub token when testing with Lmod 8.x + Python 3.6 or 3.9, to avoid hitting GitHub rate limit;
12899
# tests that require a GitHub token are skipped automatically when no GitHub token is available
129-
if [[ ! "${{matrix.modules_tool}}" =~ 'Lmod-7' ]] && [[ ! "${{matrix.modules_tool}}" =~ 'modules-' ]] && [[ "${{matrix.module_syntax}}" == 'Lua' ]]; then
100+
if [[ "${{matrix.modules_tool}}" =~ 'Lmod-8' ]] && [[ "${{matrix.python}}" =~ 3.[69] ]]; then
130101
if [ ! -z $GITHUB_TOKEN ]; then
131-
if [ "x${{matrix.python}}" == 'x2.6' ];
132-
then SET_KEYRING="keyring.set_keyring(keyring.backends.file.PlaintextKeyring())";
133-
else SET_KEYRING="import keyrings.alt.file; keyring.set_keyring(keyrings.alt.file.PlaintextKeyring())";
134-
fi;
102+
SET_KEYRING="import keyrings.alt.file; keyring.set_keyring(keyrings.alt.file.PlaintextKeyring())";
135103
python -c "import keyring; $SET_KEYRING; keyring.set_password('github_token', 'easybuild_test', '$GITHUB_TOKEN')";
136104
fi
137105
echo "GitHub token installed!"
@@ -169,8 +137,6 @@ jobs:
169137
- name: run test suite
170138
env:
171139
EB_VERBOSE: 1
172-
EASYBUILD_MODULE_SYNTAX: ${{matrix.module_syntax}}
173-
TEST_EASYBUILD_MODULE_SYNTAX: ${{matrix.module_syntax}}
174140
LC_ALL: ${{matrix.lc_all}}
175141
run: |
176142
# run tests *outside* of checked out easybuild-framework directory,
@@ -186,28 +152,37 @@ jobs:
186152
export PYTHONPATH=$PREFIX/lib/python${{matrix.python}}/site-packages:$PYTHONPATH
187153
eb --version
188154
# tell EasyBuild which modules tool is available
189-
if [[ ${{matrix.modules_tool}} =~ ^modules-tcl- ]]; then
190-
export EASYBUILD_MODULES_TOOL=EnvironmentModulesTcl
191-
elif [[ ${{matrix.modules_tool}} =~ ^modules-3 ]]; then
192-
export EASYBUILD_MODULES_TOOL=EnvironmentModulesC
193-
elif [[ ${{matrix.modules_tool}} =~ ^modules-4 ]]; then
155+
if [[ ${{matrix.modules_tool}} =~ ^modules-4 ]]; then
194156
export EASYBUILD_MODULES_TOOL=EnvironmentModules
195157
else
196158
export EASYBUILD_MODULES_TOOL=Lmod
197159
fi
198-
export TEST_EASYBUILD_MODULES_TOOL=$EASYBUILD_MODULES_TOOL
199-
eb --show-config
200-
# gather some useful info on test system
201-
eb --show-system-info
202-
# check GitHub configuration
203-
eb --check-github --github-user=easybuild_test
204-
# create file owned by root but writable by anyone (used by test_copy_file)
205-
sudo touch /tmp/file_to_overwrite_for_easybuild_test_copy_file.txt
206-
sudo chmod o+w /tmp/file_to_overwrite_for_easybuild_test_copy_file.txt
207-
# run test suite
208-
python -O -m test.framework.suite 2>&1 | tee test_framework_suite.log
209-
# try and make sure output of running tests is clean (no printed messages/warnings)
210-
IGNORE_PATTERNS="no GitHub token available|skipping SvnRepository test|requires Lmod as modules tool|stty: 'standard input': Inappropriate ioctl for device|CryptographyDeprecationWarning: Python 3.[56]|from cryptography.* import |CryptographyDeprecationWarning: Python 2|Blowfish|GC3Pie not available, skipping test"
211-
# '|| true' is needed to avoid that Travis stops the job on non-zero exit of grep (i.e. when there are no matches)
212-
PRINTED_MSG=$(egrep -v "${IGNORE_PATTERNS}" test_framework_suite.log | grep '\.\n*[A-Za-z]' || true)
213-
test "x$PRINTED_MSG" = "x" || (echo "ERROR: Found printed messages in output of test suite" && echo "${PRINTED_MSG}" && exit 1)
160+
export TEST_EASYBUILD_MODULES_TOOL=${EASYBUILD_MODULES_TOOL}
161+
162+
# Run tests with LUA and Tcl module syntax (where supported)
163+
for module_syntax in Lua Tcl; do
164+
# Only Lmod supports Lua
165+
if [[ "${module_syntax}" == "Lua" ]] && [[ "${EASYBUILD_MODULES_TOOL}" != "Lmod" ]]; then
166+
echo "Not testing with '${module_syntax}' as module syntax with '${EASYBUILD_MODULES_TOOL}' as modules tool"
167+
continue
168+
fi
169+
printf '\n\n=====================> Using $module_syntax module syntax <=====================\n\n'
170+
export EASYBUILD_MODULE_SYNTAX="${module_syntax}"
171+
export TEST_EASYBUILD_MODULE_SYNTAX="${EASYBUILD_MODULE_SYNTAX}"
172+
173+
eb --show-config
174+
# gather some useful info on test system
175+
eb --show-system-info
176+
# check GitHub configuration
177+
eb --check-github --github-user=easybuild_test
178+
# create file owned by root but writable by anyone (used by test_copy_file)
179+
sudo touch /tmp/file_to_overwrite_for_easybuild_test_copy_file.txt
180+
sudo chmod o+w /tmp/file_to_overwrite_for_easybuild_test_copy_file.txt
181+
# run test suite
182+
python -O -m test.framework.suite 2>&1 | tee test_framework_suite.log
183+
# try and make sure output of running tests is clean (no printed messages/warnings)
184+
IGNORE_PATTERNS="no GitHub token available|skipping SvnRepository test|requires Lmod as modules tool|stty: 'standard input': Inappropriate ioctl for device|CryptographyDeprecationWarning: Python 3.[56]|from cryptography.* import |CryptographyDeprecationWarning: Python 2|Blowfish|GC3Pie not available, skipping test"
185+
# '|| true' is needed to avoid that GitHub Actions stops the job on non-zero exit of grep (i.e. when there are no matches)
186+
PRINTED_MSG=$(egrep -v "${IGNORE_PATTERNS}" test_framework_suite.log | grep '\.\n*[A-Za-z]' || true)
187+
test "x$PRINTED_MSG" = "x" || (echo "ERROR: Found printed messages in output of test suite" && echo "${PRINTED_MSG}" && exit 1)
188+
done

‎RELEASE_NOTES‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,81 @@ For more detailed information, please see the git log.
44
These release notes can also be consulted at https://easybuild.readthedocs.io/en/latest/Release_notes.html.
55

66

7+
v4.9.0 (30 December 2023)
8+
-------------------------
9+
10+
feature release
11+
12+
- various enhancements, including:
13+
- allow tweaking of easyconfigs from different toolchains (#3669)
14+
- add support for `--list-software --output-format=json` (#4152)
15+
- add `--module-cache-suffix` configuration setting to allow multiple (Lmod) caches (#4403)
16+
- various bug fixes, including:
17+
- deduplicate warnings & errors found in logs and add initial newline + tab in output (#4361)
18+
- fix support for Environment Modules as modules tool to pass unit tests with v4.2+ (#4369)
19+
- adapt module function check for Environment Modules v4+ (#4371)
20+
- only install GitHub token when testing with Lmod 8.x + Python 3.6 or 3.9 (#4375)
21+
- use `-qopenmp` instead of `-fiopenmp` for OpenMP in Intel compilers (#4377)
22+
- fix `LIBBLAS_MT` for FlexiBLAS, ensure `-lpthread` is included (#4379)
23+
- relax major version match regex in `find_related_easyconfigs` using for `--review-pr` (#4385)
24+
- eliminate duplicate multideps from generated module files (#4386)
25+
- resolve templated values in extension names in `_make_extension_list` (#4392)
26+
- use source toolchain version when passing only `--try-toolchain` (#4395)
27+
- fix writing spider cache for Lmod >= 8.7.12 (#4402)
28+
- fix `--inject-checksums` when extension specifies patch file in tuple format (#4405)
29+
- fix `LooseVersion` when running with Python 2.7 (#4408)
30+
- use more recent easyblocks PR in `test_github_merge_pr` (#4414)
31+
- other changes:
32+
- extend test that checks build environment to recent `foss/2023a` toolchain (#4391)
33+
34+
35+
v4.8.2 (29 October 2023)
36+
------------------------
37+
38+
update/bugfix release
39+
40+
- various enhancements, including:
41+
- add support for `%(sysroot)s` template value (#4359)
42+
- add `dependency_names` method to `EasyConfig` class to get set of names of (direct) dependencies (#4360)
43+
- various bug fixes, including:
44+
- add CI workflow to run unit tests with Python 2 (again) (#4333)
45+
- fix typo in help message for `--silence-hook-trigger` (#4343)
46+
- include major version (`*majver`) templates in auto-generated documentation (#4347)
47+
- reset `tempfile.tempdir` to `None` to avoid that tmpdir path gets progressively deeper with each easystack item (#4350)
48+
- fix `findPythonDeps.py` script when called with an (absolute or relative) path to an easyconfig instead of a filename (#4365)
49+
- fix broken test for `reasons_for_closing`, which fails because commit status of easyconfigs PR is no longer available (#4366)
50+
- other changes:
51+
- reduce number of CI jobs by testing for Lua and Tcl module syntax in a single CI job (#4192)
52+
53+
54+
v4.8.1 (11 September 2023)
55+
--------------------------
56+
57+
update/bugfix release
58+
59+
- various enhancements, including:
60+
- add end-to-end test for running EasyBuild in different Linux distros using containers (#3968)
61+
- suggest default title in `--review-pr` (#4287)
62+
- add `build_and_install_loop` hooks to run before and after the install loop for individual easyconfigs (#4304)
63+
- implement support for `cancel_hook` and `fail_hook` (#4315, #4325)
64+
- add postiter hook to the list of steps so the corresponding hook can be used (#4316)
65+
- add `run_shell_cmd` hook (#4323)
66+
- add `build_info_msg` easyconfig parameter to print message during installation of an easyconfig (#4324)
67+
- add `--silence-hook-trigger` configuration option to supress printing of debug message every time a hook is triggered (#4329)
68+
- add support for using fine grained Github tokens (#4332)
69+
- add definitions for ifbf and iofbf toolchain (#4337)
70+
- add support for submodule filtering and specifying extra Git configuration in `git_config` (#4338, #4339)
71+
- various bug fixes, including:
72+
- improve error when checksum dict has no entry for a file (#4150)
73+
- avoid error being logged when `checksums.json` is not found (#4261)
74+
- don't fail in `mkdir` if path gets created while processing it (#4300, #4328)
75+
- ignore request for external module (meta)data when no modules tool is active (#4308)
76+
- use sys.executable to obtain path to `python` command in tests, rather than assuming that `python` command is available in `$PATH` (#4309)
77+
- fix `test_add_and_remove_module_path` by replacing string comparison of paths by checking whether they point to the same path (since symlinks may cause trouble) (#4312)
78+
- enhance `Toolchain.get_flag` to handle lists (#4319)
79+
- only add extensions in module file if there are extensions (#4331)
80+
81+
782
v4.8.0 (7 July 2023)
883
--------------------
984

‎easybuild/base/fancylogger.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -285,7 +285,7 @@ def makeRecord(self, name, level, pathname, lineno, msg, args, excinfo, func=Non
285285
overwrite make record to use a fancy record (with more options)
286286
"""
287287
logrecordcls = logging.LogRecord
288-
if hasattr(self, 'fancyrecord') and self.fancyrecord:
288+
if getattr(self, 'fancyrecord', None):
289289
logrecordcls = FancyLogRecord
290290
try:
291291
new_msg = str(msg)

0 commit comments

Comments
 (0)