Skip to content

Conversation

@Thyre
Copy link
Collaborator

@Thyre Thyre commented Jul 22, 2025

By using _cmakeopts, the argument was only passed for the first stage of the build, and ignored for subsequent stages. We therefore build all available architectures for AMDGPU / NVPTX when not necessary. Fix this by moving the argument to general_opts.

By using `_cmakeopts`, the argument was only passed for the first stage
of the build, and ignored for subsequent stages. We therefore build all
available architectures for AMDGPU / NVPTX when not necessary. Fix this
by moving the argument to `general_opts`.

Signed-off-by: Jan André Reuter <[email protected]>
@Thyre Thyre changed the title LLVM: Fix LIBOMPTARGET_DEVICE_ARCHITECTURES LLVM: Fix missing LIBOMPTARGET_DEVICE_ARCHITECTURES in third stage of bootstrap build Jul 22, 2025
@Thyre Thyre changed the title LLVM: Fix missing LIBOMPTARGET_DEVICE_ARCHITECTURES in third stage of bootstrap build LLVM: Fix missing LIBOMPTARGET_DEVICE_ARCHITECTURES in third stage of bootstrap build Jul 22, 2025
@Thyre
Copy link
Collaborator Author

Thyre commented Jul 22, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="LLVM-19.1.7-GCCcore-13.3.0.eb LLVM-20.1.5-GCCcore-13.3.0.eb"
CORE_CNT=16

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3851 EB_ARGS="LLVM-19.1.7-GCCcore-13.3.0.eb LLVM-20.1.5-GCCcore-13.3.0.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3851 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7303

Test results coming soon (I hope)...

- notification for comment with ID 3102298599 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

Copy link
Contributor

@Flamefire Flamefire left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Maybe add documentation to those 2 members because it is hard to tell what the difference is which led to this issue

@Thyre
Copy link
Collaborator Author

Thyre commented Jul 22, 2025

Good catch. Maybe add documentation to those 2 members because it is hard to tell what the difference is which led to this issue

Good idea. I'll try to find a place where we can document this.
Just noticed that while testing #3824, because I saw all the amdgpu options but not gfx1101.

@Flamefire
Copy link
Contributor

Yeah, I introduced that mistake in bb67396#diff-9560297abd14bd1649f4ea97ffc545517160060a13db3b3194ac97a3c2541049R506-R792 due to this confusion

Not sure why my tests didn't show this, i.e. why I've only seen the expected arch files.

@Thyre
Copy link
Collaborator Author

Thyre commented Jul 22, 2025

It fortunately isn't a really noticeable regression. We just build a few architectures too much.
It's really easy to miss some things when dealing with such a huge EasyBlock.

I wonder though if it would be worth adding tests / sanity checks to avoid this from happening again (in a separate PR...).
Might be complicated though, as we've seen e.g. the NVPTX bitcode library being built even without NVPTX support.

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS LLVM-19.1.7-GCCcore-13.3.0.eb
  • SUCCESS LLVM-20.1.5-GCCcore-13.3.0.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/f964694c3f93ba888aa3a813c0f19017 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jul 23, 2025

-DLIBOMPTARGET_DEVICE_ARCHITECTURES='sm_80' shows up in the logs again:

$ grep -rni "\-DLIBOMPTARGET_DEVICE_ARCHITECTURES"
[...]
easybuild-LLVM-19.1.7-20250722.135204.log:1540:== 2025-07-22 11:34:16,167 run.py:502 INFO Running shell command 'cmake  -DCMAKE_INSTALL_LIBDIR:PATH=lib -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DLLVM_ENABLE_PROJECTS='llvm;clang;lld' -DLLVM_ENABLE_RUNTIMES='compiler-rt;libunwind;libcxx;libcxxabi' -DCMAKE_VERBOSE_MAKEFILE=ON -DLLVM_INCLUDE_BENCHMARKS=OFF -DLLVM_INSTALL_UTILS=ON -DPython3_FIND_VIRTUALENV=STANDARD -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON -DLIBCXX_ENABLE_SHARED=ON -DLIBCXXABI_ENABLE_SHARED=ON -DLIBUNWIND_ENABLE_SHARED=ON -DLLVM_ENABLE_RTTI=ON -DLLVM_REQUIRES_RTTI=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCLANG_DEFAULT_LINKER=lld -DFLANG_DEFAULT_LINKER=lld -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD='X86;NVPTX' -DCMAKE_INSTALL_PREFIX=/project/def-maintainers/boegelbot/rocky9/zen3/software/LLVM/19.1.7-GCCcore-13.3.0 -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_ENABLE_LIBXML2=ON -DLLVM_ENABLE_ZLIB=ON -DLLVM_ENABLE_ZSTD=OFF -DLLDB_ENABLE_SWIG=OFF -DLIBOMP_OMPD_GDB_SUPPORT=OFF -DLLVM_ENABLE_Z3_SOLVER=ON -DLLVM_Z3_INSTALL_DIR=/project/def-maintainers/boegelbot/rocky9/zen3/software/Z3/4.13.0-GCCcore-13.3.0 -DPYTHON_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DPython_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DPython3_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DLIBOMPTARGET_DEVICE_ARCHITECTURES='sm_80' -DRUNTIMES_CMAKE_ARGS='-DCMAKE_C_FLAGS=--gcc-install-dir=/project/def-maintainers/boegelbot/rocky9/zen3/software/GCCcore/13.3.0/lib/gcc/x86_64-pc-linux-gnu/13.3.0;-DCMAKE_CXX_FLAGS=--gcc-install-dir=/project/def-maintainers/boegelbot/rocky9/zen3/software/GCCcore/13.3.0/lib/gcc/x86_64-pc-linux-gnu/13.3.0;-DPYTHON_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python;-DPython_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python;-DPython3_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python' /tmp/boegelbot/LLVM/19.1.7/GCCcore-13.3.0/llvm-project-19.1.7.src/llvm' in /tmp/boegelbot/LLVM/19.1.7/GCCcore-13.3.0/llvm.obj.1
[...]
easybuild-LLVM-19.1.7-20250722.135204.log:26661:== 2025-07-22 11:55:39,680 run.py:502 INFO Running shell command 'cmake -DCMAKE_VERBOSE_MAKEFILE=ON -DLLVM_INCLUDE_BENCHMARKS=OFF -DLLVM_INSTALL_UTILS=ON -DPython3_FIND_VIRTUALENV=STANDARD -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON -DLIBCXX_ENABLE_SHARED=ON -DLIBCXXABI_ENABLE_SHARED=ON -DLIBUNWIND_ENABLE_SHARED=ON -DLLVM_ENABLE_RTTI=ON -DLLVM_REQUIRES_RTTI=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCLANG_DEFAULT_LINKER=lld -DFLANG_DEFAULT_LINKER=lld -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD='X86;NVPTX' -DCMAKE_INSTALL_PREFIX=/project/def-maintainers/boegelbot/rocky9/zen3/software/LLVM/19.1.7-GCCcore-13.3.0 -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_ENABLE_LIBXML2=ON -DLLVM_ENABLE_ZLIB=ON -DLLVM_ENABLE_ZSTD=OFF -DLLDB_ENABLE_SWIG=OFF -DLIBOMP_OMPD_GDB_SUPPORT=OFF -DLLVM_ENABLE_Z3_SOLVER=ON -DLLVM_Z3_INSTALL_DIR=/project/def-maintainers/boegelbot/rocky9/zen3/software/Z3/4.13.0-GCCcore-13.3.0 -DPYTHON_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DPython_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DPython3_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DLIBOMPTARGET_DEVICE_ARCHITECTURES='sm_80' -DRUNTIMES_CMAKE_ARGS='-DCMAKE_C_COMPILER=/tmp/eb-wnuge6w7/tmpukrjpa9e/rpath_wrappers/clang_wrapper/clang;-DCMAKE_C_FLAGS=--gcc-install-dir=/project/def-maintainers/boegelbot/rocky9/zen3/software/GCCcore/13.3.0/lib/gcc/x86_64-pc-linux-gnu/13.3.0;-DCMAKE_CXX_COMPILER=/tmp/eb-wnuge6w7/tmpukrjpa9e/rpath_wrappers/clangxx_wrapper/clang++;-DCMAKE_CXX_FLAGS=--gcc-install-dir=/project/def-maintainers/boegelbot/rocky9/zen3/software/GCCcore/13.3.0/lib/gcc/x86_64-pc-linux-gnu/13.3.0;-DPYTHON_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python;-DPython_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python;-DPython3_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python' -DLLVM_ENABLE_PROJECTS='llvm;clang;lld' -DLLVM_ENABLE_RUNTIMES='compiler-rt;libunwind;libcxx;libcxxabi' -DCMAKE_C_COMPILER=/tmp/eb-wnuge6w7/tmp4c_qge53/rpath_wrappers/clang_wrapper/clang -DCMAKE_CXX_COMPILER=/tmp/eb-wnuge6w7/tmp4c_qge53/rpath_wrappers/clangxx_wrapper/clang++ -DCMAKE_ASM_COMPILER=/tmp/eb-wnuge6w7/tmp4c_qge53/rpath_wrappers/clang_wrapper/clang -DCMAKE_ASM_COMPILER_ID=Clang /tmp/boegelbot/LLVM/19.1.7/GCCcore-13.3.0/llvm-project-19.1.7.src/llvm' in /tmp/boegelbot/LLVM/19.1.7/GCCcore-13.3.0/llvm.obj.2
[...]
easybuild-LLVM-19.1.7-20250722.135204.log:74143:== 2025-07-22 12:14:54,566 run.py:502 INFO Running shell command 'cmake -DCMAKE_VERBOSE_MAKEFILE=ON -DLLVM_INCLUDE_BENCHMARKS=OFF -DLLVM_INSTALL_UTILS=ON -DPython3_FIND_VIRTUALENV=STANDARD -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON -DLIBCXX_ENABLE_SHARED=ON -DLIBCXXABI_ENABLE_SHARED=ON -DLIBUNWIND_ENABLE_SHARED=ON -DLLVM_ENABLE_RTTI=ON -DLLVM_REQUIRES_RTTI=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCLANG_DEFAULT_LINKER=lld -DFLANG_DEFAULT_LINKER=lld -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD='X86;NVPTX' -DCMAKE_INSTALL_PREFIX=/project/def-maintainers/boegelbot/rocky9/zen3/software/LLVM/19.1.7-GCCcore-13.3.0 -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_ENABLE_LIBXML2=ON -DLLVM_ENABLE_ZLIB=ON -DLLVM_ENABLE_ZSTD=OFF -DLLDB_ENABLE_SWIG=OFF -DLIBOMP_OMPD_GDB_SUPPORT=OFF -DLLVM_ENABLE_Z3_SOLVER=ON -DLLVM_Z3_INSTALL_DIR=/project/def-maintainers/boegelbot/rocky9/zen3/software/Z3/4.13.0-GCCcore-13.3.0 -DPYTHON_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DPython_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DPython3_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DLIBOMPTARGET_DEVICE_ARCHITECTURES='sm_80' -DRUNTIMES_CMAKE_ARGS='-DCMAKE_C_COMPILER=/tmp/eb-wnuge6w7/tmpomzrnagi/rpath_wrappers/clang_wrapper/clang;-DCMAKE_C_FLAGS=--gcc-install-dir=/project/def-maintainers/boegelbot/rocky9/zen3/software/GCCcore/13.3.0/lib/gcc/x86_64-pc-linux-gnu/13.3.0;-DCMAKE_CXX_COMPILER=/tmp/eb-wnuge6w7/tmpomzrnagi/rpath_wrappers/clangxx_wrapper/clang++;-DCMAKE_CXX_FLAGS=--gcc-install-dir=/project/def-maintainers/boegelbot/rocky9/zen3/software/GCCcore/13.3.0/lib/gcc/x86_64-pc-linux-gnu/13.3.0;-DPYTHON_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python;-DPython_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python;-DPython3_EXECUTABLE=/project/def-maintainers/boegelbot/rocky9/zen3/software/Python/3.12.3-GCCcore-13.3.0/bin/python' -DLLVM_ENABLE_PROJECTS='llvm;mlir;clang;flang;openmp;polly;clang-tools-extra;lld;lldb;bolt' -DLLVM_ENABLE_RUNTIMES='compiler-rt;libunwind;libcxx;libcxxabi;offload' -DLIBOMPTARGET_PLUGINS_TO_BUILD='host;cuda' -DLIBOMPTARGET_DLOPEN_PLUGINS='cuda' -DOPENMP_ENABLE_LIBOMPTARGET=ON -DLIBOMP_INSTALL_ALIASES=OFF -DLLVM_LIT_ARGS="-j 16 -v --timeout 300" -DLLVM_POLLY_LINK_INTO_TOOLS=ON -DLLVM_INCLUDE_TESTS=ON -DLLVM_BUILD_TESTS=ON -DCMAKE_C_COMPILER=/tmp/eb-wnuge6w7/tmpvo100a5a/rpath_wrappers/clang_wrapper/clang -DCMAKE_CXX_COMPILER=/tmp/eb-wnuge6w7/tmpvo100a5a/rpath_wrappers/clangxx_wrapper/clang++ -DCMAKE_ASM_COMPILER=/tmp/eb-wnuge6w7/tmpvo100a5a/rpath_wrappers/clang_wrapper/clang -DCMAKE_ASM_COMPILER_ID=Clang /tmp/boegelbot/LLVM/19.1.7/GCCcore-13.3.0/llvm-project-19.1.7.src/llvm' in /tmp/boegelbot/LLVM/19.1.7/GCCcore-13.3.0/llvm.obj.3

Copy link
Contributor

@Crivella Crivella left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, tested both here and in easybuilders/easybuild-easyconfigs#23459

@Crivella Crivella added this to the release after 5.1.1 milestone Jul 24, 2025
@Crivella
Copy link
Contributor

Going in, thanks @Thyre!

@Crivella Crivella merged commit dc81bee into easybuilders:develop Jul 24, 2025
17 checks passed
# CMake options passed to each build stage.
# Will be cleared between stages. If arguments are needed in multiple stages,
# consider adding them to general_opts instead.
self._cmakeopts = {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too late for this PR now, but I thought we could set this to None at this point such that adding dict entries to this too early would fail loudly.

What do you think?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably a good idea. We can either add this in a separate PR, or just add it to one of the remaining LLVM ones.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think #3799 would be a good candidate as IMO that is ready.

@Crivella ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are already setting it at the edge where we want to access it. If you try to use it before in the init it will be non-existent and fail regardless

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it must only be used after the _configure_general_build call in configure_step: https://github.com/Thyre/easybuild-easyblocks/blob/9771da4f8969ba373c19b99a6988b3b65113d225/easybuild/easyblocks/l/llvm.py#L833

There cmakeopts should be initialized to a copy of generalopts. Or is there any earlier use intended? Maybe derived easyblocks adding options to that before calling the super-method?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I dont understand your remark:

Only if _cmakeopts was not an attribute there which i still think it should.

Getting an error is intended to detect using it too early. And IMO the explicit None shows that it does not exist yet and provides means to add a comment what it is and where it will be set.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What i meant is that i think _cmakeopts should be already usable in the configure step.

I think unless we decide do alter the logic quite a bit this will not really save us from PRs using the wrong attributes/methods if we do not catch it in review...
What i think is more worth the effort is increase the sanity checks to make sure all the feature we want on are on.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But should it be usable before _configure_general_build in the configure step?

I'd just make it as hard as possible to use the wrong one if it doesn't impact what we really need and my suggestion is a step towards it while not impacting current usage as far as I can tell.

Better sanity checks would be good either way, yes.

Then maybe the minimal common ground would be renaming the generically named cmakeopts to something more specific like "_stage_cmake_opts"? That would make it easier to review without any real change

Copy link
Contributor

@Crivella Crivella Jul 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But should it be usable before _configure_general_build in the configure step?

I would say so, otherwise we would need a section for updating general_opts based on present packages (what we have right now before _configure_general_build in the configure_step and one to update the single stage after.
I feel that having 2 section there could also lead to confusion and people misplacing stuff (eg modifying general_opts after the call to _configure_general_build

Then maybe the minimal common ground would be renaming the generically named cmakeopts to something more specific like "_stage_cmake_opts"? That would make it easier to review without any real change
Not against renaming variables for clarity, might even make it _current_stage_cmake_opts for clarity

I think the only way to really make sure stuff is not misused is to introduce 4 methods:

  • add_stage1_cmake_opts
  • add_stage2_cmake_opts
  • add_stage3_cmake_opts
  • add_all_stages_cmake_opts

And than we could ensure they are never called after a particular stage has already been ran, and that the internal methods are never touched manually.
This would require quite a lot of refactoring, and high possibility to introduce new errors which i would avoid for now

PS would also require rewriting the logic in how the configopts coming from the EC file now are treated as general_opts + would need to streamline and clarify how stages are treated in the case of bootstrap vs not

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only way to really make sure stuff is not misused is to introduce 4 methods:

Would it then need to be a property at all? The current member methods that additionally modify it could just a return a dict that is then used to update a local cmakeopts

@boegel boegel changed the title LLVM: Fix missing LIBOMPTARGET_DEVICE_ARCHITECTURES in third stage of bootstrap build fix missing LIBOMPTARGET_DEVICE_ARCHITECTURES in third stage of bootstrap build in LLVM easyblock Jul 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants