Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .github/workflows/parse_missing-installations-output.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import os
import re

missing = os.environ['missing']
missing = missing.split('\n')
missing_cuda = []
missing_cpu = []
for ec in missing:
if re.search('CUDA', ec):
missing_cuda.append(ec)
else:
missing_cpu.append(ec)
if len(missing_cpu) != 0 and len(missing_cuda) != 0:
print(f'Please open a seperate pr for these dependencies: {missing_cpu}')
os.write(2, b'Error: CPU dependencies for CUDA build must be build in a seperate pr')
exit(1)
elif len(missing_cuda) != 0:
# TODO: Make this set the accelorator label?
print(f'Have fun installing the following gpu builds: {missing_cuda}')
elif len(missing_cpu) != 0:
print(f'Have fun installing the following gpu builds: {missing_cpu}')
else:
print('no missing modules')
77 changes: 77 additions & 0 deletions .github/workflows/test-check_easystacks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# documentation: https://help.github.com/en/articles/workflow-syntax-for-github-actions
name: Check for CUDA builds in changed easystacks
on:
push:
branches: [ "*-software.eessi.io" ]
pull_request:
workflow_dispatch:
permissions:
contents: read # to fetch code (actions/checkout)
env:
EESSI_ACCELERATOR_TARGETS: |
x86_64/amd/zen2:
- nvidia/cc80
x86_64/amd/zen3:
- nvidia/cc80
jobs:
check_missing:
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
EESSI_VERSION:
- 2023.06
EESSI_SOFTWARE_SUBDIR_OVERRIDE:
- x86_64/amd/zen2
- x86_64/amd/zen3
- x86_64/amd/zen4
- x86_64/intel/haswell
- x86_64/intel/skylake_avx512
- x86_64/generic
steps:
- name: Check out software-layer repository
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

- name: Mount EESSI CernVM-FS pilot repository
uses: cvmfs-contrib/github-action-cvmfs@55899ca74cf78ab874bdf47f5a804e47c198743c # v4.0
with:
cvmfs_config_package: https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb
cvmfs_http_proxy: DIRECT
cvmfs_repositories: software.eessi.io
- name: find changed easystacks
uses: dorny/paths-filter@v3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would need to use a specific commit from that repo to pass our security checks

Suggested change
uses: dorny/paths-filter@v3
uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 #v3.0.2

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does something else need to be done because I'm still seeing this? https://github.com/EESSI/software-layer/actions/runs/13587202930

id: changes
with:
list-files: shell
filters: |
changed:
- 'easystacks/**'
- name: Test check_missing_installations.sh script
if: ${{ steps.changes.outputs.changed == 'true' }}
run: |
export EESSI_SOFTWARE_SUBDIR_OVERRIDE=${{matrix.EESSI_SOFTWARE_SUBDIR_OVERRIDE}}
source /cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}/init/bash
# set $EESSI_CPU_FAMILY to the CPU architecture that corresponds to $EESSI_SOFTWARE_SUBDIR_OVERRIDE (part before the first slash),
# to prevent issues with checks in the Easybuild configuration that use this variable
export EESSI_CPU_FAMILY=${EESSI_SOFTWARE_SUBDIR_OVERRIDE%%/*}
module load EasyBuild
which eb
eb --version
export EESSI_PREFIX=/cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}
export EESSI_OS_TYPE=linux
env | grep ^EESSI | sort
# check for missing installations if a file in the easystacks directory is changed
echo "check for missing installation in changed easystack files"
easystack_files=(${{ steps.changes.outputs.changed_files }})
for easystack_file in ${easystack_files[@]}; do
echo "check missing installatios for ${easystack_file}..."
./check_missing_installations.sh ${easystack_file} || ec=$?
export eb_missing_out=$(./check_missing_installations.sh ${easystack_file})
if [[ ${ec} -ne 0 ]]; then
echo "missing installations found for ${easystack_file}!" >&2
export missing=$(env | grep .eb\) | tr -d \*)
python .github/workflows/parse_missing-installations-output.py
fi
done
6 changes: 2 additions & 4 deletions .github/workflows/test-software.eessi.io.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,7 @@ jobs:
echo "just run check_missing_installations.sh (should use easystacks/software.eessi.io/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-*.yml with latest EasyBuild release)"
for easystack_file in $(EESSI_VERSION=${{matrix.EESSI_VERSION}} .github/workflows/scripts/only_latest_easystacks.sh); do
echo "check missing installations for ${easystack_file}..."
./check_missing_installations.sh ${easystack_file}
ec=$?
./check_missing_installations.sh ${easystack_file} || ec=$?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this will work, it only sets ec if the script did not exit successfully

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that is what I want otherwise the workflow stops. And than this error message does not get printed "missing installations found for ${easystack_file}!"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also tested it with an easystack were everything was already installed and that went fine as well.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can test again to see if I overlooked something.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah no your right

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok than I'll have to find another way around it

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't you just do

ec=0
./check_missing_installations.sh ${easystack_file} || ec=$?

If it exits succesfully, ec remains 0, if it fails, you get the exit code. That's what you want, right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes I will try tomorrow

if [[ ${ec} -ne 0 ]]; then echo "missing installations found for ${easystack_file}!" >&2; exit ${ec}; fi
done

Expand All @@ -75,8 +74,7 @@ jobs:
echo "checking missing installations for accelerator ${accel} using modulepath: ${MODULEPATH}"
for easystack_file in $(EESSI_VERSION=${{matrix.EESSI_VERSION}} ACCEL_EASYSTACKS=1 .github/workflows/scripts/only_latest_easystacks.sh); do
echo "check missing installations for ${easystack_file}..."
./check_missing_installations.sh ${easystack_file}
ec=$?
./check_missing_installations.sh ${easystack_file} || ec=$?
if [[ ${ec} -ne 0 ]]; then echo "missing installations found for ${easystack_file}!" >&2; exit ${ec}; fi
done
module unuse ${EESSI_SOFTWARE_PATH}/accel/${accel}/modules/all
Expand Down