Incorrect module loads in modules built by users with HMNS, minimal-toolchain, recursive-unload

[akesandgren]

We have the following system setup:

eb --show-config
#
# Current EasyBuild configuration
# (C: command line argument, D: default value, E: environment variable, F: configuration file)
#
buildpath               (F) = /scratch
filter-deps             (F) = pkg-config
git-working-dirs-path   (F) = /afs/hpc2n.umu.se/home/a/ake/Projects/git-repos/easybuild
github-user             (F) = akesandgren
include-easyblocks      (F) = /hpc2n/eb/custom/easyblocks/*.py
installpath             (F) = /hpc2n/eb
job-backend-config      (F) = /hpc2n/eb/gc3pie.slurm.cfg
minimal-toolchains      (F) = True
module-naming-scheme    (F) = HierarchicalMNS
packagepath             (F) = /hpc2n/eb/packages
prefix                  (F) = /hpc2n/eb
recursive-module-unload (F) = True
repositorypath          (F) = /hpc2n/eb/ebfiles_repo
robot-paths             (F) = /afs/hpc2n.umu.se/home/a/ake/Public/easybuild/easyconfigs, /hpc2n/eb/custom/easyconfigs, /hpc2n/eb/ebfiles_repo, /hpc2n/eb/software/Core/EasyBuild/3.1.1/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.1-py2.7.egg/easybuild/easyconfigs
sourcepath              (F) = /hpc2n/eb/sources:/pfs/software/sources
subdir-user-modules     (F) = Public/easybuild/modules
tmpdir                  (F) = /scratch

Things work as expected and modules get correctly created.
But when adding the SoftInst module from @geimer on top of this to allow for users to build their own softwares we get incorrectly built modules.

ml SoftInst
eb --show-config
#
# Current EasyBuild configuration
# (C: command line argument, D: default value, E: environment variable, F: configuration file)
#
buildpath               (E) = /scratch/eb-ake
experimental            (E) = True
filter-deps             (F) = pkg-config
git-working-dirs-path   (F) = /afs/hpc2n.umu.se/home/a/ake/Projects/git-repos/easybuild
github-user             (F) = akesandgren
include-easyblocks      (F) = /hpc2n/eb/custom/easyblocks/*.py
installpath             (E) = /home/a/ake/Public/easybuild
job-backend-config      (F) = /hpc2n/eb/gc3pie.slurm.cfg
job-cores               (E) = 24
job-max-walltime        (E) = 4
minimal-toolchains      (F) = True
module-naming-scheme    (F) = HierarchicalMNS
optarch                 (E) = {'GCC': 'march=x86-64 -mmmx -msse3 -mcx16 -mfxsr -mpopcnt -msahf -mtune=generic'}
packagepath             (E) = /home/a/ake/Public/easybuild/packages
prefix                  (E) = /home/a/ake/Public/easybuild
recursive-module-unload (F) = True
repositorypath          (E) = /home/a/ake/Public/easybuild/ebfiles_repo
robot                   (E) = /hpc2n/eb/custom/easyconfigs, /afs/hpc2n.umu.se/home/a/ake/Public/easybuild/easyconfigs, /hpc2n/eb/custom/easyconfigs, /hpc2n/eb/ebfiles_repo, /hpc2n/eb/software/Core/EasyBuild/3.1.1/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.1-py2.7.egg/easybuild/easyconfigs
robot-paths             (E) = /hpc2n/eb/custom/easyconfigs, /afs/hpc2n.umu.se/home/a/ake/Public/easybuild/easyconfigs, /hpc2n/eb/custom/easyconfigs, /hpc2n/eb/ebfiles_repo, /hpc2n/eb/software/Core/EasyBuild/3.1.1/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.1-py2.7.egg/easybuild/easyconfigs
sourcepath              (E) = /home/a/ake/Public/easybuild/sources:/hpc2n/eb/sources
subdir-user-modules     (E) = ''
suffix-modules-path     (E) = ''
tmpdir                  (F) = /scratch
zip-logs                (E) = bzip2

With everything else in the chain installed at system level building, as user with SoftInst loaded,

eb x264-20160614-foss-2016b.eb --try-toolchain-version=2017a

the resulting x264 module contains:

~/Public/easybuild/modules/MPI/GCC/6.3.0-2.27/OpenMPI/2.0.2/x264/20160614.lua 
help([[x264 is a free software library and application for encoding video streams into the H.264/MPEG-4
 AVC compression format, and is released under the terms of the GNU GPL. - Homepage: http://www.videolan.org/developers/x264.html]])

whatis([[Description: x264 is a free software library and application for encoding video streams into the H.264/MPEG-4
 AVC compression format, and is released under the terms of the GNU GPL. - Homepage: http://www.videolan.org/developers/x264.html]])

local root = "/home/a/ake/Public/easybuild/software/MPI/GCC/6.3.0-2.27/OpenMPI/2.0.2/x264/20160614"

conflict("x264")

load("GCC/6.3.0-2.27")

load("OpenMPI/2.0.2")

load("OpenBLAS/0.2.19-LAPACK-3.7.0")

load("FFTW/3.3.6")

load("ScaLAPACK/2.0.2-OpenBLAS-0.2.19-LAPACK-3.7.0")

load("Yasm/1.3.0")

prepend_path("CPATH", pathJoin(root, "include"))
prepend_path("LD_LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("PATH", pathJoin(root, "bin"))
prepend_path("PKG_CONFIG_PATH", pathJoin(root, "lib/pkgconfig"))
setenv("EBROOTX264", root)
setenv("EBVERSIONX264", "20160614")
setenv("EBDEVELX264", pathJoin(root, "easybuild/MPI-GCC-6.3.0-2.27-OpenMPI-2.0.2-x264-20160614-easybuild-devel"))

-- Built with EasyBuild version 3.1.1

Note the incorrect load of OpenMPI in there.

SoftInst module is:

conflict("SoftInst")

setenv("EASYBUILD_MODULES_TOOL", "Lmod")
setenv("EASYBUILD_ZIP_LOGS", "bzip2")
setenv("EASYBUILD_JOB_CORES", "24")
setenv("EASYBUILD_PREFIX", "/hpc2n/eb")
setenv("EASYBUILD_UMASK", "022")
setenv("EASYBUILD_SET_GID_BIT", "1")
setenv("EASYBUILD_JOB_MAX_WALLTIME", "4")
setenv("EASYBUILD_OPTARCH", "GCC:march=x86-64 -mmmx -msse3 -mcx16 -mfxsr -mpopcnt -msahf -mtune=generic")
setenv("EASYBUILD_ROBOT", "/hpc2n/eb/custom/easyconfigs")
setenv("EASYBUILD_SUFFIX_MODULES_PATH", "")
setenv("EASYBUILD_MODULE_SYNTAX", "Lua")
setenv("EASYBUILD_REPOSITORY", "FileRepository")
setenv("EASYBUILD_MODULE_NAMING_SCHEME", "HierarchicalMNS")
setenv("EASYBUILD_SUBDIR_USER_MODULES", "Public/easybuild/modules")
setenv("EASYBUILD_SOURCEPATH", "/hpc2n/eb/sources")
setenv("EASYBUILD_JOB_BACKEND_CONFIG", "/hpc2n/eb/gc3pie.slurm.cfg")
setenv("EASYBUILD_REPOSITORYPATH", "/hpc2n/eb/ebfiles_repo")
setenv("EASYBUILD_MINIMAL_TOOLCHAINS", "1")
-- setenv("EASYBUILD_HIDE_TOOLCHAINS", "GCCcore,gompi")
setenv("EASYBUILD_INCLUDE_EASYBLOCKS", "/hpc2n/eb/custom/easyblocks/*.py")
setenv("EASYBUILD_JOB_BACKEND", "GC3Pie")
setenv("EASYBUILD_EXPERIMENTAL", "1")
load("EasyBuild/3.1.1")
load("GC3Pie/2.4.2")

setenv("EASYBUILD_BUILDPATH", pathJoin("/scratch", "eb-" .. os.getenv("USER")))
if os.getenv("USER") ~= "admin" then
    userprefix=pathJoin(os.getenv("HOME"), "Public", "easybuild")
    setenv("EASYBUILD_PREFIX", pathJoin(os.getenv("HOME"), "Public", "easybuild"))
    prepend_path("EASYBUILD_SOURCEPATH", pathJoin(userprefix, "sources"))
    setenv("EASYBUILD_SUBDIR_USER_MODULES", "")
    unsetenv("EASYBUILD_REPOSITORYPATH");
    unsetenv("EASYBUILD_SET_GID_BIT")
    unsetenv("EASYBUILD_UMASK")
    load("eb_user/.current")
end

and the eb_user module is:

conflict("eb_user")

prepend_path("PATH", "/hpc2n/eb/software/Core/eb_user/bin")

with the eb command in eb_user/bin being:

cat /hpc2n/eb/software/Core/eb_user/bin/eb 
#!/bin/sh
export MODULEPATH=/hpc2n/eb/modules/all:${MODULEPATH}
${EBROOTEASYBUILD}/bin/eb "$@"

We need to figure this out rather quickly if possible...

[ocaisa]

In what way is the OpenMPI load incorrect? Don't you want to guarantee that the hierarchy is correctly loaded since you will have to explicitly module use the modules of the user (i.e. they may be visible without having expanded your hierarchy yet)?

[ocaisa]

For this user space i would probably just turn off recursive unloading by default

[geimer]

@akesandgren Isn't this the same issue than addressed in #2152? In fact, I would assume the same load statements to appear with a system-wide installation.

BTW: you probably want to adjust the EASYBUILD_OPTARCH setting in the SoftInst module. I needed to be creative here, as our test system has heterogeneous hardware.

[akesandgren]

@ocaisa OpenMPI is already loaded due to the hierarchy, and it is causing problems when building ffmpeg on top of x264, just the same problem i had with eb not finding binutils due to GCCcore getting loaded when it was already loaded.

[akesandgren]

@geimer Sort of the same as in #2152 but i already have that fix in place, and system-wide install doesn't get load openmpi in the x264 module.
System-wide (no SoftInst module loaded) x264 module looks like:

local root = "/hpc2n/eb/software/MPI/GCC/6.3.0-2.27/OpenMPI/2.0.2/x264/20160614"

conflict("x264")

load("OpenBLAS/0.2.19-LAPACK-3.7.0")

load("FFTW/3.3.6")

load("ScaLAPACK/2.0.2-OpenBLAS-0.2.19-LAPACK-3.7.0")

load("Yasm/1.3.0")

prepend_path("CPATH", pathJoin(root, "include"))
prepend_path("LD_LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("LIBRARY_PATH", pathJoin(root, "lib"))
prepend_path("PATH", pathJoin(root, "bin"))
prepend_path("PKG_CONFIG_PATH", pathJoin(root, "lib/pkgconfig"))
setenv("EBROOTX264", root)
setenv("EBVERSIONX264", "20160614")
setenv("EBDEVELX264", pathJoin(root, "easybuild/MPI-GCC-6.3.0-2.27-OpenMPI-2.0.2-x264-.20160614-easybuild-devel"))

-- Built with EasyBuild version 3.1.1

[ocaisa]

Maybe I don't understand fully how this setting functions. Did you have to rebuild your compiler/MPI modules to get module path extensions into the user space? If not then don't you have to explicitly do

module use /path/to/users/modules

(either in the environment or in another module) to get access to the users module space? If you do this then what is the guarantee that the hierarchy is populated when you do "module load x264"? It could easily be visible without OpenMPI being loaded. For the user space I would simply not set recursive module unload, this will protect you against unloading the OpenMPI module.

[akesandgren]

@ocaisa The "subdir-user-modules (F) = Public/easybuild/modules" setting makes the modules that expand MODULEPATH contain things like

prepend_path("MODULEPATH", "/hpc2n/eb/modules/all/Compiler/GCCcore/6.3.0")
if isDir(pathJoin(os.getenv("HOME"), "Public/easybuild/modules/Compiler/GCCcore/6.3.0")) then
    prepend_path("MODULEPATH", pathJoin(os.getenv("HOME"), "Public/easybuild/modules/Compiler/GCCcore/6.3.0"))
end

So users do not have to do "module use $HOME/Public/easybuild/modules"

[akesandgren]

I will try unsetting recursive for user level builds, but it's still a bug that needs to be figured out.

[boegel]

This is probably related to the way in which we exclude dependencies in make_module_dep, where we hunt down the modules that extend $MODULEPATH with a specific location.

Since as a user you are installing modules in a different location, the check is probably done with an incorrect location...

@bartoldeman Thoughts on this?

[bartoldeman]

I've seen similar things, using a simple --prefix=$HOME/easybuild to test out modules.
I think the issue is that MODULEPATH is compared literally, not taking into account the

if isDir(pathJoin(os.getenv("HOME"), "Public/easybuild/modules/Compiler/GCCcore/6.3.0")) then
    prepend_path("MODULEPATH", pathJoin(os.getenv("HOME"), "Public/easybuild/modules/Compiler/GCCcore/6.3.0"))
end

stuff. I'll have a further look; sometimes this the output for such logic can be parsed by "module show" in Lmod.

[bartoldeman]

My PR #2104 addresses this issue. However it fails Travis. Let me check why.

[bartoldeman]

PR #2104 has been updated to (hopefully) pass Travis.

[boegel]

@geimer Are you also seeing the superfluous loads in user-installed modules like @akesandgren does, with current develop?