Module Configuration Tutorial

This tutorial will guide you through the customization of both content and naming of module files generated by Spack.

Starting from the default Spack settings you will add an increasing number of directives to the modules.yaml configuration file to satisfy a number of constraints that mimic those that you may encounter in a typical production environment at HPC sites.

Even though the focus will be for the most part on customizing TCL non-hierarchical module files, everything you’ll see applies also to other kinds of module files generated by Spack.

OPTIONAL The generation of Lua hierarchical module files will be addressed at the end of the tutorial, and you’ll see that with minor modifications to an existing modules.yaml written for TCL non-hierarchical modules you’ll get almost for free the possibility to try a hierarchical layout.

Let’s start!

Prerequisites

Before proceeding further ensure:

  • have shell support activated in Spack
  • Add spack modulefiles directory to your MODULEPATH:
$ export MODULEPATH="$MODULEPATH:$SPACK_ROOT/share/spack/modules/cray-CNL-haswell

$SPACK_ROOT being the location where your spack is installed.

Fake install gcc-6.2.0

Because installs will take long and we are using the shared login nodes, we will fake install gcc-6.2.0:

$ spack install --fake gcc@6.2.0%gcc@4.9.3

If your default PrgEnv is intel, you might get some intel compiler output icpc: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance

Since we are not really doing a real install this can be ignored.

Build software that will be used in the tutorial

Before you build uninstall everything so we have a clean Spack.

$ spack uninstall --all

Next you should install a few modules that will be used in the tutorial. Rather than wait for all these packages to build, let’s fake install them:

$ spack install --fake netlib-scalapack ^mpich ^openblas

Fake install these packages as well

  • $ spack install --fake netlib-scalapack ^openmpi ^openblas
  • $ spack install --fake netlib-scalapack ^openmpi ^netlib-lapack
  • $ spack install --fake netlib-scalapack ^mpich ^netlib-lapack

In the end your environment should look something like:

$ module avail
# NERSC module output...
----------- /global/homes/m/mamelara/spack_tutorial/share/spack/modules/cray-CNL-haswell/ -----------
binutils-2.27-gcc-4.9.3-cb4h6tu          mpc-1.0.3-gcc-4.9.3-5eniuop
bison-3.0.4-gcc-4.9.3-li2aeku            mpfr-3.1.4-gcc-4.9.3-oldrzev
bzip2-1.0.6-gcc-4.9.3-4vdw2sy            ncurses-6.0-gcc-4.9.3-eojom4i
cmake-3.7.1-gcc-6.2.0-groolt4            ncurses-6.0-gcc-6.2.0-ti24bta
flex-2.6.1-gcc-4.9.3-z77xnab             netlib-lapack-3.6.1-gcc-6.2.0-xbpjs6n
gcc-6.2.0-gcc-4.9.3-phcwckh              netlib-scalapack-2.0.2-gcc-6.2.0-2ukncs7
gettext-0.19.8.1-gcc-4.9.3-v4pfe73       netlib-scalapack-2.0.2-gcc-6.2.0-4ok2ap5
gmp-6.1.2-gcc-4.9.3-m25fwyp              netlib-scalapack-2.0.2-gcc-6.2.0-gfj6bxd
help2man-1.47.4-gcc-4.9.3-ynzj7ma        netlib-scalapack-2.0.2-gcc-6.2.0-mqhupzw
hwloc-1.11.4-gcc-6.2.0-djmeswy           openblas-0.2.19-gcc-6.2.0-mm74ids
isl-0.14-gcc-4.9.3-uyqqxuu               openmpi-2.0.1-gcc-6.2.0-6vi4ni5
libpciaccess-0.13.4-gcc-6.2.0-akapxtm    openssl-1.0.2j-gcc-6.2.0-rtibvnm
libsigsegv-2.10-gcc-4.9.3-o3eifqy        tar-1.29-gcc-4.9.3-wx4clye
libsigsegv-2.10-gcc-6.2.0-cro3222        util-macros-1.19.0-gcc-6.2.0-ojeaurs
libtool-2.4.6-gcc-6.2.0-rt7axrx          xz-5.2.2-gcc-4.9.3-6pu6nbt
libxml2-2.9.4-gcc-4.9.3-f4vv5ko          zlib-1.2.8-gcc-4.9.3-34tqb7c
m4-1.4.17-gcc-4.9.3-ukvopef              zlib-1.2.8-gcc-6.2.0-vnwrdo3
m4-1.4.17-gcc-6.2.0-ixzogge

Filter unwanted modifications to the environment

The non-hierarchical TCL module files that have been generated so far follow the default rules for module generation, which are given here in the reference part of the manual. Taking a look at the gcc module you’ll see something like:

As expected, a few environment variables representing paths will be modified by the modules according to the default prefix inspection rules.

Consider now the case that your site has decided that e.g. CPATH and LIBRARY_PATH modifications should not be present in module files. What you can do to abide by the rules is to create a configuration file ~/.spack/modules.yaml with the following content:

modules:
  tcl:
    all:
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']

Do this by using the command spack config edit modules.

Next you should regenerate all the module files:

$ spack module refresh --module-type tcl
==> You are about to regenerate tcl module files for:

-- cray-CNL-haswell / gcc@4.9.3 ---------------------------------
cb4h6tu binutils@2.27  v4pfe73 gettext@0.19.8.1  f4vv5ko libxml2@2.9.4  wx4clye tar@1.29
li2aeku bison@3.0.4    m25fwyp gmp@6.1.2         ukvopef m4@1.4.17      6pu6nbt xz@5.2.2
4vdw2sy bzip2@1.0.6    ynzj7ma help2man@1.47.4   5eniuop mpc@1.0.3      34tqb7c zlib@1.2.8
z77xnab flex@2.6.1     uyqqxuu isl@0.14          oldrzev mpfr@3.1.4
phcwckh gcc@6.2.0      o3eifqy libsigsegv@2.10   eojom4i ncurses@6.0

-- cray-CNL-haswell / gcc@6.2.0 ---------------------------------
groolt4 cmake@3.7.1          ti24bta ncurses@6.0             mm74ids openblas@0.2.19
djmeswy hwloc@1.11.4         xbpjs6n netlib-lapack@3.6.1     6vi4ni5 openmpi@2.0.1
akapxtm libpciaccess@0.13.4  mqhupzw netlib-scalapack@2.0.2  rtibvnm openssl@1.0.2j
cro3222 libsigsegv@2.10      2ukncs7 netlib-scalapack@2.0.2  ojeaurs util-macros@1.19.0
rt7axrx libtool@2.4.6        4ok2ap5 netlib-scalapack@2.0.2  vnwrdo3 zlib@1.2.8
ixzogge m4@1.4.17            gfj6bxd netlib-scalapack@2.0.2

==> Do you want to proceed ? [y/n]
y
==> Regenerating tcl module files

If you take a look now at the module for gcc you’ll see that the unwanted paths have disappeared:

$ module show gcc-6.2.0-gcc-4.9.3-3wm2efx
-------------------------------------------------------------------
/global/homes/m/mamelara/spack/share/spack/modules/cray-CNL-haswell/gcc-6.2.0-gcc-4.9.3-3wm2efx:

module-whatis  gcc @6.2.0
prepend-path   PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/bin
prepend-path   CMAKE_PREFIX_PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/
prepend-path   LD_LIBRARY_PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/lib
prepend-path   MANPATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/man
-------------------------------------------------------------------

Prevent some module files from being generated

Another common request at many sites is to avoid exposing software that is only needed as an intermediate step when building a newer stack. Let’s try to prevent the generation of module files for anything that is compiled with gcc@4.9.3.

To do this you should add a blacklist keyword to the configuration file:

modules:
  tcl:
    blacklist:
      -  '%gcc@4.9.3'
    all:
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']

and regenerate the module files:

$ spack module refresh --module-type tcl --delete-tree
==> You are about to regenerate tcl module files for:

-- cray-CNL-haswell / gcc@4.9.3 ---------------------------------
cb4h6tu binutils@2.27  v4pfe73 gettext@0.19.8.1  f4vv5ko libxml2@2.9.4  wx4clye tar@1.29
li2aeku bison@3.0.4    m25fwyp gmp@6.1.2         ukvopef m4@1.4.17      6pu6nbt xz@5.2.2
4vdw2sy bzip2@1.0.6    ynzj7ma help2man@1.47.4   5eniuop mpc@1.0.3      34tqb7c zlib@1.2.8
z77xnab flex@2.6.1     uyqqxuu isl@0.14          oldrzev mpfr@3.1.4
phcwckh gcc@6.2.0      o3eifqy libsigsegv@2.10   eojom4i ncurses@6.0

-- cray-CNL-haswell / gcc@6.2.0 ---------------------------------
groolt4 cmake@3.7.1          ti24bta ncurses@6.0             mm74ids openblas@0.2.19
djmeswy hwloc@1.11.4         xbpjs6n netlib-lapack@3.6.1     6vi4ni5 openmpi@2.0.1
akapxtm libpciaccess@0.13.4  mqhupzw netlib-scalapack@2.0.2  rtibvnm openssl@1.0.2j
cro3222 libsigsegv@2.10      2ukncs7 netlib-scalapack@2.0.2  ojeaurs util-macros@1.19.0
rt7axrx libtool@2.4.6        4ok2ap5 netlib-scalapack@2.0.2  vnwrdo3 zlib@1.2.8
ixzogge m4@1.4.17            gfj6bxd netlib-scalapack@2.0.2

==> Do you want to proceed ? [y/n]
y
==> Regenerating tcl module files

$ module avail

----------- /global/homes/m/mamelara/spack_tutorial/share/spack/modules/cray-CNL-haswell/ -----------
cmake-3.7.1-gcc-6.2.0-groolt4            netlib-scalapack-2.0.2-gcc-6.2.0-4ok2ap5
hwloc-1.11.4-gcc-6.2.0-djmeswy           netlib-scalapack-2.0.2-gcc-6.2.0-gfj6bxd
libpciaccess-0.13.4-gcc-6.2.0-akapxtm    netlib-scalapack-2.0.2-gcc-6.2.0-mqhupzw
libsigsegv-2.10-gcc-6.2.0-cro3222        openblas-0.2.19-gcc-6.2.0-mm74ids
libtool-2.4.6-gcc-6.2.0-rt7axrx          openmpi-2.0.1-gcc-6.2.0-6vi4ni5
m4-1.4.17-gcc-6.2.0-ixzogge              openssl-1.0.2j-gcc-6.2.0-rtibvnm
ncurses-6.0-gcc-6.2.0-ti24bta            util-macros-1.19.0-gcc-6.2.0-ojeaurs
netlib-lapack-3.6.1-gcc-6.2.0-xbpjs6n    zlib-1.2.8-gcc-6.2.0-vnwrdo3
netlib-scalapack-2.0.2-gcc-6.2.0-2ukncs7

This time it is convenient to pass the option --delete-tree to the command that regenerates the module files to instruct it to delete the existing tree and regenerate a new one instead of overwriting the files in the existing directory.

If you pay careful attention you’ll see though that we went too far in blacklisting modules: the module for gcc@6.2.0 disappeared as it was bootstrapped with gcc@4.9.3. To specify exceptions to the blacklist rules you can use whitelist:

modules:
  tcl:
    whitelist:
      - gcc
    blacklist:
      - '%gcc@4.9.3'
  all:
    filter:
       environment_blacklist = ['CPATH', 'LIBRARY_PATH']

whitelist rules always have precedence over blacklist rules. If you generate the modules again:

$ spack module refresh --module-type tcl -y
==> Regenerating tcl module files

You’ll see that now the module for gcc@6.2.0 has reappeared.

$ module avail gcc-6.2.0-gcc-4.9.3-3wm2efx
---------------- /global/homes/m/mamelara/spack/share/spack/modules/cray-CNL-haswell ----------------
gcc-6.2.0-gcc-4.9.3-3wm2efx

The next step in making module files more user-friendly is to improve their naming scheme. To reduce the length of the hash or remove it altogether you can use the hash_length keyword in the configuration file:

modules:
  tcl:
    hash_length: 0
    blacklist:
      -  '%gcc@4.9.3'
    all:
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']

Try to regenerate the module files now:

$ spack module refresh --module-type tcl --delete-tree -y
==> Regenerating tcl module files
==> Error: Name clashes detected in module files:

file : /global/u2/m/mamelara/spack/share/spack/modules/cray-CNL-haswell/netlib-scalapack-2.0.2-gcc-6.2.0
spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=cray-CNL-haswell
spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=cray-CNL-haswell
spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=cray-CNL-haswell
spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=cray-CNL-haswell

==> Error: Operation aborted

Note

We try to check for errors upfront!
Name clashes will happen if you have multiple packages of the same name. In Spack we check for errors upfront whenever possible, so don’t worry about your module files: as a name clash was detected nothing has been changed on disk.

The problem here is that without the hashes the four different flavors of netlib-scalapack map to the same module file name. We have the possibility to add suffixes to differentiate them:

 modules:
   tcl:
     hash_length: 0
     whitelist:
       -  gcc
     blacklist:
       -  '%gcc@4.9.3'
     all:
       suffixes:
         '^openblas': openblas
         '^netlib-lapack': netlib
       filter:
         environment_blacklist: ['CPATH', 'LIBRARY_PATH']
     netlib-scalapack:
       suffixes:
         '^openmpi': openmpi
         '^mpich': mpich

As you can see it is possible to specify rules that applies only to a restricted set of packages using anonymous specs. Regenerating module files now we obtain:

$ spack module refresh --module-type tcl --delete-tree -y
==> Regenerating tcl module files
$ module avail
----------- /global/homes/m/mamelara/spack_tutorial/share/spack/modules/cray-CNL-haswell/ -----------
cmake-3.7.1-gcc-6.2.0                             netlib-scalapack-2.0.2-gcc-6.2.0-netlib-mpich
gcc-6.2.0-gcc-4.9.3                               netlib-scalapack-2.0.2-gcc-6.2.0-netlib-openmpi
hwloc-1.11.4-gcc-6.2.0                            netlib-scalapack-2.0.2-gcc-6.2.0-openblas-mpich
libpciaccess-0.13.4-gcc-6.2.0                     netlib-scalapack-2.0.2-gcc-6.2.0-openblas-openmpi
libsigsegv-2.10-gcc-6.2.0                         openblas-0.2.19-gcc-6.2.0
libtool-2.4.6-gcc-6.2.0                           openmpi-2.0.1-gcc-6.2.0
m4-1.4.17-gcc-6.2.0                               openssl-1.0.2j-gcc-6.2.0
ncurses-6.0-gcc-6.2.0                             util-macros-1.19.0-gcc-6.2.0
netlib-lapack-3.6.1-gcc-6.2.0                     zlib-1.2.8-gcc-6.2.0

Finally we can set a naming_scheme to prevent users from loading modules that refer to different flavors of the same library/application:

modules:
  tcl:
    hash_length: 0
    naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}'
    whitelist:
      -  gcc
    blacklist:
      -  '%gcc@4.9.3'
    all:
      conflict:
        - '${PACKAGE}'
      suffixes:
        '^openblas': openblas
        '^netlib-lapack': netlib
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']
    netlib-scalapack:
      suffixes:
        '^openmpi': openmpi
        '^mpich': mpich

The final result should look like:

$ spack module refresh --module-type tcl --delete-tree -y
==> Regenerating tcl module files

$ module avail

----------- /global/homes/m/mamelara/spack_tutorial/share/spack/modules/cray-CNL-haswell/ -----------
cmake/3.7.1-gcc-6.2.0                             netlib-scalapack/2.0.2-gcc-6.2.0-netlib-mpich
gcc/6.2.0-gcc-4.9.3                               netlib-scalapack/2.0.2-gcc-6.2.0-netlib-openmpi
hwloc/1.11.4-gcc-6.2.0                            netlib-scalapack/2.0.2-gcc-6.2.0-openblas-mpich
libpciaccess/0.13.4-gcc-6.2.0                     netlib-scalapack/2.0.2-gcc-6.2.0-openblas-openmpi
libsigsegv/2.10-gcc-6.2.0                         openblas/0.2.19-gcc-6.2.0
libtool/2.4.6-gcc-6.2.0                           openmpi/2.0.1-gcc-6.2.0
m4/1.4.17-gcc-6.2.0                               openssl/1.0.2j-gcc-6.2.0
ncurses/6.0-gcc-6.2.0                             util-macros/1.19.0-gcc-6.2.0
netlib-lapack/3.6.1-gcc-6.2.0                     zlib/1.2.8-gcc-6.2.0

Note

TCL specific directive
The directives naming_scheme and conflict are TCL specific and do not apply to the dotkit or lmod sections in the configuration file.

Add custom environment modifications

At many sites it is customary to set an environment variable in a package’s module file that points to the folder in which the package is installed. You can achieve this with Spack by adding an environment directive to the configuration file:

modules:
  tcl:
    hash_length: 0
    naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}'
    whitelist:
      -  gcc
    blacklist:
      -  '%gcc@4.9.3'
    all:
      conflict:
        - '${PACKAGE}'
      suffixes:
        '^openblas': openblas
        '^netlib-lapack': netlib
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']
      environment:
        set:
          '${PACKAGE}_ROOT': '${PREFIX}'
    netlib-scalapack:
      suffixes:
        '^openmpi': openmpi
        '^mpich': mpich

There are many variable tokens available to use in the environment and naming_scheme directives, such as ${PACKAGE}, ${VERSION}, etc. (see the format() API documentation for the complete list).

Regenerating the module files should result in something like:

$ spack module refresh -y --module-type tcl
==> Regenerating tcl module files

$ module show gcc/6.2.0-gcc-4.9.3
-------------------------------------------------------------------
/global/homes/m/mamelara/spack/share/spack/modules/cray-CNL-haswell/gcc/6.2.0-gcc-4.9.3:

module-whatis  gcc @6.2.0
prepend-path   PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/bin
prepend-path   CMAKE_PREFIX_PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/
prepend-path   LD_LIBRARY_PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/lib
prepend-path   MANPATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/man
setenv         GCC_ROOT /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy
conflict       gcc
-------------------------------------------------------------------

As you see the gcc module has the environment variable GCC_ROOT set.

Sometimes it’s also useful to apply environment modifications selectively and target only certain packages. You can, for instance set the common variables CC, CXX, etc. in the gcc module file and apply other custom modifications to the openmpi modules as follows:

modules:
  tcl:
    hash_length: 0
    naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}'
    whitelist:
      - gcc
    blacklist:
      - '%gcc@4.9.3'
    all:
      conflict:
        - '${PACKAGE}'
      suffixes:
        '^openblas': openblas
        '^netlib-lapack': netlib
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']
      environment:
        set:
          '${PACKAGE}_ROOT': '${PREFIX}'
    gcc:
      environment:
        set:
          CC: gcc
          CXX: g++
          FC: gfortran
          F90: gfortran
          F77: gfortran
    openmpi:
      environment:
        set:
          SLURM_MPI_TYPE: pmi2
          OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
    netlib-scalapack:
      suffixes:
        '^openmpi': openmpi
        '^mpich': mpich

This time we will be more selective and regenerate only the gcc and openmpi module files:

$ spack module refresh -y --module-type tcl gcc
==> Regenerating tcl module files

$ spack module refresh -y --module-type tcl openmpi
==> Regenerating tcl module files

$ module show gcc/6.2.0-4.9.3
-------------------------------------------------------------------
/global/homes/m/mamelara/spack/share/spack/modules/cray-CNL-haswell/gcc/6.2.0-gcc-4.9.3:

module-whatis  gcc @6.2.0
prepend-path   PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/bin
prepend-path   CMAKE_PREFIX_PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/
prepend-path   LD_LIBRARY_PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/lib
prepend-path   MANPATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy/man
setenv     GCC_ROOT /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-4.9.3/gcc-6.2.0-3wm2efxnt3zlu4rkjyfztiwcpquxqeqy
setenv     CC gcc
setenv     CXX g++
setenv     F90 gfortran
setenv     FC gfortran
setenv     F77 gfortran
conflict   gcc
-------------------------------------------------------------------

$ module show openmpi
-------------------------------------------------------------------
/global/homes/m/mamelara/spack/share/spack/modules/cray-CNL-haswell/openmpi/2.0.1-gcc-6.2.0:

module-whatis  openmpi @2.0.1
prepend-path   PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-6.2.0/openmpi-2.0.1-6vi4ni5z7l4pihbugck6rdylnzuws4ak/bin
prepend-path   CMAKE_PREFIX_PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-6.2.0/openmpi-2.0.1-6vi4ni5z7l4pihbugck6rdylnzuws4ak/
prepend-path   LD_LIBRARY_PATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-6.2.0/openmpi-2.0.1-6vi4ni5z7l4pihbugck6rdylnzuws4ak/lib
prepend-path   MANPATH /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-6.2.0/openmpi-2.0.1-6vi4ni5z7l4pihbugck6rdylnzuws4ak/man
setenv     SLURM_MPI_TYPE pmi2
setenv     OMPI_MCA_BTL_OPENIB_WARN_DEFAULT_GID_PREFIX 0
setenv     OPENMPI_ROOT /global/u2/m/mamelara/spack/opt/spack/cray-CNL-haswell/gcc-6.2.0/openmpi-2.0.1-6vi4ni5z7l4pihbugck6rdylnzuws4ak
conflict   openmpi
-------------------------------------------------------------------

Note

I wanted to show case autoloading of dependencies but to have many people downloading scipy will take a LONG time. Please feel free to try this in your spare time, the build for scipy took long even for -j32. For now, just reading the material and getting the idea will suffice.

Autoload dependencies

Spack can also generate module files that contain code to load the dependencies automatically. You can, for instance generate python modules that load their dependencies by adding the autoload directive and assigning it the value direct:

modules:
  tcl:
    hash_length: 0
    naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}'
    whitelist:
      - gcc
    blacklist:
      - '%gcc@4.8'
    all:
      conflict:
        - '${PACKAGE}'
      suffixes:
        '^openblas': openblas
        '^netlib-lapack': netlib
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']
      environment:
        set:
          '${PACKAGE}_ROOT': '${PREFIX}'
    gcc:
      environment:
        set:
          CC: gcc
          CXX: g++
          FC: gfortran
          F90: gfortran
          F77: gfortran
    openmpi:
      environment:
        set:
          SLURM_MPI_TYPE: pmi2
          OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
    netlib-scalapack:
      suffixes:
        '^openmpi': openmpi
        '^mpich': mpich
    ^python:
      autoload:  'direct'

and regenerating the module files for every package that depends on python:

$ spack module refresh -y --module-type tcl ^python
==> Regenerating tcl module files

Now the py-scipy module will be:

#%Module1.0
## Module file created by spack (https://github.com/LLNL/spack) on 2016-11-02 20:53:21.283547
##
## py-scipy@0.18.1%gcc@6.2.0 arch=linux-Ubuntu14-x86_64-e6uljfi
##
module-whatis "py-scipy @0.18.1"

proc ModulesHelp { } {
puts stderr "SciPy (pronounced "Sigh Pie") is a Scientific Library for Python. It"
puts stderr "provides many user-friendly and efficient numerical routines such as"
puts stderr "routines for numerical integration and optimization."
}

if ![ is-loaded python/2.7.12-gcc-6.2.0 ] {
    puts stderr "Autoloading python/2.7.12-gcc-6.2.0"
    module load python/2.7.12-gcc-6.2.0
}

if ![ is-loaded openblas/0.2.19-gcc-6.2.0 ] {
    puts stderr "Autoloading openblas/0.2.19-gcc-6.2.0"
    module load openblas/0.2.19-gcc-6.2.0
}

if ![ is-loaded py-numpy/1.11.1-gcc-6.2.0-openblas ] {
    puts stderr "Autoloading py-numpy/1.11.1-gcc-6.2.0-openblas"
    module load py-numpy/1.11.1-gcc-6.2.0-openblas
}

prepend-path CMAKE_PREFIX_PATH "~/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh/"
prepend-path LD_LIBRARY_PATH "~/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh/lib"
prepend-path PYTHONPATH "~/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh/lib/python2.7/site-packages"
setenv PY_SCIPY_ROOT "~/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh"
conflict py-scipy

and will contain code to autoload all the dependencies:

$ module load py-scipy
Autoloading python/2.7.12-gcc-6.2.0
Autoloading openblas/0.2.19-gcc-6.2.0
Autoloading py-numpy/1.11.1-gcc-6.2.0-openblas

Note

The rest of this tutorial focuses on lmod modules which are not present on Cori. If you are interested and have lmod installed on a different machine feel free to follow the next section.

Lua hierarchical module files

In the final part of this tutorial you will modify modules.yaml to generate Lua hierarchical module files. You will see that most of the directives used before are also valid in the lmod context.

Core/Compiler/MPI

Warning

Only LMod supports Lua hierarchical module files
For this part of the tutorial you need to be using LMod to manage your environment.

The most common hierarchy is the so called Core/Compiler/MPI. To have an idea how a hierarchy is organized you may refer to the Lmod guide. Since lmod is not enabled by default, you need to add it to the list of enabled module file generators. The other things you need to do are:

  • change the tcl tag to lmod
  • remove tcl specific directives (naming_scheme and conflict)
  • set which compilers are considered core
  • remove the mpi related suffixes (as they will be substituted by hierarchies)

After modifications the configuration file will be:

modules:
  enable::
    - lmod
  lmod:
    core_compilers:
      - 'gcc@4.8'
    hash_length: 0
    whitelist:
      - gcc
    blacklist:
      - '%gcc@4.8'
    all:
      suffixes:
        '^openblas': openblas
        '^netlib-lapack': netlib
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']
      environment:
        set:
          '${PACKAGE}_ROOT': '${PREFIX}'
    gcc:
      environment:
        set:
          CC: gcc
          CXX: g++
          FC: gfortran
          F90: gfortran
          F77: gfortran
    openmpi:
      environment:
        set:
          SLURM_MPI_TYPE: pmi2
          OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'

Note

The double colon
The double colon after enable is intentional and it serves the purpose of overriding the default list of enabled generators so that only lmod will be active (see the reference manual for a more detailed explanation of config scopes).

The directive core_compilers accepts a list of compilers : everything built using these compilers will create a module in the Core part of the hierarchy. It is common practice to put the OS provided compilers in the list and only build common utilities and other compilers in Core.

If you regenerate the module files

$ spack module refresh --module-type lmod --delete-tree -y

and update MODULEPATH to point to the Core folder, and list the available modules, you’ll see:

$ module unuse ~/spack/share/spack/modules/linux-Ubuntu14-x86_64
$ module use ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core
$ module avail

----------------------------------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core -----------------------------------------------------------------------
   gcc/6.2.0

The only module visible now is gcc. Loading that you will make visible the Compiler part of the software stack that was built with gcc/6.2.0:

$ module load gcc
$ module avail

-------------------------------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/gcc/6.2.0 ---------------------------------------------------------------------
   binutils/2.27    curl/7.50.3    hwloc/1.11.4           libtool/2.4.6    lzo/2.09       netlib-lapack/3.6.1    openssl/1.0.2j              py-scipy/0.18.1-openblas    util-macros/1.19.0
   bison/3.0.4      expat/2.2.0    libarchive/3.2.1       libxml2/2.9.4    m4/1.4.17      nettle/3.2             pkg-config/0.29.1           py-setuptools/25.2.0        xz/5.2.2
   bzip2/1.0.6      flex/2.6.0     libpciaccess/0.13.4    lz4/131          mpich/3.2      openblas/0.2.19        py-nose/1.3.7               python/2.7.12               zlib/1.2.8
   cmake/3.6.1      gmp/6.1.1      libsigsegv/2.10        lzma/4.32.7      ncurses/6.0    openmpi/2.0.1          py-numpy/1.11.1-openblas    sqlite/3.8.5

----------------------------------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core -----------------------------------------------------------------------
   gcc/6.2.0 (L)

The same holds true for the MPI part of the stack, that you can enable by loading either mpich or openmpi. The nice features of LMod will become evident once you’ll try switching among different stacks:

$ module load mpich
$ module avail

----------------------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/mpich/3.2-5n5xoep/gcc/6.2.0 ------------------------------------------------------------
   netlib-scalapack/2.0.2-netlib    netlib-scalapack/2.0.2-openblas (D)

-------------------------------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/gcc/6.2.0 ---------------------------------------------------------------------
   binutils/2.27    curl/7.50.3    hwloc/1.11.4           libtool/2.4.6    lzo/2.09           netlib-lapack/3.6.1    openssl/1.0.2j              py-scipy/0.18.1-openblas    util-macros/1.19.0
   bison/3.0.4      expat/2.2.0    libarchive/3.2.1       libxml2/2.9.4    m4/1.4.17          nettle/3.2             pkg-config/0.29.1           py-setuptools/25.2.0        xz/5.2.2
   bzip2/1.0.6      flex/2.6.0     libpciaccess/0.13.4    lz4/131          mpich/3.2   (L)    openblas/0.2.19        py-nose/1.3.7               python/2.7.12               zlib/1.2.8
   cmake/3.6.1      gmp/6.1.1      libsigsegv/2.10        lzma/4.32.7      ncurses/6.0        openmpi/2.0.1          py-numpy/1.11.1-openblas    sqlite/3.8.5

----------------------------------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core -----------------------------------------------------------------------
   gcc/6.2.0 (L)

$ module load openblas netlib-scalapack/2.0.2-openblas
$ module list

Currently Loaded Modules:
  1) gcc/6.2.0   2) mpich/3.2   3) openblas/0.2.19   4) netlib-scalapack/2.0.2-openblas

$ module load openmpi

Lmod is automatically replacing "mpich/3.2" with "openmpi/2.0.1"


Due to MODULEPATH changes the following have been reloaded:
  1) netlib-scalapack/2.0.2-openblas

This layout is already a great improvement over the usual non-hierarchical layout, but it still has an asymmetry: LAPACK providers are semantically the same as MPI providers, but they are still not part of the hierarchy. We’ll see a possible solution next.

Extend the hierarchy to other virtual providers

Warning

This is an experimental feature
Having a hierarchy deeper than Core/Compiler/MPI is an experimental feature, still not fully supported by module spider, see here. Furthermore its use with hierarchies more complex than Core/Compiler/MPI/LAPACK has not been thoroughly tested in production environments.

Spack permits you to generate Lua hierarchical module files where users can add an arbitrary list of virtual providers to the triplet Core/Compiler/MPI. A configuration file like:

modules:
  enable::
    - lmod
  lmod:
    core_compilers:
      - 'gcc@4.8'
    hierarchical_scheme:
      - lapack
    hash_length: 0
    whitelist:
      - gcc
    blacklist:
      - '%gcc@4.8'
      - readline
    all:
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH']
      environment:
        set:
          '${PACKAGE}_ROOT': '${PREFIX}'
    gcc:
      environment:
        set:
          CC: gcc
          CXX: g++
          FC: gfortran
          F90: gfortran
          F77: gfortran
    openmpi:
      environment:
        set:
          SLURM_MPI_TYPE: pmi2
          OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'

will add lapack providers to the mix. After the usual regeneration of module files:

$ module purge
$ spack module refresh --module-type lmod --delete-tree -y
==> Regenerating lmod module files

you will have something like:

$ module load gcc
$ module load openblas
$ module load openmpi
$ module avail

--------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/openblas/0.2.19-js33umc/openmpi/2.0.1-s3qbtby/gcc/6.2.0 ----------------------------------------------
   netlib-scalapack/2.0.2

-------------------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/openblas/0.2.19-js33umc/gcc/6.2.0 ---------------------------------------------------------
   py-numpy/1.11.1    py-scipy/0.18.1

-------------------------------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/gcc/6.2.0 ---------------------------------------------------------------------
   binutils/2.27    curl/7.50.3    hwloc/1.11.4           libtool/2.4.6    lzo/2.09       netlib-lapack/3.6.1        openssl/1.0.2j          python/2.7.12         zlib/1.2.8
   bison/3.0.4      expat/2.2.0    libarchive/3.2.1       libxml2/2.9.4    m4/1.4.17      nettle/3.2                 pkg-config/0.29.1       sqlite/3.8.5
   bzip2/1.0.6      flex/2.6.0     libpciaccess/0.13.4    lz4/131          mpich/3.2      openblas/0.2.19     (L)    py-nose/1.3.7           util-macros/1.19.0
   cmake/3.6.1      gmp/6.1.1      libsigsegv/2.10        lzma/4.32.7      ncurses/6.0    openmpi/2.0.1       (L)    py-setuptools/25.2.0    xz/5.2.2

----------------------------------------------------------------------- ~/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core -----------------------------------------------------------------------
   gcc/6.2.0 (L)

Now both the MPI and the LAPACK providers are handled by LMod as hierarchies:

$ module load py-numpy netlib-scalapack
$ module load mpich

Lmod is automatically replacing "openmpi/2.0.1" with "mpich/3.2"


Due to MODULEPATH changes the following have been reloaded:
  1) netlib-scalapack/2.0.2

$ module load netlib-lapack

Lmod is automatically replacing "openblas/0.2.19" with "netlib-lapack/3.6.1"


Inactive Modules:
  1) py-numpy

Due to MODULEPATH changes the following have been reloaded:
  1) netlib-scalapack/2.0.2

making the use of tags to differentiate them unnecessary. Note that because we only compiled py-numpy with openblas the module is made inactive when we switch the LAPACK provider. The user environment will now be consistent by design!