File indexing completed on 2025-01-18 09:15:26
0001 <!DOCTYPE html>
0002 <html>
0003 <head>
0004 <title>Spack: package management for HPC</title>
0005 <meta charset="utf-8">
0006 <style>
0007 @import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz);
0008 @import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic);
0009 @import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);
0010
0011 body { font-family: 'Droid Serif'; }
0012 h1, h2, h3 {
0013 font-family: 'Yanone Kaffeesatz';
0014 font-weight: normal;
0015 }
0016 .remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
0017 </style>
0018 </head>
0019 <body>
0020 <textarea id="source">
0021
0022 class: center, middle
0023
0024 # Spack: package management for HPC
0025
0026 Wouter Deconinck
0027
0028 At the JLab Scientific Computing Group
0029
0030 ---
0031
0032 # What is spack?
0033 "[Spack](https://spack.readthedocs.io/en/latest/) is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments."
0034 - "Spack was *designed for large supercomputing centers*, where many users and application teams share common installations of software on clusters with exotic architectures, using libraries that do not have a standard ABI."
0035 - "Spack is *non-destructive*: installing a new version does not break existing installations, so many configurations can coexist on the same system."
0036
0037 ---
0038
0039 # What is spack not?
0040 Spack is not a package manager in the sense of 'rpm' or 'apt', or even 'portage'.
0041 - There is *no 'spack upgrade'* that will upgrade all packages to their latest version ("Spack is non-destructive"). Spack installations will keep growing until you decide to uninstall older version or remove unused dependencies that were not explicitly installed (garbage collection).
0042 - Version differences and dependency tree differences, no matter how small, are treated as *completely separate version*. A version of root@6.20.04 that was compiled against python@3.7.7 is just as different from root@6.20.04 compiled against python@3.7.6 as it is from root@6.18.04.
0043
0044 ---
0045
0046 # Why is spack useful?
0047
0048 For regular users:
0049 - Spack has a large community which collectively supports a large number of niche scientific software packages that are relevant for the Jefferson Lab community through a single installation interface.
0050
0051 For Jefferson Lab Scientific Computing:
0052 - Spack transparently support the microarchitecture opimizations relevant to the scicomp cluster systems, including support for Intel and other compilers.
0053 - Spack plugs into the environment modules that are already in use on scicomp systems.
0054
0055 ---
0056
0057 # How can JLab use spack?
0058
0059 - Support development of spack packages to encourage adoption of locally developed software elsewhere.
0060 - Provide a (chained) stack of general purpose scientific and hall-specific software packages.
0061
0062 ---
0063
0064 # Packages and repositories
0065
0066 Packages are defined by python scripts which build the package from scratch and enforce rpath linking. All packages are install by hash. The hash includes the package version and variant information of the package and its dependencies, but e.g. modification of only the build arguments will result in the same hash. Spack hashes are not meant to indicate binary identical installation.
0067
0068 ---
0069
0070 ## Jana2
0071 ```python
0072 class Jana2(CMakePackage):
0073 """Multi-threaded HENP Event Reconstruction."""
0074
0075 homepage = "http://jeffersonlab.github.io/JANA2/"
0076 url = "http://github.com/JeffersonLab/JANA2/archive/v2.0.3.tar.gz"
0077 git = "http://github.com/JeffersonLab/JANA2.git"
0078
0079 maintainer = ["wdconinc"]
0080
0081 version('2.0.3', sha256='fd34c40e2d6660ec08aca9208999dd9c8fe17de21c144ac68b6211070463e415')
0082 version('2.0.2', sha256='161d29c2b1efbfb36ec783734b45dff178b0c6bd77a2044d5a8829ba5b389b14')
0083 version('2.0.1', sha256='1471cc9c3f396dc242f8bd5b9c8828b68c3c0b72dbd7f0cfb52a95e7e9a8cf31')
0084 version('2.0.0-alpha', sha256='4a093caad5722e9ccdab3d3f9e2234e0e34ef2f29da4e032873c8e08e51e0680')
0085
0086 variant('root',
0087 default=False,
0088 description='Use ROOT for janarate.')
0089 variant('zmq',
0090 default=False,
0091 description='Use zeroMQ for janacontrol.')
0092
0093 depends_on('cmake@3.9:', type='build')
0094 depends_on('cppzmq', when='+zmq')
0095 depends_on('root', when='+root')
0096 depends_on('xerces-c')
0097 ```
0098
0099 ---
0100
0101 ## Jana2
0102 ```python
0103 def cmake_args(self):
0104 args = []
0105 # ZeroMQ directory
0106 if '+zmq' in self.spec:
0107 args.append('-DZEROMQ_DIR=%s'
0108 % self.spec['cppzmq'].prefix)
0109 # C++ Standard
0110 if '+root' in self.spec:
0111 args.append('-DCMAKE_CXX_STANDARD=%s'
0112 % self.spec['root'].variants['cxxstd'].value)
0113 else:
0114 args.append('-DCMAKE_CXX_STANDARD=11')
0115
0116 return args
0117
0118 def setup_run_environment(self, env):
0119 import os
0120 env.append_path('JANA_PLUGIN_PATH', os.path.join(self.prefix, 'plugins'))
0121 env.set('JANA_HOME', self.prefix)
0122 ```
0123
0124 ---
0125
0126 # Packages and repositories
0127
0128 Package recipes are stored in repositories. A built-in repository has ~5k packages. Spack has a low threshold for inclusion of packages, but requires the packages to conform to some coding and style guidelines to ensure maintainability.
0129
0130 External repositories can be easily added e.g. `git clone https://github.com/spack/eic-spack && spack add repo eic-spack`.
0131
0132 ---
0133
0134 ## Spec vs concretization
0135 Spack commands use specs to indicate which package to install, e.g. `root@6.20.04 +fftw -gdml cxxstd=17 %gcc@10.0.1 ^python@3.8.3 ^/ovsuoet target=x86_64`.
0136 - Versions can specified as single or ranged (`root@:6.18.04`).
0137 - Variants can be boolean (`+fftw`, `-gdml`) or valued (`cxxstd=17`).
0138 - Compilers can be specified, but must be included in a registered list updated with `spack compiler find`.
0139 - Upstream dependencies can be specified to get around concretization errors (`^python@3.8.3`) or to explicitly reuse installed packages by hash (`^/ovsuoet`).
0140 - Target microarchitectures can be specified as more generic than e.g. `target=skylake_avx512`.
0141
0142 ---
0143
0144 ## Spec vs concretization
0145 Many collections of dependency packages, versions, and their variants can satisfy a spec. Spack installs one of them: this is the concretization of the spec. The core of spack is resolving the 'best' concretization given the full directed acylic dependency graph (DAG).
0146
0147 ---
0148
0149 # Common points of confusion
0150
0151 ## Spack does not 'do' entire system upgrades.
0152
0153 When spack installs a concretization of a spec, it links the binaries of the package and all dependencies with rpath. This means that there is no such thing as an 'upgrade' to an installed spack package: you can create new concretization of the same spec with updated package versions in the dependency tree, and some installed hashes may be identical, but as soon as one package ends up with a different hash, all other packages that depend on it will as well.
0154
0155 ---
0156
0157 ## Spack installs the same package multiple times.
0158
0159 When a user is installing multiple specs, spack makes only minimal attempts to reuse already installed dependencies (the 'best' concretization does not make installed version any 'better' than newer non-installed version). This means that users often complain about having 'the same' package installed multiple times. In the eyes of spack, the packages are different since the dependencies are different. This is particularly annoying for 'large' packages such as llvm, root, geant4, etc.
0160
0161 This can be avoided by ensuring that spack tries to find one single concretization that simultaneously satisfies all specs. Spack environments help with this.
0162
0163 ---
0164
0165 ## Spack doesn't unload when I tell it to.
0166
0167 When a user loads a concretization with `spack load root`, this will load environment variables (e.g. `$ROOTSYS`), and set `$PATH`/`$LD_LIBRARY_PATH` for the entire dependency tree. When the user unloads with `spack unload root`, this only unloads the environment variables associate with root, but not the other dependencies (`spack unload -a` unloads everything).
0168
0169 Users should be recommended to use spack environments which they may be familiar with from conda, pipenv, or virtualenv.
0170
0171 ---
0172
0173 ## Spack doesn't use all the OS packages I already have.
0174
0175 Spack can find on its own a limited number of externally installed packages, e.g. automake, cmake (`spack external find`), and can be told where other system package are installed. While this reduces disk usage and may result in having depdendent packages on your system sooner, it makes them less portable and subject to breakage when the system package is upgraded. A concretization for `arch=linux-rhel7-x86_64` that does not rely on external packages will work on any `os=linux` and `target=x86_64` system for as long as those systems are around.
0176
0177 ---
0178
0179 ## Types of bugs in spack
0180
0181 There are three types of bugs in spack:
0182
0183 - build errors: a package fails to build on some system.
0184 - concretization errors: spack fails to find a concretization even though one exists.
0185 - internal spack errors: spack does something wrong outside of this.
0186
0187 On github, spack uses issues and pull requests also for new packages, new versions, etc.
0188
0189 ---
0190
0191 # Environments
0192
0193 Spack environments are used to group together a set of specs for the purpose of building, rebuilding and deploying in a coherent fashion.
0194 - Environments separate the steps of (a) choosing what to install, (b) concretizing, and (c) installing.
0195 - Environments that are built as a whole can be loaded as a whole into the user environment.
0196 - Environment can be built to maintain a filesystem view of its packages.
0197
0198 ### EIC
0199
0200 ```yaml
0201 spack:
0202 specs:
0203 - escalate
0204 - eicroot
0205 - eictoymodel
0206 view: true
0207 ```
0208
0209 ### Escalate
0210
0211 ---
0212
0213 ```yaml
0214 spack:
0215 specs:
0216 - cmake
0217 - boost
0218 - python
0219 - root@6.20.04 +vmc +pythia6 +pythia8 +root7 cxxstd=17
0220 - geant4 +opengl +qt cxxstd=17 ^xerces-c cxxstd=17 ^clhep cxxstd=17
0221 - eigen
0222 - vgm
0223 - genfit
0224 - hepmc
0225 - hepmc3 +interfaces +python +rootio
0226 - acts
0227 - delphes
0228 - fastjet
0229 - lhapdf
0230 - pythia8
0231 - dire
0232 - cernlib
0233 - lhapdf5
0234 - pythia6 +root
0235 - eic-smear +pythia6
0236 - ejana +acts +genfit
0237 - g4e +compat
0238 - jana2 +root
0239 packages:
0240 all:
0241 compiler: [gcc@9.2.0]
0242 target: [x86_64]
0243 concretization: together
0244 view: true
0245 ```
0246
0247 ---
0248
0249 # Environments
0250
0251 - Environments separate the steps of (a) choosing what to install, (b) concretizing, and (c) installing.
0252
0253 ```sh
0254 spack env create escalate environments/escalate/spack.yaml
0255 spack env escalate activate
0256 spack concretize
0257 spack install
0258 spack env deactivate
0259 ```
0260
0261 ---
0262
0263 ## Environment containers
0264
0265 Spack can easily create containers from environments `spack.yaml` files with `spack containerize`.
0266
0267 ```docker
0268 # Build stage with Spack pre-installed and ready to be used
0269 FROM spack/ubuntu-bionic:latest as builder
0270
0271 # What we want to install and how we want to install it
0272 # is specified in a manifest file (spack.yaml)
0273 RUN mkdir /opt/spack-environment \
0274 && (echo "spack:" \
0275 && echo " concretization: together" \
0276 && echo " view: /opt/view" \
0277 && echo " packages:" \
0278 && echo " all:" \
0279 && echo " compiler:" \
0280 && echo " - gcc@9.2.0" \
0281 && echo " target:" \
0282 && echo " - x86_64" \
0283 && echo " specs:" \
0284 && echo " - eicroot" \
0285 && echo " config:" \
0286 && echo " install_tree: /opt/software") > /opt/spack-environment/spack.yaml
0287
0288 # Install the software, remove unecessary deps
0289 RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y
0290 ```
0291
0292 ---
0293
0294 ```docker
0295 # Strip all the binaries
0296 RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
0297 xargs file -i | \
0298 grep 'charset=binary' | \
0299 grep 'x-executable\|x-archive\|x-sharedlib' | \
0300 awk -F: '{print $1}' | xargs strip -s
0301
0302 # Modifications to the environment that are necessary to run
0303 RUN cd /opt/spack-environment && \
0304 spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
0305
0306
0307 # Bare OS image to run the installed executables
0308 FROM ubuntu:18.04
0309
0310 COPY --from=builder /opt/spack-environment /opt/spack-environment
0311 COPY --from=builder /opt/software /opt/software
0312 COPY --from=builder /opt/view /opt/view
0313 COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
0314
0315 ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
0316 ```
0317
0318 ---
0319
0320 # Personal installation
0321
0322 A user can install spack in their home directory:
0323 ```
0324 git clone https://github.com/spack/spack
0325 export SPACK_ROOT=$PWD/spack
0326 source $SPACK_ROOT/setup-env.sh
0327 ```
0328 In light of large dependency trees, it makes more sense for users to install in /opt or /usr/local. The package installation directory can be moved out of the spack repository directory as well.
0329
0330 Recommendations:
0331 - Updating spack and its builtin repository frequently by tracking the develop branch and git pulling frequently will result a lot of dependency tree recompilations. Therefore, track the latest release branch (currently releases/v0.15) for a more stable experience.
0332
0333 ---
0334
0335 ## Build caches
0336
0337 Institutions can set up mirrors for binary packages. These are (ideally) relocatable binaries compiled for specific microarchitectures. At Jefferson Lab we have `/scigroup/spack/` which is accessible at `https://spack.jlab.org/`.
0338
0339 ---
0340
0341 ## Contributing to spack
0342
0343 New packages, new versions, or package bugfixes can be contributed to spack by submitting github pull requests. PEP8 python coding styles are enforced. Therefore, if you anticipate doing this frequently, fork the spack/spack repository into organization/spack or user/spack, set up a few different origins, and use your forked version:
0344 ```
0345 origin git@github.com:eic/spack.git
0346 upstream git@github.com:spack/spack.git
0347 wdconinc git@github.com:wdconinc/spack.git
0348 ```
0349 with local master tracking origin/master, which gets upstream/releases merged into it periodically. Development branches are branched off upstream/develop as origin/new-package. Pull requests are typically merged into upstream/develop in a few days, if no changes are requested.
0350
0351 ---
0352
0353 # Systemwide installation
0354
0355 This avoids the need for users to install spack. Only loading the systemwide spack installation into the user environment is needed, e.g. with `$SPACK_ROOT=/opt/spack`, `/site/spack`, or `/cvmfs/eic.opensciencegrid.org/packages`, users only need
0356 ```sh
0357 source $SPACK_ROOT/share/spack/setup-env.sh
0358 ```
0359 or
0360 ```sh
0361 source $SPACK_ROOT/share/spack/setup-env.csh
0362 ```
0363
0364 This exposes all installed packages to the user, but `spack install` will fail when no write permissions (this may be something that can be resolved).
0365
0366 ## Exposing environments to users
0367
0368 Environment loading scripts can be created with `spack env loads`.
0369
0370 ---
0371
0372 ## CVMFS network file system
0373
0374 Organization of packages:
0375 - `install_path_scheme: "${PACKAGE}/${VERSION}/${ARCHITECTURE}-${COMPILERNAME}-${COMPILERVER}-${HASH}"`
0376
0377 Where to put `.cvmfscatalog`?
0378 - Latency of pulling small file lists vs. pulling large file lists that are not going to be used.
0379 - At root of install_tree.
0380 - At root of every package.
0381 - At root of builtin repository.
0382
0383 ---
0384
0385 ## Other useful settings
0386
0387 Build stage location:
0388 ```yaml
0389 build_stage:
0390 - $tempdir/$user/spack-stage
0391 - /scratch/$user/spack-stage
0392 ```
0393
0394 ---
0395
0396 ## Things I have not had a chance to look into
0397
0398 - Interaction with modules on ifarm: `module avail` shows installed spack packages, but `use` fails to see them. Not sure if only environments can be exposed.
0399 - Chaining spack installs: extending a common site-wide install tree with collaboration-wide install trees.
0400
0401 ---
0402
0403 # Additional resources
0404
0405 - Spack readthedocs, https://spack.readthedocs.io
0406 - Spack on slack, https://spackpm.herokuapp.com
0407 - Spack mailing list, https://groups.google.com/g/spack
0408
0409 </textarea>
0410 <script src="https://remarkjs.com/downloads/remark-latest.min.js">
0411 </script>
0412 <script>
0413 var slideshow = remark.create();
0414 </script>
0415 </body>
0416 </html>
