Developer Guide

This guide is intended for people who want to work on Spack itself. If you just want to develop packages, see the Packaging Guide.

It is assumed that you’ve read the Basic Usage and Packaging Guide sections, and that you’re familiar with the concepts discussed there. If you’re not, we recommend reading those first.

Overview

Spack is designed with three separate roles in mind:

1. Users, who need to install software without knowing all the details about how it is built.
2. Packagers who know how a particular software package is built and encode this information in package files.
3. Developers who work on Spack, add new features, and try to make the jobs of packagers and users easier.

Users could be end users installing software in their home directory, or administrators installing software to a shared directory on a shared machine. Packagers could be administrators who want to automate software builds, or application developers who want to make their software more accessible to users.

As you might expect, there are many types of users with different levels of sophistication, and Spack is designed to accommodate both simple and complex use cases for packages. A user who only knows that he needs a certain package should be able to type something simple, like spack install <package name>, and get the package that he wants. If a user wants to ask for a specific version, use particular compilers, or build several versions with different configurations, then that should be possible with a minimal amount of additional specification.

This gets us to the two key concepts in Spack’s software design:

1. Specs: expressions for describing builds of software, and
2. Packages: Python modules that build software according to a spec.

A package is a template for building particular software, and a spec as a descriptor for one or more instances of that template. Users express the configuration they want using a spec, and a package turns the spec into a complete build.

The obvious difficulty with this design is that users under-specify what they want. To build a software package, the package object needs a complete specification. In Spack, if a spec describes only one instance of a package, then we say it is concrete. If a spec could describes many instances, (i.e. it is under-specified in one way or another), then we say it is abstract.

Spack’s job is to take an abstract spec from the user, find a concrete spec that satisfies the constraints, and hand the task of building the software off to the package object. The rest of this document describes all the pieces that come together to make that happen.

Directory Structure

So that you can familiarize yourself with the project, we’ll start with a high level view of Spack’s directory structure:

spack/                  <- installation root
   bin/
      spack             <- main spack executable

   etc/
      spack/            <- Spack config files.
                           Can be overridden by files in ~/.spack.

   var/
      spack/            <- build & stage directories
          repos/            <- contains package repositories
             builtin/       <- pkg repository that comes with Spack
                repo.yaml   <- descriptor for the builtin repository
                packages/   <- directories under here contain packages
          cache/        <- saves resources downloaded during installs

   opt/
      spack/            <- packages are installed here

   lib/
      spack/
         docs/          <- source for this documentation
         env/           <- compiler wrappers for build environment

         external/      <- external libs included in Spack distro
         llnl/          <- some general-use libraries

         spack/         <- spack module; contains Python code
            cmd/        <- each file in here is a spack subcommand
            compilers/  <- compiler description files
            test/       <- unit test modules
            util/       <- common code

Spack is designed so that it could live within a standard UNIX directory hierarchy, so lib, var, and opt all contain a spack subdirectory in case Spack is installed alongside other software. Most of the interesting parts of Spack live in lib/spack.

Spack has one directory layout and there is no install process. Most Python programs don’t look like this (they use distutils, setup.py, etc.) but we wanted to make Spack very easy to use. The simple layout spares users from the need to install Spack into a Python environment. Many users don’t have write access to a Python installation, and installing an entire new instance of Python to bootstrap Spack would be very complicated. Users should not have to install install a big, complicated package to use the thing that’s supposed to spare them from the details of big, complicated packages. The end result is that Spack works out of the box: clone it and add bin to your PATH and you’re ready to go.

Code Structure

This section gives an overview of the various Python modules in Spack, grouped by functionality.

Package-related modules

spack.package

Contains the Package class, which is the superclass for all packages in Spack. Methods on Package implement all phases of the package lifecycle and manage the build process.

spack.packages

Contains all of the packages in Spack and methods for managing them. Functions like packages.get and class_name_for_package_name handle mapping package module names to class names and dynamically instantiating packages by name from module files.

spack.relations

Relations are relationships between packages, like depends_on and provides. See Dependencies and Virtual dependencies.

spack.multimethod

Implementation of the @when decorator, which allows multimethods in packages.

Spec-related modules

spack.spec

Contains Spec and SpecParser. Also implements most of the logic for normalization and concretization of specs.

spack.parse

Contains some base classes for implementing simple recursive descent parsers: Parser and Lexer. Used by SpecParser.

spack.concretize

Contains DefaultConcretizer implementation, which allows site administrators to change Spack’s Concretization Policies.

spack.version

Implements a simple Version class with simple comparison semantics. Also implements VersionRange and VersionList. All three are comparable with each other and offer union and intersection operations. Spack uses these classes to compare versions and to manage version constraints on specs. Comparison semantics are similar to the LooseVersion class in distutils and to the way RPM compares version strings.

spack.compilers

Submodules contains descriptors for all valid compilers in Spack. This is used by the build system to set up the build environment.

Warning

Not yet implemented. Currently has two compiler descriptions, but compilers aren’t fully integrated with the build process yet.

spack.architecture

architecture.sys_type is used to determine the host architecture while building.

Warning

Not yet implemented. Should eventually have architecture descriptions for cross-compiling.

Build environment

spack.stage

Handles creating temporary directories for builds.

spack.compilation

This contains utility functions used by the compiler wrapper script, cc.

spack.directory_layout

Classes that control the way an installation directory is laid out. Create more implementations of this to change the hierarchy and naming scheme in $spack_prefix/opt

Spack Subcommands

spack.cmd

Each module in this package implements a Spack subcommand. See writing commands for details.

Unit tests

spack.test

Implements Spack’s test suite. Add a module and put its name in the test suite in __init__.py to add more unit tests.

spack.test.mock_packages

This is a fake package hierarchy used to mock up packages for Spack’s test suite.

Other Modules

spack.url

URL parsing, for deducing names and versions of packages from tarball URLs.

spack.error

SpackError, the base class for Spack’s exception hierarchy.

llnl.util.tty

Basic output functions for all of the messages Spack writes to the terminal.

llnl.util.tty.color

Implements a color formatting syntax used by spack.tty.

llnl.util

In this package are a number of utility modules for the rest of Spack.

Spec objects

Package objects

Most spack commands look something like this:

1. Parse an abstract spec (or specs) from the command line,
2. Normalize the spec based on information in package files,
3. Concretize the spec according to some customizable policies,
4. Instantiate a package based on the spec, and
5. Call methods (e.g., install()) on the package object.

The information in Package files is used at all stages in this process.

Conceptually, packages are overloaded. They contain:

Stage objects

Writing commands

Unit tests

Unit testing

Developer commands

spack doc

spack test

Profiling

Spack has some limited built-in support for profiling, and can report statistics using standard Python timing tools. To use this feature, supply --profile to Spack on the command line, before any subcommands.

spack --profile

spack --profile output looks like this:

$ spack --profile graph dyninst
==> Error: No available compiler version matches 'None' on operating_system ubuntu14for target x86_64
...

The bottom of the output shows the top most time consuming functions, slowest on top. The profiling support is from Python’s built-in tool, cProfile.