How to use the LSST science pipelines¶
In this tutorial you will find information on how to use the LSST science pipelines, colloquially known as the stack, at CC-IN2P3.
There you will find a subdirectory for each supported platform (e.g.
darwin-x86_64) and for each platform there is a LSST package distribution, for instance
lsst_sims. To execute the LSST software on CC-IN2P3 hosts, you may want to use the software available under
/cvmfs/sw.lsst.eu/linux-x86_64 which is the one built for execution on nodes running Linux.
This is the name of the distribution of the LSST software which contains the science pipelines.
In the directory
/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distribyou will find several releases, both stable and weekly. Stable releases start by
v19.0.0) and weekly releases start by
w_2020_18). All the available releases are configured for Python 3 only.
Each of the installed releases is independent, self-consistent and almost self-contained: the Python interpreter, the set of Python packages and shared libraries each release depends on are all contained in its installation directory. However, tools such as the C++ runtime library or the
gitcommand are included only for releases
For your convenience, some Python packages not required by the LSST software framework are also included in each installed release. They are installed via the
condacommand, which means that the installed versions of those packages are the ones available via the mechanisms used by
conda. For instance, packages such as Jupyter and its dependencies are included. See Quick start guide for getting the list of installed Python packages for a given release.
The LSST software is built and deployed following the official instructions on a host running the CentOS 7 operating system and using the version of the C/C++ compiler recommended by the project for each particular version. It may change from version to version, though.
Only a limited set of weekly releases is available at any given moment, typically those published by the project over the last 3 months. This means that older weekly releases are removed without prior notice. You are therefore strongly encouraged to use a recent release for your own needs. To know what modifications a given release includes, you can look the Rubin Science Pipelines Changelog.
This is the distribution which includes the software packages for producing simulated LSST images. The available releases of this distribution can be found under the directory
As is the case for
lsst_distrib, each installed release of
lsst_simsis independent and as self-contained as possible. The naming scheme for each release of
lsst_simsfollows the project’s naming convention and is therefore different than for
lsst_distrib, but it should be self explanatory.
If you are interested in using the LSST software environment from within Jupyter notebooks, please refer to How to execute LSST-enabled JupyterLab notebooks.
On your personal computer or virtual machine you can use exactly the same LSST software packages than you use when working on CC-IN2P3 nodes, which may be useful for reproducibility purposes. You can find more details on how to configure your personal computer to do so at sw.lsst.eu.
What releases are available¶
At any moment you can know what stable or weekly releases are available. Connect to the Login Farm and do:
tree -L 2 /cvmfs/sw.lsst.eu/linux-x86_64
Quick start guide¶
To use a given release of the software distribtion
lsst_distrib, you must first log in to a host in the Login Farm:
Since all the hosts in the login farm run Linux, assuming you want to use
w_2020_18 and your shell is
This will bootstrap your environment to use the specified release. As a result of executing this command, some environmental variables are extended or initialized, such as
To activate the whole set of the packages of this particular release, that is, making the commands available in you
PATH and the Python packages in your
Alternatively, you can select a specific set of packages to activate. The LSST software uses EUPS for managing the set of packages which are part of a given release. EUPS allows you to select the packages you need to use in a work session. For instance, to use the command line tasks for processing CFHT images, instead of activating all the packages in
lsst_distrib you can activate a subset of them:
$ setup obs_cfht $ setup pipe_tasks
After these steps, your working environment is modified so that you can use the command line tasks (e.g.
processCcd.py, etc.) and import Python modules in your own programs (e.g.
If you need to work with a different release of the stack, say
w_2020_20, you must create a new terminal session and bootstrap your environment with the desired release. For instance:
# In a new terminal session $ source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2020_20/loadLSST.bash $ setup obs_cfht $ setup pipe_tasks $ processCcd.py --help
To retrieve the list of Python packages included in release
$ source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2020_20/loadLSST.bash $ conda list
lsst_sims is analogous as the usage of
To get a deeper understanding of what you can do with the LSST software pipelines the resources below may help:
Understanding building blocks: numpy, matplotlib and astropy by N. Chotard
Official documentation of the LSST Science Pipelines
Set of notebooks developed and maintained by the LSST science collaborations stack club which showcase some of the capabilities of the LSST software
LSST stack by D. Boutigny
Using the stack with Python by J. Bosch. Tutorial given at the LSST All Hands Meeting, Aug 2017