How to execute LSST-enabled JupyterLab notebooks
This tutorial presents step-by-step instructions to execute JupyterLab notebooks configured to use the LSST software framework using the notebooks platform provided by CC-IN2P3. It is an alternative method to execute Jupyter notebooks to the one presented in How to use stackyter to run JupyterLab notebooks. Although both methods work, we intend to develop and maintain this method in the the long term.
A JupyterLab notebook is a web application that allows you to create documents that contain code (in particular Python code which uses the LSST science pipelines), narrative text, visualizations, etc. It is also a convenient way to explore data and document your work. You can find more information in the Jupyter project web site.
By executing notebooks at CC-IN2P3 you can explore datasets of interest for LSST members using the LSST software framework, without needing to install on your personnal computer neither the LSST software nor the datasets.
In this section you will find information about the usage of notebooks specific to users of the lsst group. Additional, more generic information about CC-IN2P3 notebooks platform, including the available extensions, can be found in the section Jupyter Notebooks Platform of CC-IN2P3 Documentation.
Launch your notebook server
To launch your individual notebook server for your specific needs, point your web browser at the address
You will be welcomed with a dialog like the one below for you to provide your individual user credentials, that is, the same credentials you would use for connecting to the Login Farm.
After successful authentication, you will be able to launch your own JupyterLab notebook server.
Without further customization, you will be able to create and execute notebooks using the Python interpreter included in a recent Anaconda distribution. In addition, your notebook server will be positionned to find existing notebooks and to save new ones at your $HOME directory (see Data Storage and File Systems).
See the next section for customizing your notebook server for your specific needs.
Customize your notebook server
You can customize your notebook server in several ways. For instance, you may want to configure it to create notebooks ready to use the LSST science pipelines or to set the location where your notebooks are saved. In this section you will find how to do both.
Set the default directory for your notebooks
By default, your notebook server is configured to find and save your notebooks in your $HOME directory. But you may want to use another directory, say $HOME/notebooks.
To configure your server for this purpose you need to create a configuration file for Jupyter:
# Connect to CC-IN2P3 login farm in a new window
ssh cca.in2p3.fr
# Set the LSST environment to find the 'jupyter' command in your $PATH
source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/v19.0.0/loadLSST.bash
# Use the jupyter command to generate a default configuration file, if needed
jupyter notebook --generate-config
You will find a file named $HOME/.jupyter/jupyter_notebook_config.py which contains several configuration settings. To set the notebooks directory uncomment and modify the line as shown below, by specifying the absolute path of the desired directory:
c.NotebookApp.notebook_dir = '/pbs/home/l/lsstuser/notebooks'
Important
You need to modify the directory path so that it points to an existing directory of your choice where you have read and write permissions. The path you specify must be absolute: you cannot use neither relative path components (i.e. . or ..) nor environment variables (e.g. $HOME).
There are many other settings in the Jupyter configuration file $HOME/.jupyter/jupyter_notebook_config.py that you may want to explore for customizing your notebook server. Beware that not all of them are applicable to the CC-IN2P3 notebook service, though.
Configure your server to use the LSST software
To configure your notebook server to create notebooks ready that use the LSST science pipelines, you need to create one or more Jupyter kernel specifications according to your specific needs (see the Jupyter documentation for details).
For your convenience we have created a Jupyter kernel that configures your server to use the LSST science pipelines. You will find that kernel and its installation instructions at its source repository. This installation is a one-time process.
Once installed, the next time you will create a new notebook, the JupyterLab launcher will display a dialog similar to:
You can press on the button labeled lsst_distrib and that new notebook will be ready to use the LSST science pipelines with the latest available version of lsst_distrib. You can also modify the configuration of the lsst-jupyter-kernel to use another release: please refer to the kernel documentation.
Configure your server to use the DESC environment
You may as well want to add to your individual configuration a kernel so that your notebooks are ready to use the DESC software environment and data available at CC-IN2P3. Please follow the instructions available with the DESC Jupyter kernel contributed by @ccombet.
How your notebook server works
When you log into the service, a notebook server is launched for you using a shared pool of resources dedicated to run notebooks. All your notebooks are executed in your unique individual server. Your server will be kept alive even of you quit your web browser: if you reconnect again your notebooks will be ready for you to resume your work session.
The hardware resources allocated to your individual notebook server are limited. Currently you are provided with one server configured with up to a few GB of RAM and a few CPU cores. After an inactivity period currently configured to be 72 hours, your server will be stopped and your notebooks will be saved on persistent storage. When you reconnect again later on, you will need to execute your notebooks from the beginning.
How to stop your notebook server
To stop your individual notebook server do: menu File > Log Out.
Tip
Please note that closing the browser window does not stop your server: it continues running in background even if you are not using it. Since your server runs on a infrastructure shared with other users, it is recommended practice to explicitely stop your server when you know you won’t need it for a while.
Troubleshooting your server
When configuring your notebook server (e.g. adding or configuring a Jupyter kernel), you may find yourself in a situation where you cannot execute your notebook. In such a case, inspecting the contents of the log file of your individual server may help finding the cause of the issue: open a terminal (menu File > New > Terminal) and do:
cat /var/log/notebookserver/notebookserver.log
When Jupyter starts it attempts to open and display the notebooks you had opened in your last work session. Jupyter stores that information under the directory $HOME/.jupyter/lab/workspaces. If you find yourself in a situation where the Jupyter graphical interface is not displayed even if your notebook server was successfully launched, you may consider removing that directory via the command:
rm -rf $HOME/.jupyter/lab/workspaces