How to use community butler repos

FrDF maintains butler repos of interest for LSST France members for specific science studies or official Rubin releases, that we refer to as community butler repositories.

This tutorial outlines the steps involved in setting up your working environment to utilize those repositories.

Please note that you will get access to those repositories only if you hold access rights to Rubin data.

Important

⚠️ The information below assumes that you have configured your environment as described in Configuring your working environment.

Step 1: register to the Indigo IAM LSST France instance

To access community butler repos, you must first register as a user at the LSST France’s dedicated Indigo IAM instance. You can find details on how to proceed in How to create your Indigo-IAM account.

Registering to Indigo IAM is a one time process.

Step 2: configure the butler repository index

The Rubin data butler inspects an index to retrieve details on the configured butler repositories. Using that index you can refer to a butler repo via its alias, like main or dp1, in addition to its URL.

The butler index is pointed to by the environment variable DAF_BUTLER_REPOSITORY_INDEX if it is initialized. You can set up your own index or use the one prepared for you and automatically enabled if you followed up the instructions in Configuring your working environment.

To know what repos are available and their aliases inspect the contents of the file pointed to by DAF_BUTLER_REPOSITORY_INDEX:

$ cat $DAF_BUTLER_REPOSITORY_INDEX
dp1: "davs://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/groups/lsstfr/repos/dp1/butler.yaml"
main: "davs://ccdavrubinint.in2p3.fr:2880/pnfs/in2p3.fr/lsst/groups/lsstfr/repos/main/butler.yaml"

Step 3: configure the credentials for the butler registry database

The butler needs access to each repository registry database. You need to configure the details by connecting to the Login Farm and add the required configuration details to the file located in ${HOME}/.lsst/db-auth.yaml.

- url: "postgresql://ccpglsstdev.in2p3.fr:6553/lsstfr"
  username: "lsstfr"
  password: "secret"

Note that there is a secret in this file. The one presented above is not the actual password: please ask for it via instant message and we will provide it via a secure channel.

For security reasons, the permissions of this file must be set correctly so that only you can read it. Execute the command below to set them:

$ chmod go-rwx ${HOME}/.lsst/db-auth.yaml

Configuring these credentials is a one time process.

Step 4: acquire credentials for the butler datastore

Credentials to access each butler repository datastore, which at FrDF is managed by dCache, are obtained via the command lsstfr login. Those credentials are called access tokens. See Step 3: get an access token for details on how to get your token.

Please note that those access tokens have a limited validity period so you may need to obtain a fresh token if it already expired.

Verify you can access the repos

Once your setup is complete, you can access for instance the main butler repo using the butler commands available in LSST Science Pipelines.

Activate a recent release of the the LSST Science Pipelines:

$ # Works with Bash and Z shell
$ source /cvmfs/sw.lsst.eu/almalinux-x86_64/lsst_distrib/w_2026_16/loadLSST.sh
$ setup lsst_distrib

Inpect a repository via the butler command, in this case the repo main:

$ butler query-dataset-types --verbose main
name                                         dimensions                                      storage class
------ ------------------------------------------------------------------------------------- -------------
raw    ['band', 'instrument', 'day_obs', 'detector', 'group', 'physical_filter', 'exposure'] Exposure
skyMap ['skymap']                                                                            SkyMap
...


$ butler query-collections --chains TREE main
Name            Type
--------------- ----
LSSTCam/raw/all RUN
skymaps         RUN
...

You can also inspect the repository using the Python API:

>>> from lsst.daf.butler import Butler
>>> butler = Butler("main")
>>> butler.registry.queryDatasetTypes()
[DatasetType('skyMap', {skymap}, SkyMap), DatasetType('raw', {band, instrument, day_obs, detector, group, physical_filter, exposure}, Exposure)]
>>> butler.collections.query("*")
['skymaps', 'LSSTCam/raw/all']