How to decide where to store my data

In this section we provide information on the recommended practice for storing LSST data at CC-IN2P3 to complement what is already presented in Data Storage and File Systems.

What to store under /pbs/throng/lsst/users

There is a directory created for your individual use under /pbs/throng/lsst/users. The name of that directory is the same as the result of executing the whoami command or the value of the $USER environment variable.

Data under /pbs/throng/lsst/users is backed-up so it may be possible to retrieve back data deleted by accident. This area is not intended for storing big datasets. There is no associated quota for your individual area but there is a storage quota set for the whole group. As a consequence, usage of your individual area impacts all the members of the lsst group.

What to store under /sps/lsst/users

There is a directory created for your individual use under /sps/lsst/users. The name of that directory is the same as the result of executing the whoami command or the value of the $USER environment variable. The initial quota of your individual directory is set to a few terabytes.

At any moment you can check how much storage you have used via the command:

spsquotalist /sps/lsst/users/$USER

This area is intended for storing data for your individual research activities, as opposed to data sets or products of interest to several or all the members of the lsst group. By default, the permissions are set so that only you can read and write data under your individual directory, other members of the lsst group have read-only access and non-members don’t have access to your individual directory at all.

You can check the permissions of your directory with the getfacl(1) command as shown below:

getfacl /sps/lsst/users/$USER

As the owner of that directory, you can also modify its permissions using the setfacl(1) command. This may be of interest, for instance, for sharing some results with a fellow user or granting them write access to your individual area.

Please take into account that this area is not backed up (see Shared group area (large datasets): /sps/lsst).

Tip

You can use the command

spsquotalist /sps/lsst

to get information about current usage of storage under /sps/lsst, including data outside /sps/lsst/users/$USER.

What to store under /sps/lsst/groups

Storage areas for members of the lsst group working on a topic of shared interest can be found under /sps/lsst/groups. Those topical group directories are created on-demand and the name of each directory is intended to explicitly convey the topic of interest of the group.

For each topical group directory a specific set of members of the lsst group are authorized to read and write. For confidentiality reasons, initial permissions of those directories are set so that all members of the lsst group have read-only access. As an owner of a topical group directory you can modify the permissions of the directory at any time: you can grant or remove access to other users as you need.

The quota of the topical group directory is set at creation time and can be adjusted as need arises, within the limits of the storage allocation for the lsst group. To retrieve the current quota of a topical group storage area use the command spsquotalist. For instance to get information about the quota of the topical group area desc use:

spsquotalist /sps/lsst/groups/desc

Czars of the lsst group responsible for storage (see Czars roles and responsibilities) have authority to create a new topical group directory, set or extend its quota and grant ownership to you. To know who within the lsst group you should contact to for matters related to topical group directories use the command:

laboinfo --group lsst --czar storage

Warning

⚠️ As a result of switching to CephFS, the tools for delegating management of the storage under /sps/lsst/groups to the czars are not fully functional yet. Czars wanting to create a new group are kindly asked to submit a ticket: see How to Get Help for details.