How to decide where to store my data

In this section we provide information on the recommended practice for storing LSST data at CC-IN2P3 to complement what is already presented in Data Storage and File Systems.

What to store under /pbs/throng/lsst/users

There is a directory created for your individual use under /pbs/throng/lsst/users. The name of that directory is the same as the result of executing the whoami command or the value of the $USER environment variable.

Data under /pbs/throng/lsst/users is backed-up so it may be possible to retrieve back data deleted by accident. This area is not intended for storing big datasets. There is no associated quota for your individual area but there is a storage quota set for the whole group. As a consequence, usage of your individual area impacts all the members of the lsst group.

What to store under /sps/lsst/users

There is a directory created for your individual use under /sps/lsst/users. The name of that directory is the same as the result of executing the whoami command or the value of the $USER environment variable. The initial quota of your individual directory is set to a few terabytes.

At any moment you can check how much storage you have used via the command:

spsquota list /sps/lsst/users/$USER

This area is intended for storing data for your individual research activities, as opposed to data sets or products of interest to several or all the members of the lsst group. By default, the permissions are set so that only you can read and write data under your individual directory, other members of the lsst group can only read that and non-members don’t have access to that directory at all.

You can check the permissions of your directory with the getfacl(1) command as shown below:

getfacl /sps/lsst/users/$USER

As the owner of that directory, you can also modify its permissions using the setfacl(1) command. This may be of interest, for instance, for sharing some results with a fellow user or allowing her to write into your individual area.

Please take into account that this area is not backed up (see Shared group area (large datasets): /sps/lsst).

Tip

You can use the command

spsquota list /sps/lsst

to get information about your current usage of storage under /sps/lsst, including data outside /sps/lsst/users/$USER.

More information about spsquota is available here.

What to store under /sps/lsst/groups

Storage areas for members of the lsst group working on a topic of shared interest can be found under /sps/lsst/groups. Those areas are created on-demand and the name of each directory is intended to explicitly convey the topic of the group.

For each topical group storage area a set of members of the lsst group are authorized to read and write. For confidentiality reasons, initial permissions of those areas are set so that non-members of the lsst group have no access at all. Permissions can be modified by the owner of the topical group directory: they will be able to grant access to other users or remove access to them.

The quota of the topical group directory is set at creation time according to the needs and can be adjusted as need arises. Czars of the lsst group reponsible for storage (see Czars roles and responsibilities) have permissions to create a new topical group storage area and set or extend its quota.

To know who within lsst you should contact to for matters related to topical group areas do:

laboinfo -g lsst -p afs