Working Environment at CC-IN2P3¶
Here you will find an overview of the CC-IN2P3 working environment, relevant for your activities as a member of the LSST-France community.
CC-IN2P3 computing environment is made up of several components that you can use:
- a login farm, where you can connect and do interactive work
- a batch farm, where you can submit jobs for asynchronous execution in CPU- or GPU-equipped compute nodes
- several data storage services, where you can access data and software, and store your own
Below you can find more details on each of those components.
How to Get Help¶
You can get help by asking other LSST-France members via instant messaging (see Instant messaging).
In addition, for questions related to CC-IN2P3 operations, passwords, account support, group membership, storage quotas, etc. you can contact the help desk at:
To use any of the computing resources allocated to LSST-France at CC-IN2P3 you need an individual account.
If you don’t have an account yet, please follow these instructions to apply for one.
If you already have an account and want it to be a member of the group
lsst, one of the LSST-France representatives must formally approve your membership. You can trigger the process by creating a ticket (see How to Get Help for details).
More information on the administrative organization of CC-IN2P3 user accounts in laboratories and groups can be found in the generic end-user documentation, where you can also find details on the roles and responsibilities of those representatives.
⚠️ As a user of CC-IN2P3 computing center services you are expected to comply with its charters. Please make sure you have read them and understand your rights and responsibilities before using those services.
You can get a live overview of the status of CC-IN2P3 computing services at:
There are 4 scheduled maintenance periods per year (about one per quarter), each one of them typically lasting 24 hours or less. The calendar of those periods is established one year in advance. To be informed when those periods are scheduled, configure your ICS-compatible calendar application to subscribe to the feed:
During each maintenance period, a potentially different subset of the computing services are impacted. To get informed about the details of each outage, scheduled or otherwise, you may want to subscribe to the very low-traffic mailing list
USERS-CC-L by sending an e-mail to email@example.com with the body of your message:
subscribe USERS-CC-L Your_Given_Name Your_Last_Name
You won’t be able to post to this list but will receive all announcements sent by the people in charge of CC-IN2P3 support and operations.
(See also the Monitoring and Dashboards section.)
The login farm is composed of a set of computers where you can connect via
ssh using your individual user credentials. All the computers in this farm can access the data in your
$HOME or in any other storage space (see Data Storage and File Systems).
You can use the computers in the login farm to perform interactive work (e.g. editing files, writing documents, compiling code, etc.) and to submit jobs to the batch farm (see Batch Farm).
LSST-France members use the computers in the login farm which run the CentOS 7 distribution of the Linux operating system. To connect to a computer in the login farm, use the generic host name
cca.in2p3.fr (technically, it is a DNS alias), for instance:
Your connection request made through that generic host name will be dynamically directed to a concrete host (say
cca001.in2p3.fr) depending on several parameters such as the number of connected users or the load of each host in the farm, etc. at the time of your request. Therefore, you are encouraged to use the generic host name instead of the name of a specific host.
See How to customize your SSH client to get detailed information on how to configure your SSH client for secure, passwordless connexion to the login farm.
You may also be interested in learning about How to execute LSST-enabled JupyterLab notebooks.
The batch farm is composed of several hundred hosts devoted to compute- and data-intensive tasks executed mostly asynchronously.
CC-IN2P3 uses Univa GridEngine batch system. Some of the GridEngine commands you may find useful are:
qsub: to submit a batch job to Univa Grid Engine
qlogin: to submit submit an interactive login session
qstat: to show the status of jobs and queues
qacct: to report and account for Univa Grid Engine usage
qdel: to delete jobs from queues
qhold: to hold back jobs from execution
qrls: to release jobs from previous hold states
qalter: to modify a pending or running batch job
qresub: to submit a copy of an existing job
qmod: to modify a queue, running job or job class
qquota: to show current usage of resource quotas
qhost: to show the status of hosts, queues, jobs
To retrieve specific information about each one of them, connect to the Login Farm and then do:
The batch farm is configured using execution queues each with its own characteristics. You may also want to browse the hardware configuration and operating system of the compute nodes in the farm.
Unfortunately, because unfathomable restrictions imposed by Univa, we are not allowed to make the end user documentation available via a public web page (!). Therefore, to access the documentation of the currently deployed version of Univa GridEngine, you need first to connect to the login farm and then look for the available documents in the directory
Finally, to know which version of GridEngine is currently in production, do:
Data Storage and File Systems¶
As a user of CC-IN2P3, you can use several storage areas for your data. Selecting one of those areas for your needs depend on several criteria such as the intended use of the data, their lifetime, desired accessibility, etc.
See also How to decide where to store my data.
Your account is configured with a home directory pointed to by the environment variable
$HOME. That storage space is managed by NFS and it has a quota of a few tens of gigabytes.
You are strongly encouraged to not hardcode the path to your home directory and instead use
$HOME (or, alternatively
~) in your scripts, including your batch jobs and shell profiles.
Your home directory is accessible in read/write mode from all the hosts in both the Login Farm and in the Batch Farm. In other words, a file created during an interactive session in your home directory will be visible by your jobs when executing in one compute node, using the same file path.
Files you store under your home directory are backed-up so may be retrieved in case of accidental deletion. There is no purge policy associated to your home directory but the storage quota is somehow low.
To manage the access rights to your files and directories with fine granularity, you can use the commands
nfs4_setfacl(1) which allow you to get and set access control entries, respectively. An introduction to how NFS access control lists work can be found using the command
man 5 nfs4_acl
Additional details on how to set access control lists to your data are available in the Manage ACLs section of the user documentation.
Intended use: source code, executables, configuration files, documents, etc. Optimized for small- to medium-sized files. It is not intended for storing the output of your application runs.
Interactive working area:
This storage area is useful for storing data while working in interactive sessions in the machines in the Login Farm. It is visible only from the hosts in the login farm: your batch job won’t be able to access files stored in this area.
Recommended practice is to create a subdirectory for your own needs named after your login name (i.e. the output of the
whoami command). Please note that this area is not intended to permanently store data: files in this area are regularly purged based on usage criteria and space availability but typically last beyond a single interactive work session.
Batch job working area:
Jobs submitted for execution in the Batch Farm can use a temporary storage area. This area is local to the compute node where your job executes (i.e. it is allocated in the compute node’s local disks) and is volatile: the files and directories you create there in will dissappear when your job finishes its execution.
Recommended practice is to use this compute node’s local storage area for storing files produced by your job. As your job’s execution progresses and before it finishes, you can copy the data you want to keep to other storage areas intended for permanent storage. In a similar way, if your job needs to repeatedly read some files, you may want to consider copying those files into the temporary storage of your job and process those files locally.
The path to this area is unique for each job and is prepared by the batch system. Recommended practice is to access it via the environmental variable
CC-IN2P3 uses HPSS for data archival purposes. Data on this area are physically stored on magnetic tape cartridges in an automated tape library. You can find more information in the Mass storage section of the CC-IN2P3 User Documentation.
Summary: overview of available storage areas¶
|Storage area||Visibility||File system type||Backed-up||Quota||Purge policy|
||login and batch farms||NFS||yes||a few 10s GB||No|
||login and batch farms||NFS||yes||a few TB||No|
||login and batch farms||GPFS and CephFS||no||hundreds of TB||No|
||login farm only||NFS||no||N/A||Yes|
||compute node only||Local file system (typically XFS)||no||a few 10s GB per job||Yes|
See also the Monitoring and Dashboards section for getting additional information about those areas.
Software deployed by CC-IN2P3 staff¶
General purpose software tools such as file editors, typsetting tools, compilers, interpreters, source code management tools, etc. are available for you to use on hosts in both the Login Farm and the Batch Farm. They are installed and maintained by CC-IN2P3 staff.
You will find more information on what packages are available and how to activate a specific version of one them in the software section of the CC-IN2P3 user documentation.
Software deployed by members of the group¶
Software deployments under the responsibility of members of the
lsst group are available under
There you will find a subdirectory for each available software package or software environment. Generally speaking, the owner of each directory is the maintainer of the deployment unless stated otherwise in the
README file inside each directory.
Your contribution is welcome if you are willing to install and commit to maintain software packages other members of the group can rely on. Before proceeding, you are encouraged to discuss your intentions with other members and ask for guidance via Instant messaging.
DESC software environment¶
The LSST Dark Energy Science Collaboration (DESC) sofware environment is installed under
See How to activate the DESC software environment for details on how to use it.
Monitoring and Dashboards¶
Several tools are provided for monitoring your individual activity and the activity of the members of LSST at CC-IN2P3.
You can get an overall view of your individual activity by visiting
You will be presented a login page like the one below:
Once authenticated with your individual credentials (the same user name and password that you use for logging in to the Login Farm) you will be presented with information about your own batch jobs and those of your group, the storage capacity you are using, the computing ressources asked by the groups you belong to, how to contact the help desk and pointers to several other services provided by CC-IN2P3 that you as a member of LSST could use (e.g. Indico, gitlab, Atrium, etc.)
You can have an overall view of the computing activity at CC-IN2P3 induced by members of LSST by visiting
You will be presented a dialog similar to:
Type in your individual credentials, that is, the same user name and password that you use for logging in to the Login Farm.
Once authenticated, you will be presented an interactive dashboard like the one below:
You will be able to drill down and get specific real-time information about the activities related to storage, batch processing, data exchange and the catalog database.
In this section you can find ways to get information about the usage of storage areas used by LSST.
The table below presents the locations where you can find information of storage used per individual for the storage areas under
This information is refreshed daily: see at the top right corner of the target page for information on when it was last updated.
The images below present the storage allocated vs the storage used for several ranges of file ages, for both
/sps/lsstcest. Click on one the images below to explore other visualizations via the pull down menus at the top of the target page: you will be able to select the specific metric and the timeframe you want to get information about.
The figure below shows the allocated (blue) vs used (green) storage under
/pbs/throng/lsst. Click on the image to get more detailed information.