Thursday Plenary 4: Contributed talk
When
Where
Theme: Cloud infrastructures for astronomical data analysis
Located in Lyon, France, the IN2P3 / CNRS Computing Centre (CC-IN2P3) has been preparing its contribution to produce the Legacy Survey of Space and Time in its role as the Rubin Observatory’s France Data Facility.
An integral copy of the raw images will be imported and stored there for the duration of the 10 year-long survey and annual campaigns of reprocessing of 40% of the raw images recorded since the beginning of the survey will be performed on its premises. The data products of those campaigns will be sent back to the Observatory’s archive center in the USA.
As a scientific data processing facility shared by several dozen international projects in high energy physics, nuclear physics and astroparticle physics, in recent years we have observed a significant increase in both the computing and storage capacity demand as well as in the complexity of the services required for supporting astroparticle physics projects. We expect their needs to continue increasing for the foreseeable future: major international projects like Rubin, Euclid, KM3NeT, Virgo/LIGO represent a sizeable fraction of the resources CC-IN2P3 provides to the science projects it supports, even if not yet at the level of the high energy physics projects.
In this contribution we will address how we have been preparing to perform bulk image processing for the needs of the Rubin Observatory annual data release processing campaigns for the duration of the survey. We will present the architecture of the system we deployed with focus on the storage, compute and data transfer components and how we have been testing the system at significant scale. We will highlight and motivate some of the solutions we adopted which have proven effective for our successful contribution to other large science projects like CERN’s Large Hadron Collider. We will also cover our initial experience with components deployed for the specific needs of scientific exploitation of Rubin data such as the astronomical catalog database and the Rubin science platform.