Monday Plenary 3: Contributed talk
When
Where
Theme: Ground and space mission operations software
I will briefly introduce the Dark Energy Spectroscopic Instrument (DESI) and DESI survey before focusing on the work of a small, dedicated team to develop open-source software for the reduction, analysis, and dissemination of the DESI data. The MPI- and GPU-enabled python code, with wrapped C functions in a few locations, is capable of parallelizing to multiple compute nodes. We further leverage the embarrassingly parallel nature of the data to process each set of individual targets independently, giving us the ability to scale our processing to tens of thousands of CPU cores. For each set of 5000 targets, the code simultaneously extracts the 5000 spectra from the raw data, wavelength-calibrates, sky-subtracts, and flux-calibrates them. It also estimates their inverse variances while propagating a resolution matrix encoding the wavelength-dependent non-Gaussian line spread function per fiber, for proper modeling of the spectra in an analysis. The raw data, intermediate data products, and final data products are stored in Flexible Image Transport System (FITS) files along with metadata and derived quantities stored in the headers of the Header-Data Units.
I will give details about key improvements in the software to reduce wall-clock time, discuss the performance of the software, and conclude with a discussion of the recent release of early data with accompanying code tags in DESI’s Early Data Release.