Thursday Plenary 3: Contributed talk
When
Where
Theme: Cloud infrastructures for astronomical data analysis
The high data rates from current and next generation radio interferometers (MeerKAT, JVLA; SKA, ngVLA) necessarily require the data to be processed via a highly parallelized architecture in order to complete processing at reasonable timescales.
In this talk, I will discuss the Institute for Data Intensive Astronomy (IDIA) facility in Cape Town, South Africa - a pathfinder SKA science regional data centre; the tools and systems developed and adapted to perform processing at scale with the aim of producing high-fidelity images from radio interferometers. I present the IDIA MeerKAT pipeline - an automated, parallel, scaleable full Stokes calibration and imaging pipeline for MeerKAT data designed to operate on the ilifu cluster using off-the-shelf software. Our setup uses the IDIA platform running on hardware provided by the ilifu national facility, taking advantage of cluster-level parallelism, resource management and software containers.
I also discuss use of the CARTA software to efficiently visualize terabyte scale image products remotely, and briefly discuss some of the algorithm developments in progress to produce high fidelity widefield polarimetric maps with MeerKAT and other interferometers.