LSST Data Management Team a Finalist at SC08

Anna Spitz and Arun Jagatheesan

The LSST Data Management (DM) team arrived as one of three finalists for the High Performance Computing (HPC) Storage Challenge (SC) at the 20th SC08 conference in Austin, Texas in November, and came home first runner-up out of a field of strong competitors! The SC Conference is the foremost international conference for high performance computing, networking, storage and analysis attended this year by more than 11,000 scientists, engineers, programmers, researchers, system administrators, managers and educators. The conference features the latest scientific and technical innovations from around the world and showcases demonstrations of how these innovations will enhance people’s abilities to understand information. In its first attempt at this prestigious competition, LSST placed second, winning a certificate of participation and an invitation to come back next year. LSST earned this distinction by demonstrating how the proposed data management infrastructure will manage anticipated data volumes along distributed pathways between the telescope, base facility, archival center and three data centers over the decade long survey lifetime.

LSST’s data management requirements provide opportunities for new data management paradigms to be used for effective solutions in data management. Arun Jagatheesan led the international team on the Challenge, Data-lifecycle Management Over a Loosely Coupled Distributed Infrastructure. The LSST data management team and collaborators prepared the various aspects of this Challenge for a year prior to the Austin meeting.

For the SC08 Challenge, LSST’s data management team focused on just one of the LSST requirements: data-lifecycle management. LSST data has to be managed over its lifecycle on heterogeneous media at multiple sites. LSST’s storage infrastructure resides in geographically distributed data centers. It is a challenge to provide an efficient long-term infrastructure that allows the union of storage from multiple organizations and from multiple storage systems.

The LSST challenge simulated data management with various data centers taking on different roles within the LSST project: CC-IN2P3 in France played the role of the LSST telescope, the Royal Observatory at the University of Edinburgh played the role of a data center at the base of the telescope in Chile, and the National Center for Supercomputing Applications (NCSA) played the role of the data archival center. Three data centers including San Diego Supercomputer Center (SDSC), Cerro Tololo Inter-American Observatory (CTIO) and Renaissance Computing Institute (Renci), acted as data access centers with different capabilities. The team then ran experiments to demonstrate how LSST will handle data.

The team performed multiple experiments to demonstrate the scalability, performance, and innovation of LSST solutions. These experiments (or simulations) showed that the proposed LSST infrastructure can manage more than nine quintillion files (a billion- billion) with hierarchical rules (or policies) at different sites that participate in the LSST—many more than will be required. The SC08 Selection Committee issued the Finalist Certificate with a statement of recognition for LSST’s achievement and an invitation to compete next year. Lessons learned by participating in these challenges improve not only LSST’s future solutions but also steer the future of data management technologies in the worldwide community.