LSST-NCSA: A Partnership to Change the Way Science is Done

Anna H. Spitz

LSST is partnering with the University of Illinois at Urbana-Champaign’s National Center for Supercomputing Applications (NCSA) to design large-scale computing, storage and networking infrastructure for the Data Management (DM) system. Since its opening in 1986, NCSA has made significant contributions to the growth of cyberinfrastructure for science and engineering. NCSA provides the design of infrastructure and middleware for LSST, and it will host the archive. The partnership of LSST and NCSA experts will make possible LSST’s ability to process the expected 15 terabytes of raw data and more than 100 terabytes of processed data each night.

NCSA is a unique state-federal partnership to develop and deploy national-scale cyberinfrastructure that advances science and engineering. NCSA is one of the original National Science Foundation (NSF) Supercomputer Centers. Funding now comes from the state of Illinois, the University of Illinois, private sector partners, NSF and other federal agencies.

Who is making it all happen?

Currently six NCSA-ers are involved in the LSST project. They are engaged on two fronts with the LSST effort: middleware design and development and archive design. Ray Plante is head of the NCSA LSST team and leading the middleware design for the LSST DM. He has an amazing combination of skills: he is an astronomer by training, and has enormous experience with software design and development. The NCSA personnel effort will increase to about ten FTE per year once the LSST enters its construction phase.

Cristina Beldica, initial manager for the LSST effort, has an interest in the project typical of many participants: “I came to work with LSST in 2004 after finishing the NEESgrid earthquake engineering project. My experience there with a large NSF effort was a perfect match for LSST. Plus, I also had experience with working with diverse, geographically distributed teams. I think what makes LSST so exciting are its ambitious science goals; from the DM perspective it’s probably the sheer amount of data and derived from this, the degree of precision and automation that the DM system requires. There are all these checks and balances that need to be in place to make sure every bit of information goes where it is supposed to go.”

The Blue Waters Project

LSST is certainly not the only cutting edge activity underway at NCSA. With the advent of the Blue Waters system, due online in 2011, NCSA will be home to the first computer system dedicated to open scientific research sustaining more than one petaflops (one quadrillion calculations per second). Blue Waters, a supercomputer of unprecedented power, is a joint effort of NCSA, the University of Illinois at Urbana-Champaign (an LSST institutional member), IBM, and the Great Lakes Consortium for Petascale Computation.

NCSA is creating teams with great breath and depth of expertise for this project. Currently approximately eighty people at University of Illinois are directly involved in the Blue Waters project at least part-time. Other people working on the project include staff from the University of North Carolina, Louisiana State University, Los Alamos National Laboratory, University of Michigan, and Shodor Foundation. The number of people involved in the project will increase as it engages with more potential users of the system, and for the deployment and commissioning phase. “We expect to have around seventy FTEs during the production stage,” says Beldica, Senior Program Manager of Blue Waters.

Success of a project as large and ambitious as Blue Waters requires special management attention and expertise. This is a large and complex project that involves not only construction of a building and the deployment of hardware and equipment (as if that weren’t enough!), but also software development (NCSA is working on this in partnership with IBM), application support, and education and outreach. Beldica became involved with the Blue Waters project during the proposal writing. NCSA needed to convince NSF and the reviewers that the team knew how to manage such project. Therefore the proposal included a detailed management plan. After winning the award, it came time to implement this plan and to execute it. The team understood that Blue Waters is not one of the typical research projects that one expects to find at a university. Instead, it requires a greater degree of discipline and formal project management. The challenge is to implement all the controls needed in a project of this magnitude without making it a burden for the team. “So far, I think we’re doing a good job!” says a cautiously optimistic Beldica.

How could LSST and Blue Waters Interface?

LSST will have its own dedicated computers, disks and archive for storing and processing the data that comes from the telescope. But in addition to the nightly processing, LSST DM will periodically reprocess the entire data set collected to that point. It is possible that such massive tasks could exceed the capability of the dedicated LSST resources (or require a long processing time). In these situations, running these pipelines on the Blue Waters system could not only save time, but enable some types of processing that might otherwise be prohibitive or even impossible.

Large simulations and computationally intensive science applications (astronomy, high-energy physics) that use LSST data could benefit from the use of the Blue Waters system. Petascale Computing Resource Allocations (PRAC awards) from the NSF allow research teams to work closely with the Blue Waters project team in preparing their codes to run on Blue Waters. In order to receive an award of time on Blue Waters when it comes online in 2011 through the NSF allocation process, teams must receive a pre-allocation through the current PRAC process. Both PRAC awards and allocations of time on Blue Waters will be based on peer-review. LSST-related projects would use this mechanism to obtain time on the system.

The initial set of PRAC awards that were recently announced (six so far) includes two astronomy applications. (see Petascale Computing Resource Allocations at nsf.gov). Beldica says, “In my opinion this is a good indicator that NSF recognizes the potential of Blue Waters for enabling astronomy research.”

The Blue Waters consulting office responds to questions from prospective users of the system. Interested parties can contact the office at bwconsult@ncsa.uiuc.edu.

A strong and growing collaboration

From both NCSA and LSST, come praise for the collaboration and ideas for even more exchanges: “The experience that we are gaining from the Blue Waters will help us in the design and deployment of the LSST Archive Center. For example, in the early stages of LSST, the size of the LSST archive seemed quite daunting. However, now it appears rather reasonable compared to the archive that will support Blue Waters (currently estimated at 500 petabytes),” according to Beldica.

“NCSA’s deep experience in developing and operating supercomputing facilities is considered critical to the success of the DM project. NCSA’s October 1, 2008 NSF award to host the Tier 1 Petascale Facility (Blue Waters) is very important to LSST DM, as the LSST Archive Center is scheduled to be hosted in that facility, given its advanced design, connectivity, and capacity,” says Jeffrey Kantor, LSST Data Management Project Manager.

The LSST-NCSA collaboration is unique: “In terms of the collaboration model, I believe that the collaboration with LSST is much stronger than what we plan for Blue Waters. The way I see it the NCSA team is an intrinsic part of LSST. In Blue Waters, as expected, a great part of our efforts is focused on getting the system ready for production. At the same time, we are fully aware that the system itself is not worth much unless there are codes ready to take advantage of its capabilities from the very first day of production. Therefore, we have devised an application and user support plan that includes engagement with the PRAC awardees and the formation of PACTs (Petascale Applications Collaborations Teams). The goal is to provide the science and engineering research teams with the resource and knowledge to help them scale up their applications for running on Blue Waters. In these early stages, since the actual hardware is not yet available, the teams use simulators and emulators to analyze and tune the codes.” The Blue Waters project offers even more potential for evaluation of LSST data through the mechanism of PRAC awards.

The partnership of LSST and NSCA on data management research creates a collaborative effort with great personal talent and commitment to employ unprecedented supercomputing capabilities. This collaboration is changing the way science is done.

For more information about NCSA, see http://www.ncsa.illinois.edu/.