DATA MANAGEMENT AND THE GOOGLE PARTNERSHIP
LSST presents immense and stimulating challenges in data management. LSST will collect, process and store more than 1000 times the data of any existing optical telescope each night. LSST will take over 1000 panoramic images per night, 30 terabytes of data in a ten-hour winter night. The data management system (DMS) will compare new with previous images to monitor change. Multiple Data Challenges, prototypes of the full DMS, take place during the Design and Development phase of the project.
Each data challenge validates a different aspect of the system. DC1, completed in 2006, tested the scalability of the overall processing architecture and data flows. DC2, just completed, prototyped the nightly processing pipelines that produce transient alerts and the middleware that supports them. DC3 will prototype the data release pipelines in 2008. Finally DC4 will examine data access by the astronomical community and the data processing that supports scientific use of the LSST data.
The data management challenges and the information LSST will gather have attracted Google to the partnership. LSST and Google staff had a face-to-face meeting in October at Google headquarters in Mountain View, CA. Approximately fifty people attended with representatives from all data management partners and LSST project management, Royal Edinburgh Observatory and Google. Principals of Sky and Google Earth attended as well. Google presented information on Google technologies that may be useful to DMS. The group established the basic scope for the Data Challenge 3 (DC3) and identified areas for exploring open source versions of Google map reduce, bigtable, and file system technologies in DC3.
For more information, contact Tim Axelrod, Data Management Scientist and Jeff Kantor, Data Management Manager
LSST E-News is a free email publication of the Large Synoptic Survey Telescope Project. It is for informational purposes only, and the information is subject to change without notice.
Subscribe/Unsubscribe
Copyright © 2008 LSST Corp., Tucson, AZ • www.lsst.org