LSST Data Management Pipelines
The pipelines process the images to produce the catalogs, which are then made accessible to the community via open interfaces in a Virtual Observatory model. Since new data is being collected nightly throughout the LSST's 10-year survey period, and scientific algorithms will evolve during this time frame, significant re-processing will occur. This must be taken into account in sizing the LSST technology resources and making the LSST middleware easily extendable.
The pipelines can be categorized in terms of either being near real-time or "static", depending on how stringent the associated latency and throughput deadlines are. Examples of near real-time pipelines include data quality assessment providing feedback to telescope operations, instrument calibration processing, and time-domain / transient science analysis. These pipelines execute at the mountain/base facility in order to avoid the latency associated with long-haul transmission of the raw data. The static pipelines include deep image co-addition, weak lensing shear processing needed for dark energy - dark matter science, and object cataloging. These pipelines execute at the archive center. The archive center also performs re-processing of the near real-time pipelines.
