Kian-Tat Lim — Handling Trillions of Web Page Visits or Trillions of Observations

Handling extremely large data sets exhilarates Kian-Tat Lim. Whether it’s Yahoo! dealing with billions of users visiting thousands of web pages for trillions of page views or LSST viewing billions of objects thousands of times for trillions of observations, Kian-Tat enjoys the challenge of managing these data.

Kian-Tat is an Information Systems Specialist at the SLAC National Accelerator Laboratory. He originally joined LSST as a volunteer in January 2007 but was so taken by the project that he started as an employee in June 2007. He builds the database, including the petabyte-scale Database Management System that will support it, and image access systems. He assists with the architecture of the data management systems with a focus on the middleware layer for which he is the Middleware Lead and member of the Data Access Working Group. Kian-Tat finds the most challenging aspect of his work to be designing a system where there is substantial uncertainty with respect to hardware capabilities and end-user needs.

Kian-Tat has experience in both science and computing. After graduating from Haverford in 1987 with a BA in chemistry, he completed his PhD in computational chemistry at CalTech in 1995. Kian-Tat was the Chief Architect for Strategic Data Solutions at Yahoo!, Inc. where he spent more than seven years building extremely large data management system and data mining applications. His time working on extremely large commercial data sets often with a time series analysis component is proving very helpful now on LSST. “Working in a commercial environment has also increased my appreciation for documentation and software development processes.” Kian-Tat has three patents and two unpublished patents and has built five proprietary large-scale software systems.

In addition to LSST Lim is consulting for the data management systems for the Linac Coherent Light Source (LCLS) at SLAC. LCLS is an extremely bright, extremely short pulse laser with X-rays that can be used to take movies of chemical reaction, its pulses powerful enough to take images of individual molecules. He is also helping Jacek Becla with the Extremely Large Database workshop series that will bring leaders from science, industry, academia, and database vendors together to discuss the unique challenges of managing massive data sets.

As to where he might be in five or ten years, Kian-Tat looks forward to “something interesting and challenging where I can apply my skills and experience to build something elegant that will help the world.”

Article written by Anna Spitz