Presentation is loading. Please wait.

Presentation is loading. Please wait.

Astronomy, Petabytes, and MySQL MySQL Conference Santa Clara, CA April 16, 2008 Kian-Tat Lim Stanford Linear Accelerator Center.

Published byTodd O’Brien’ Modified over 8 years ago

Similar presentations

Presentation on theme: "Astronomy, Petabytes, and MySQL MySQL Conference Santa Clara, CA April 16, 2008 Kian-Tat Lim Stanford Linear Accelerator Center."— Presentation transcript:

1 Astronomy, Petabytes, and MySQL MySQL Conference Santa Clara, CA April 16, 2008 Kian-Tat Lim Stanford Linear Accelerator Center

2 MySQL Conference April 16, 2008 Santa Clara, CA / 47 2 Outline LSST LSST Database LSST Database + MySQL

3 MySQL Conference April 16, 2008 Santa Clara, CA / 47 3 LSST What Is It? Why Build It?

4 MySQL Conference April 16, 2008 Santa Clara, CA / 47 4 LSST What Is It? Why Build It?

5 MySQL Conference April 16, 2008 Santa Clara, CA / 47 5 Telescope Proposed telescope to be built in Chile

6 MySQL Conference April 16, 2008 Santa Clara, CA / 47 6 Large 3.2 gigapixel camera 8.4 meter diameter mirror

7 MySQL Conference April 16, 2008 Santa Clara, CA / 47 7 Synoptic Survey Wide Deep Fast

8 MySQL Conference April 16, 2008 Santa Clara, CA / 47 8 LSST What Is It? Why Build It?

9 MySQL Conference April 16, 2008 Santa Clara, CA / 47 9 Dark Matter and Energy Photo: J. A. Tyson, W. Colley, E. L. Turner, and NASA

10 MySQL Conference April 16, 2008 Santa Clara, CA / 47 10 Variable Objects

11 MySQL Conference April 16, 2008 Santa Clara, CA / 47 11 Transient Objects

12 MySQL Conference April 16, 2008 Santa Clara, CA / 47 12 Moving Objects Photo: D. Roddy, Lunar and Planetary Institute

13 MySQL Conference April 16, 2008 Santa Clara, CA / 47 13 LSST Database What’s In It? How Big? How Often? What Queries? Unusual Needs

14 MySQL Conference April 16, 2008 Santa Clara, CA / 47 14 LSST Database What’s In It? How Big? How Often? What Queries? Unusual Needs

15 MySQL Conference April 16, 2008 Santa Clara, CA / 47 15 Database: Components Image Metadata Moving Objects Catalog Object Catalog Source Catalog Difference Image Source Catalog Provenance Statistics Summaries Calibration Engineering and Facility Database

16 MySQL Conference April 16, 2008 Santa Clara, CA / 47 16 Astronomical Objects Image Metadata Moving Objects Catalog Object Catalog Source Catalog Difference Image Source Catalog Provenance Statistics Summaries Calibration Engineering and Facility Database

17 MySQL Conference April 16, 2008 Santa Clara, CA / 47 17 Sources Image Metadata Moving Objects Catalog Object Catalog Source Catalog Difference Image Source Catalog Provenance Statistics Summaries Calibration Engineering and Facility Database

18 MySQL Conference April 16, 2008 Santa Clara, CA / 47 18 Changes Image Metadata Moving Objects Catalog Object Catalog Source Catalog Difference Image Source Catalog Provenance Statistics Summaries Calibration Engineering and Facility Database

19 MySQL Conference April 16, 2008 Santa Clara, CA / 47 19 Image Metadata Moving Objects Catalog Object Catalog Source Catalog Difference Image Source Catalog Provenance Statistics Summaries Calibration Engineering and Facility Database

20 MySQL Conference April 16, 2008 Santa Clara, CA / 47 20 Calibration and Facility Image Metadata Moving Objects Catalog Object Catalog Source Catalog Difference Image Source Catalog Provenance Statistics Summaries Calibration Engineering and Facility Database

21 MySQL Conference April 16, 2008 Santa Clara, CA / 47 21 LSST Database What’s In It? How Big? How Often? What Queries? Unusual Needs

22 MySQL Conference April 16, 2008 Santa Clara, CA / 47 22 Sagans of Rows 49 billion objects 2.8 trillion sources

23 MySQL Conference April 16, 2008 Santa Clara, CA / 47 23 Lots of Columns 308 columns for objects 56 columns for sources (for now)

24 MySQL Conference April 16, 2008 Santa Clara, CA / 47 24 Database Size Grows to >14 PB

25 MySQL Conference April 16, 2008 Santa Clara, CA / 47 25 LSST Database What’s In It? How Big? How Often? What Queries? Unusual Needs

26 MySQL Conference April 16, 2008 Santa Clara, CA / 47 26 Frequency Nightly updates Semi-annual data releases

27 MySQL Conference April 16, 2008 Santa Clara, CA / 47 27 LSST Database What’s In It? How Big? How Often? What Queries? Unusual Needs

28 MySQL Conference April 16, 2008 Santa Clara, CA / 47 28 Queries All about an object All objects meeting criteria All objects near objects meeting criteria All objects with interesting time series All pairs of objects with similar time series

29 MySQL Conference April 16, 2008 Santa Clara, CA / 47 29 LSST Database What’s In It? How Big? How Often? What Queries? Unusual Needs

30 MySQL Conference April 16, 2008 Santa Clara, CA / 47 30 Unusual Needs Flexibility Provenance

31 MySQL Conference April 16, 2008 Santa Clara, CA / 47 31 LSST Database + MySQL Why MySQL? Scalability? Performance?

32 MySQL Conference April 16, 2008 Santa Clara, CA / 47 32 LSST Database + MySQL Why MySQL? Scalability? Performance?

33 MySQL Conference April 16, 2008 Santa Clara, CA / 47 33 MySQL Relational database management system

34 MySQL Conference April 16, 2008 Santa Clara, CA / 47 34 Open Source Vibrant community Strong company support

35 MySQL Conference April 16, 2008 Santa Clara, CA / 47 35 Hardware Runs on commodity hardware

36 MySQL Conference April 16, 2008 Santa Clara, CA / 47 36 In-Memory Tables Needed for near-real-time processing

37 MySQL Conference April 16, 2008 Santa Clara, CA / 47 37 LSST Database + MySQL Why MySQL? Scalability? Performance?

38 MySQL Conference April 16, 2008 Santa Clara, CA / 47 38 “MySQL Grid”

39 MySQL Conference April 16, 2008 Santa Clara, CA / 47 39 Partitioning Large tables partitioned spatially

40 MySQL Conference April 16, 2008 Santa Clara, CA / 47 40 Replication Dimension tables likely replicated

41 MySQL Conference April 16, 2008 Santa Clara, CA / 47 41 Needs: Distributor/Combiner LSST will build prototype Need long-term support

42 MySQL Conference April 16, 2008 Santa Clara, CA / 47 42 LSST Database + MySQL Why MySQL? Scalability? Performance?

43 MySQL Conference April 16, 2008 Santa Clara, CA / 47 43 Per-Column Indexing 2X data size

44 MySQL Conference April 16, 2008 Santa Clara, CA / 47 44 Needs: Optimizer Efficient use of multiple (20- 30) indexes

45 MySQL Conference April 16, 2008 Santa Clara, CA / 47 45 Needs: Indexes Bitmap/compressed indexes

46 MySQL Conference April 16, 2008 Santa Clara, CA / 47 46 Needs: Storage Engine “Shared scan” for long- running full-table queries

47 MySQL Conference April 16, 2008 Santa Clara, CA / 47 47 Summary Building a petabyte DB MySQL can be a core component

Download ppt "Astronomy, Petabytes, and MySQL MySQL Conference Santa Clara, CA April 16, 2008 Kian-Tat Lim Stanford Linear Accelerator Center."

Similar presentations

Mark Holliman Wide Field Astronomy Unit Institute for Astronomy University of Edinburgh.

Mark Holliman Wide Field Astronomy Unit Institute for Astronomy University of Edinburgh.
© Copyright 2012 STI INNSBRUCK Apache Lucene Ioan Toma based on slides from Aaron Bannert

© Copyright 2012 STI INNSBRUCK Apache Lucene Ioan Toma based on slides from Aaron Bannert
Data Management in the Cloud Paul Szerlip. The rise of data Think about this o For the past two decades, the largest generator of data was humans -- now.

Data Management in the Cloud Paul Szerlip. The rise of data Think about this o For the past two decades, the largest generator of data was humans -- now.
Astronomy of the Next Decade: From Photons to Petabytes R. Chris Smith AURA Observatory in Chile CTIO/Gemini/SOAR/LSST.

Astronomy of the Next Decade: From Photons to Petabytes R. Chris Smith AURA Observatory in Chile CTIO/Gemini/SOAR/LSST.
Databases for the 'Pi of the Sky' experiment Marek Biskup Warsaw University.

Databases for the 'Pi of the Sky' experiment Marek Biskup Warsaw University.
Introduction The Open Science Grid (OSG) is a consortium of more than 100 institutions including universities, national laboratories, and computing centers.

Introduction The Open Science Grid (OSG) is a consortium of more than 100 institutions including universities, national laboratories, and computing centers.
A Fast Growing Market. Interesting New Players Lyzasoft.

A Fast Growing Market. Interesting New Players Lyzasoft.
All these Sky Pixels Are Yours The evolution of telescopes and CCD Arrays: The Coming Data Nightmare.

All these Sky Pixels Are Yours The evolution of telescopes and CCD Arrays: The Coming Data Nightmare.
Organizing the Extremely Large LSST Database for Real-Time Astronomical Processing ADASS London, UK September 23-26, 2007 Jacek Becla 1, Kian-Tat Lim 1,

Organizing the Extremely Large LSST Database for Real-Time Astronomical Processing ADASS London, UK September 23-26, 2007 Jacek Becla 1, Kian-Tat Lim 1,
July 8, 2008SLAC Annual Program ReviewPage 1 LSST Data Management and Access Jacek Becla LSST Data Access & Database Technology Group Leader.

July 8, 2008SLAC Annual Program ReviewPage 1 LSST Data Management and Access Jacek Becla LSST Data Access & Database Technology Group Leader.
1 LSST: Dark Energy Tony Tyson Director, LSST Project University of California, Davis Tony Tyson Director, LSST Project University of California, Davis.

1 LSST: Dark Energy Tony Tyson Director, LSST Project University of California, Davis Tony Tyson Director, LSST Project University of California, Davis.
Building a Framework for Data Preservation of Large-Scale Astronomical Data ADASS London, UK September 23-26, 2007 Jeffrey Kantor (LSST Corporation), Ray.

Building a Framework for Data Preservation of Large-Scale Astronomical Data ADASS London, UK September 23-26, 2007 Jeffrey Kantor (LSST Corporation), Ray.
Panel Summary Andrew Hanushevsky Stanford Linear Accelerator Center Stanford University XLDB 23-October-07.

Panel Summary Andrew Hanushevsky Stanford Linear Accelerator Center Stanford University XLDB 23-October-07.
Astro-DISC: Astronomy and cosmology applications of distributed super computing.

Astro-DISC: Astronomy and cosmology applications of distributed super computing.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1 Preview of Oracle Database 12 c In-Memory Option Thomas Kyte

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1 Preview of Oracle Database 12 c In-Memory Option Thomas Kyte
Opensource for Cloud Deployments – Risk – Reward – Reality

Opensource for Cloud Deployments – Risk – Reward – Reality
From Photons to Petabytes: Astronomy in the Era of Large Scale Surveys and Virtual Observatories R. Chris Smith AURA/NOAO/CTIO/LSST.

From Photons to Petabytes: Astronomy in the Era of Large Scale Surveys and Virtual Observatories R. Chris Smith AURA/NOAO/CTIO/LSST.
Hopkins Storage Systems Lab, Department of Computer Science A Workload-Driven Unit of Cache Replacement for Mid-Tier Database Caching Xiaodan Wang, Tanu.

Hopkins Storage Systems Lab, Department of Computer Science A Workload-Driven Unit of Cache Replacement for Mid-Tier Database Caching Xiaodan Wang, Tanu.
The VAO is operated by the VAO, LLC. VAO: Archival follow-up and time series Matthew J. Graham, Caltech/VAO.

The VAO is operated by the VAO, LLC. VAO: Archival follow-up and time series Matthew J. Graham, Caltech/VAO.
The Large Synoptic Survey Telescope Philip A. Pinto Steward Observatory University of Arizona for the LSST Collaboration Legacy Projects Workshop 17 May,

The Large Synoptic Survey Telescope Philip A. Pinto Steward Observatory University of Arizona for the LSST Collaboration Legacy Projects Workshop 17 May,

Similar presentations

Ads by Google