Presentation is loading. Please wait.

Presentation is loading. Please wait.

Scientific Data Mining in Astronomy

Similar presentations

Presentation on theme: "Scientific Data Mining in Astronomy"— Presentation transcript:

1 Scientific Data Mining in Astronomy
THE US NATIONAL VIRTUAL OBSERVATORY Scientific Data Mining in Astronomy Kirk D. Borne George Mason University 1

2 OUTLINE Scientific Databases Some key astronomy problems
Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

3 OUTLINE Scientific Databases Some key astronomy problems
Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

4 10 Unique Features of Scientific Data
Each of these characteristics requires special handling beyond what you read in standard data mining textbooks: Scientific data depend on experimental equipment and conditions. Scientific data have noise. Scientific data have been (or need to be) calibrated. Scientific units on data values are imperative. Scientific databases often contain associated columns: { value, error }. Scientific data values are often non-linear (log values, magnitudes, asinh). History of scientific data creation, processing, and versioning is critical = Provenance. Metadata, Metadata, Metadata = tells us “who, what, when, where, how” NOTE: Semantic Metadata are becoming more important = “why”. Context is critical (e.g., brightness in an optical catalog is expressed in mags, but expressed in counts/sec in an X-ray catalog, or milli-Jansky in a radio catalog). Scientific data have different levels of abstraction: raw, calibrated, reduced data products, derived information, extracted knowledge, published results. All of this makes the “Data Preparation” phase of any scientific data mining experiment even more critical and essential. 4

5 OUTLINE Scientific Databases Some key astronomy problems
Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

6 Some key astronomy problems
Some key astronomy problems that can be addressed with data mining techniques: Cross-Match objects from different catalogues The distance problem (e.g., Photometric Redshift estimators) Star-Galaxy Separation Cosmic-Ray Detection in images Supernova Detection and Classification Morphological Classification (galaxies, AGN, gravitational lenses, ...) Class and Subclass Discovery (brown dwarfs, methane dwarfs, ...) Dimension Reduction = Correlation Discovery Learning Rules for improved classifiers Classification of massive data streams Real-time Classification of Astronomical Events Clustering of massive data collections Novelty, Anomaly, Outlier Detection in massive databases 6

7 OUTLINE Scientific Databases Some key astronomy problems
Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

8 Classification Methods: Decision Trees, Neural Networks, SVM (Support Vector Machines)
There are 2 Classes! How do you ... Separate them? Distinguish them? Learn the rules? Classify them? Apply Kernel (SVM)

9 Decision Tree Classification Example: SKICAT Star-Galaxy Discrimination

10 Decision Tree Classification Example: Classification of candidates for new supernova in galaxies

11 Clustering is used to discover the different unique groupings (classes) of attribute values. The case shown below is not obvious: one or two groups?

12 This case is easier: there are two groups
This case is easier: there are two groups. (in fact, this is the same set of data elements as shown on the previous slide, but plotted here using a different attribute.)

13 Clustering in multiple dimensions: colors combined from SDSS & 2MASS magnitudes

14 Clustering: Class Discovery and Rule Learning
Reference: Clusters and the separation of classes depend on which attributes (dimensions) are chosen to be projected, as in the following star-galaxy discrimination test: Not good Good

15 Semisupervised Learning: Outlier Detection
Reference: A demonstration of a generic machine-assisted discovery problem — data mapping and a search for outliers. This schematic illustration is of the clustering problem in a parameter space given by three object attributes: P1, P2, and P3. In this example, most of the data points are assumed to be contained in three, dominant clusters (DC1, DC2, and DC3). However, one may want to discover less populated clusters (e.g., small groups or even isolated points), some of which may be too sparsely populated, or lie too close to one of the major data clouds. In some cases, negative clusters (holes), may exist in one of the major data clusters.

16 Outlier Detection: Serendipitous Discovery of Rare or New Objects & Events

17 Principal Components Analysis & Independent Components Analysis
Cepheid Variables: Cosmic Yardsticks -- One Correlation -- Two Classes! ... Class Discovery!

18 Example: SOM (Self-Organizing Map)
The SOM (Self-Organizing Map) is one technique for organizing information in a database based upon links between concepts. It can be used to find hidden relationships and patterns in more complex data collections, usually based on links between keywords or metadata. 18

19 Exploring the Time Domain
Astronomy Data Mining in Action Exploring the Time Domain Mega-Flares on normal Sun-like stars = a star like our Sun increased in brightness 300X one night! … say what??

20 Example: The Thinking Telescope
Sample Data Mining Applications: (credit: ) Automated Feature Extraction: Real-time identification of artifacts and transients in direct and difference images. Classifiers: Automated classification of celestial objects based on temporal and spectral properties. Anomaly Detection: Real-time recognition of important deviations from normal behavior for persistent sources. 20

21 From Sensors to Sense From Data to Knowledge: from sensors to sense
Data  Information  Knowledge 21

22 Reference:
VOEventNet Reference: 22

23 Learning From Archived Temporal Data (Time Series): Classify New Data (Bayes Analysis or Markov Modeling)

24 Photometric-Redshift Estimation
Photometric vs. Spectroscopic Redshift Estimates: Left panel: standard technique Right panel: Machine Learning (data mining) application Reference: 24

25 Star-Galaxy Separation in Clustered Feature Space

26 Bayesian Probabilistic Estimation for Catalog Cross-Matching
Reference: 26

27 Reference: Borne, Dutta, Giannella, Kargupta, & Griffin 2008
Fundamental Plane for 156,000 cross-matched Sloan+2MASS Elliptical Galaxies: plot shows variance captured by first 2 Principal Components as a function of local galaxy density. Reference: Borne, Dutta, Giannella, Kargupta, & Griffin 2008 Slide Content Slide content % of variance captured by PC1+PC2 27 low (Local Galaxy Density) high

28 OUTLINE Scientific Databases Some key astronomy problems
Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

29 Suggested Reading: Data Mining in Astronomy
Djorgovski et al. 2000, Searches for Rare and New Types of Objects. Djorgovski et al. 2000, Exploration of Large Digital Sky Surveys. Djorgovski et al. 2001, Exploration of Parameter Spaces in a Virtual Observatory. Mining the Sky, 2001, published proceedings of ESO conference. Suchkov et al. 2003, Automated Object Classification with ClassX. astro-ph/ Suchkov, Hanisch, & Margon 2005, A Census of Object Types and Redshift Estimates in the SDSS Photometric Catalog from a Trained Decision Tree Classifier. Giannella et al. 2006, Distributed Data Mining for Astronomy Catalogs. Rohde et al. 2006, Matching of Catalogues by Probabilistic Pattern Classification. Budavari & Szalay 2008, Probabilistic Cross-Identification of Astronomical Sources. 29

30 Suggested Reading, continued: Data Mining in Astronomy
Odewahn et al. 1993, Star-Galaxy Separation with a Neural Network. 2: Multiple Schmidt Plate Fields. Borne 2000, Science User Scenarios for a Virtual Observatory Design Reference Mission: Science Requirements for Data Mining. astro-ph/ Brunner et al. 2001, Massive Datasets in Astronomy. astro-ph/ Gray et al. 2002, Data Mining the SDSS SkyServer Database. Odewahn et al. 2004, The Digitized Second Palomar Observatory Sky Survey (DPOSS). III. Star-Galaxy Separation. Ball, Brunner, et al. 2006, Robust Machine Learning Applied to Astronomical Data Sets. I. Star-Galaxy Classification of the Sloan Digital Sky Survey DR3 Using Decision Trees. Ball, Brunner, et al. 2007, Robust Machine Learning Applied to Astronomical Data Sets. II. Quantifying Photometric Redshifts for Quasars Using Instance-based Learning. Ball, Brunner, et al. 2008, Robust Machine Learning Applied to Astronomical Data Sets. III. Probabilistic Photometric Redshifts for Galaxies and Quasars in the SDSS and GALEX. 30

31 Suggested Reading, continued: Data Mining in Astronomy
Rogers & Riess 1994, Detection and Classification of CCD Defects with an Artificial Neural Network. Feeney et al. 2005, Automated Detection of Classical Novae with Neural Networks. Wadadekar 2005, Estimating Photometric Redshifts Using Support Vector Machines. Bazell & Miller 2005, Class Discovery in Galaxy Classification. Bazell, Miller, & SubbaRao 2006, Objective Subclass Determination of Sloan Digital Sky Survey Spectroscopically Unclassified Objects. Ferreras et al. 2006, A Principal Component Analysis approach to the Star Formation History of Elliptical Galaxies in Compact Groups. Way & Srivastava 2006, Novel Methods for Predicting Photometric Redshifts from Broadband Photometry Using Virtual Sensors. Carliles et al. 2007, Photometric Redshift Estimation on SDSS Data Using Random Forests. 31

32 OUTLINE Scientific Databases Some key astronomy problems
Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

33 Some Data Mining Software & Projects
General data mining software packages: Weka (Java): Weka4WS (Grid-enabled): RapidMiner: Astronomy-specific software and/or user clients: VO-Neural: AstroWeka: OpenSkyQuery: ALADIN: MIRAGE: AstroBox: Astronomical and/or Scientific Data Mining Projects: GRIST: ClassX: LCDM: F-MASS: NCDM: 33

34 Weka:
Weka is in your NVOSS software distribution. Weka is a collection of open source machine learning algorithms for data mining tasks. Weka algorithms can either be applied directly to a dataset or called from your own Java code. Weka comes with its own GUI. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. 34

35 AstroWeka:


Java Package for exploratory data analysis (EDA), correlation mining, and interactive pattern discovery.

38 OUTLINE Scientific Databases Some key astronomy problems
Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

39 Science is Knowledge Work
Data  Information  Knowledge Knowledge Discovery is the central theme of science. Knowledge Discovery in Databases (KDD) is the killer app for large scientific databases. Therefore, KDD (i.e., Data Mining) is an essential tool, since “big-data” science is here to stay (at petabytes and beyond).

40 Scientific Knowledge Discovery

41 Heliophysics Space Weather Example

42 Sun-Earth Space Environment – Rich Source of Heliophysical Phenomena

43 Multi-point Observations and Models of Space Plasmas Deliver a Deluge of Physical Measurements


45 Heliophysics Space Weather Example
CME = Coronal Mass Ejection SEP = Solar Energetic Particle

46 Data Mining: It is more than just connecting the dots

47 Sample Astronomy Data Mining Application Ideas for your Projects
Neural Network for Pixel Classification: Event Detection and Prediction (e.g., Supernova or Cosmic-ray hit?) Bayesian Network for Object Classification (star or galaxy?) PCA for finding Fundamental Planes of Galaxy Parameters PCA (weakest component) for Outlier Detection: anomalies, novel discoveries, new objects Link Analysis (Association Mining) for Causal Event Detection (e.g., linking optical transients with gamma-ray events) Clustering analysis: Spatial, Temporal, or any scientific database parameters Markov models: Temporal mining, classification, and prediction from time series data

Download ppt "Scientific Data Mining in Astronomy"

Similar presentations

Ads by Google