Presentation is loading. Please wait.

Presentation is loading. Please wait.

2008 NVO Summer School11 Scientific Data Mining in Astronomy Kirk D. Borne George Mason University T HE.

Similar presentations


Presentation on theme: "2008 NVO Summer School11 Scientific Data Mining in Astronomy Kirk D. Borne George Mason University T HE."— Presentation transcript:

1 2008 NVO Summer School11 Scientific Data Mining in Astronomy Kirk D. Borne George Mason University T HE US N ATIONAL V IRTUAL O BSERVATORY

2 2008 NVO Summer School2 OUTLINE Scientific Databases Some key astronomy problems Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

3 2008 NVO Summer School3 OUTLINE Scientific Databases Some key astronomy problems Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

4 2008 NVO Summer School44 10 Unique Features of Scientific Data Each of these characteristics requires special handling beyond what you read in standard data mining textbooks: 1.Scientific data depend on experimental equipment and conditions. 2.Scientific data have noise. 3.Scientific data have been (or need to be) calibrated. 4.Scientific units on data values are imperative. 5.Scientific databases often contain associated columns: { value, error }. 6.Scientific data values are often non-linear (log values, magnitudes, asinh). 7.History of scientific data creation, processing, and versioning is critical = Provenance. 8.Metadata, Metadata, Metadata = tells us who, what, when, where, how. NOTE: Semantic Metadata are becoming more important = why. 9.Context is critical (e.g., brightness in an optical catalog is expressed in mags, but expressed in counts/sec in an X-ray catalog, or milli-Jansky in a radio catalog). 10.Scientific data have different levels of abstraction: raw, calibrated, reduced data products, derived information, extracted knowledge, published results. All of this makes the Data Preparation phase of any scientific data mining experiment even more critical and essential.

5 2008 NVO Summer School5 OUTLINE Scientific Databases Some key astronomy problems Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

6 2008 NVO Summer School66 Some key astronomy problems Some key astronomy problems that can be addressed with data mining techniques: Cross-Match objects from different catalogues The distance problem (e.g., Photometric Redshift estimators) Star-Galaxy Separation Cosmic-Ray Detection in images Supernova Detection and Classification Morphological Classification (galaxies, AGN, gravitational lenses,...) Class and Subclass Discovery (brown dwarfs, methane dwarfs,...) Dimension Reduction = Correlation Discovery Learning Rules for improved classifiers Classification of massive data streams Real-time Classification of Astronomical Events Clustering of massive data collections Novelty, Anomaly, Outlier Detection in massive databases

7 2008 NVO Summer School7 OUTLINE Scientific Databases Some key astronomy problems Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

8 2008 NVO Summer School8 Classification Methods: Decision Trees, Neural Networks, SVM (Support Vector Machines) There are 2 Classes! How do you... -Separate them? -Distinguish them? -Learn the rules? -Classify them? Apply Kernel (SVM)

9 2008 NVO Summer School9 Decision Tree Classification Example: SKICAT Star-Galaxy Discrimination Reference: ftp://iraf.noao.edu/iraf/conf/web/adass_proc/adass_95/yooj/yooj.htmlftp://iraf.noao.edu/iraf/conf/web/adass_proc/adass_95/yooj/yooj.html

10 2008 NVO Summer School10 Decision Tree Classification Example: Classification of candidates for new supernova in galaxies Reference:

11 2008 NVO Summer School11 Clustering is used to discover the different unique groupings (classes) of attribute values. The case shown below is not obvious: one or two groups?

12 2008 NVO Summer School12 This case is easier: there are two groups. (in fact, this is the same set of data elements as shown on the previous slide, but plotted here using a different attribute.)

13 2008 NVO Summer School13 Clustering in multiple dimensions: colors combined from SDSS & 2MASS magnitudes

14 2008 NVO Summer School Clustering: Class Discovery and Rule Learning Clusters and the separation of classes depend on which attributes (dimensions) are chosen to be projected, as in the following star-galaxy discrimination test: 14 Reference: Not good Good

15 2008 NVO Summer School Semisupervised Learning: Outlier Detection Reference: 15 A demonstration of a generic machine-assisted discovery problem data mapping and a search for outliers. This schematic illustration is of the clustering problem in a parameter space given by three object attributes: P1, P2, and P3. In this example, most of the data points are assumed to be contained in three, dominant clusters (DC1, DC2, and DC3). However, one may want to discover less populated clusters (e.g., small groups or even isolated points), some of which may be too sparsely populated, or lie too close to one of the major data clouds. In some cases, negative clusters (holes), may exist in one of the major data clusters.

16 2008 NVO Summer School16 Outlier Detection: Serendipitous Discovery of Rare or New Objects & Events

17 2008 NVO Summer School17 Principal Components Analysis & Independent Components Analysis Cepheid Variables: Cosmic Yardsticks -- One Correlation -- Two Classes!... Class Discovery!

18 2008 NVO Summer School18 Example: SOM (Self-Organizing Map) The SOM (Self- Organizing Map) is one technique for organizing information in a database based upon links between concepts. It can be used to find hidden relationships and patterns in more complex data collections, usually based on links between keywords or metadata.

19 2008 NVO Summer School19 Mega-Flares on normal Sun-like stars = a star like our Sun increased in brightness 300X one night! … say what?? Exploring the Time Domain Astronomy Data Mining in Action

20 2008 NVO Summer School20 Example: The Thinking Telescope Sample Data Mining Applications: (credit: ) Automated Feature Extraction: Real-time identification of artifacts and transients in direct and difference images. Classifiers: Automated classification of celestial objects based on temporal and spectral properties. Anomaly Detection: Real-time recognition of important deviations from normal behavior for persistent sources.

21 2008 NVO Summer School21 From Sensors to Sense From Data to Knowledge: from sensors to sense From Data to Knowledge: from sensors to sense Data Information Knowledge

22 2008 NVO Summer School22 VOEventNet Reference:

23 2008 NVO Summer School23 Learning From Archived Temporal Data (Time Series): Classify New Data (Bayes Analysis or Markov Modeling)

24 2008 NVO Summer School24 Photometric-Redshift Estimation Photometric vs. Spectroscopic Redshift Estimates: Left panel: standard technique Right panel: Machine Learning (data mining) application Reference:

25 2008 NVO Summer School25 Star-Galaxy Separation in Clustered Feature Space * = star = galaxy

26 2008 NVO Summer School26 Bayesian Probabilistic Estimation for Catalog Cross-Matching Reference:

27 2008 NVO Summer School27 Fundamental Plane for 156,000 cross-matched Sloan+2MASS Elliptical Galaxies: plot shows variance captured by first 2 Principal Components as a function of local galaxy density. Slide Content Slide content low (Local Galaxy Density) high % of variance captured by PC1+PC2 Reference: Borne, Dutta, Giannella, Kargupta, & Griffin 2008

28 2008 NVO Summer School28 OUTLINE Scientific Databases Some key astronomy problems Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

29 2008 NVO Summer School29 Suggested Reading: Data Mining in Astronomy Djorgovski et al. 2000, Searches for Rare and New Types of Objects. Djorgovski et al. 2000, Exploration of Large Digital Sky Surveys. Djorgovski et al. 2001, Exploration of Parameter Spaces in a Virtual Observatory. Mining the Sky, 2001, published proceedings of ESO conference.Mining the Sky Suchkov et al. 2003, Automated Object Classification with ClassX. astro-ph/ astro-ph/ Suchkov, Hanisch, & Margon 2005, A Census of Object Types and Redshift Estimates in the SDSS Photometric Catalog from a Trained Decision Tree Classifier. Giannella et al. 2006, Distributed Data Mining for Astronomy Catalogs. Rohde et al. 2006, Matching of Catalogues by Probabilistic Pattern Classification. Budavari & Szalay 2008, Probabilistic Cross-Identification of Astronomical Sources.

30 2008 NVO Summer School30 Suggested Reading, continued: Data Mining in Astronomy Odewahn et al. 1993, Star-Galaxy Separation with a Neural Network. 2: Multiple Schmidt Plate Fields. Borne 2000, Science User Scenarios for a Virtual Observatory Design Reference Mission: Science Requirements for Data Mining. astro-ph/ astro-ph/ Brunner et al. 2001, Massive Datasets in Astronomy. astro-ph/ astro-ph/ Gray et al. 2002, Data Mining the SDSS SkyServer Database. Odewahn et al. 2004, The Digitized Second Palomar Observatory Sky Survey (DPOSS). III. Star-Galaxy Separation. Ball, Brunner, et al. 2006, Robust Machine Learning Applied to Astronomical Data Sets. I. Star-Galaxy Classification of the Sloan Digital Sky Survey DR3 Using Decision Trees. Ball, Brunner, et al. 2007, Robust Machine Learning Applied to Astronomical Data Sets. II. Quantifying Photometric Redshifts for Quasars Using Instance-based Learning. Ball, Brunner, et al. 2008, Robust Machine Learning Applied to Astronomical Data Sets. III. Probabilistic Photometric Redshifts for Galaxies and Quasars in the SDSS and GALEX.

31 2008 NVO Summer School31 Suggested Reading, continued: Data Mining in Astronomy Rogers & Riess 1994, Detection and Classification of CCD Defects with an Artificial Neural Network. Feeney et al. 2005, Automated Detection of Classical Novae with Neural Networks. Wadadekar 2005, Estimating Photometric Redshifts Using Support Vector Machines. Bazell & Miller 2005, Class Discovery in Galaxy Classification. Bazell, Miller, & SubbaRao 2006, Objective Subclass Determination of Sloan Digital Sky Survey Spectroscopically Unclassified Objects. Ferreras et al. 2006, A Principal Component Analysis approach to the Star Formation History of Elliptical Galaxies in Compact Groups. Way & Srivastava 2006, Novel Methods for Predicting Photometric Redshifts from Broadband Photometry Using Virtual Sensors. Carliles et al. 2007, Photometric Redshift Estimation on SDSS Data Using Random Forests.

32 2008 NVO Summer School32 OUTLINE Scientific Databases Some key astronomy problems Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

33 2008 NVO Summer School33 Some Data Mining Software & Projects General data mining software packages: –Weka (Java): –Weka4WS (Grid-enabled): –RapidMiner: Astronomy-specific software and/or user clients: VO-Neural: AstroWeka: OpenSkyQuery: ALADIN: MIRAGE: AstroBox: Astronomical and/or Scientific Data Mining Projects: GRIST: ClassX: LCDM: F-MASS: NCDM:

34 2008 NVO Summer School34 Weka: Weka is in your NVOSS software distribution. Weka is a collection of open source machine learning algorithms for data mining tasks. Weka algorithms can either be applied directly to a dataset or called from your own Java code. Weka comes with its own GUI. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization.

35 2008 NVO Summer School35 AstroWeka:

36 2008 NVO Summer School ALADIN: 36

37 2008 NVO Summer School MIRAGE: Java Package for exploratory data analysis (EDA), correlation mining, and interactive pattern discovery. 37

38 2008 NVO Summer School38 OUTLINE Scientific Databases Some key astronomy problems Astronomy Data Mining examples Suggested Reading Some Data Mining Software Summary

39 2008 NVO Summer School39 Science is Knowledge Work Knowledge Discovery is the central theme of science. Knowledge Discovery in Databases (KDD) is the killer app for large scientific databases. Therefore, KDD (i.e., Data Mining) is an essential tool, since big-data science is here to stay (at petabytes and beyond). Data Information Knowledge

40 2008 NVO Summer School40 Scientific Knowledge Discovery

41 2008 NVO Summer School41 Heliophysics Space Weather Example

42 2008 NVO Summer School42 Sun-Earth Space Environment – Rich Source of Heliophysical Phenomena

43 2008 NVO Summer School43 Multi-point Observations and Models of Space Plasmas Deliver a Deluge of Physical Measurements

44 2008 NVO Summer School44

45 2008 NVO Summer School45 Heliophysics Space Weather Example CME = Coronal Mass Ejection SEP = Solar Energetic Particle

46 2008 NVO Summer School46 Data Mining: It is more than just connecting the dots Reference:

47 2008 NVO Summer School47 Sample Astronomy Data Mining Application Ideas for your Projects –Neural Network for Pixel Classification: Event Detection and Prediction (e.g., Supernova or Cosmic-ray hit?) –Bayesian Network for Object Classification (star or galaxy?) –PCA for finding Fundamental Planes of Galaxy Parameters –PCA (weakest component) for Outlier Detection: anomalies, novel discoveries, new objects –Link Analysis (Association Mining) for Causal Event Detection (e.g., linking optical transients with gamma-ray events) –Clustering analysis: Spatial, Temporal, or any scientific database parameters –Markov models: Temporal mining, classification, and prediction from time series data


Download ppt "2008 NVO Summer School11 Scientific Data Mining in Astronomy Kirk D. Borne George Mason University T HE."

Similar presentations


Ads by Google