Flexible Metric NN Classification based on Friedman (1995) David Madigan.

Slides:

Advertisements

Similar presentations

Principles of Density Estimation

Advertisements

Data Mining Classification: Alternative Techniques

Data Mining Classification: Alternative Techniques

1 CS 391L: Machine Learning: Instance Based Learning Raymond J. Mooney University of Texas at Austin.

Regression Analysis Module 3. Regression Regression is the attempt to explain the variation in a dependent variable using the variation in independent.

Indian Statistical Institute Kolkata

Lazy vs. Eager Learning Lazy vs. eager learning

Lecture 3 Nonparametric density estimation and classification

Navneet Goyal. Instance Based Learning  Rote Classifier  K- nearest neighbors (K-NN)  Case Based Resoning (CBR)

Pattern recognition Professor Aly A. Farag

Bayesian Learning Rong Jin. Outline MAP learning vs. ML learning Minimum description length principle Bayes optimal classifier Bagging.

MACHINE LEARNING 9. Nonparametric Methods. Introduction Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 

Lecture Notes for CMPUT 466/551 Nilanjan Ray

Carla P. Gomes CS4700 CS 4700: Foundations of Artificial Intelligence Carla P. Gomes Module: Nearest Neighbor Models (Reading: Chapter.

Statistical Methods Chichang Jou Tamkang University.

Statistical Decision Theory, Bayes Classifier

Comparison of Instance-Based Techniques for Learning to Predict Changes in Stock Prices iCML Conference December 10, 2003 Presented by: David LeRoux.

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.

CES 514 – Data Mining Lec 9 April 14 Mid-term k nearest neighbor.

1 Ensembles of Nearest Neighbor Forecasts Dragomir Yankov, Eamonn Keogh Dept. of Computer Science & Eng. University of California Riverside Dennis DeCoste.

Classification Based in part on Chapter 10 of Hand, Manilla, & Smyth and Chapter 7 of Han and Kamber David Madigan.

Chapter 4 (part 2): Non-Parametric Classification

1 An Introduction to Nonparametric Regression Ning Li March 15 th, 2004 Biostatistics 277.

General Mining Issues a.j.m.m. (ton) weijters Overfitting Noise and Overfitting Quality of mined models (some figures are based on the ML-introduction.

Nearest Neighbor Classifiers other names: –instance-based learning –case-based learning (CBL) –non-parametric learning –model-free learning.

CS Instance Based Learning1 Instance Based Learning.

More Machine Learning Linear Regression Squared Error L1 and L2 Regularization Gradient Descent.

Pattern Recognition: Baysian Decision Theory Charles Tappert Seidenberg School of CSIS, Pace University.

Machine Learning1 Machine Learning: Summary Greg Grudic CSCI-4830.

Using Neural Networks in Database Mining Tino Jimenez CS157B MW 9-10:15 February 19, 2009.

COMMON EVALUATION FINAL PROJECT Vira Oleksyuk ECE 8110: Introduction to machine Learning and Pattern Recognition.

Overview of Supervised Learning Overview of Supervised Learning2 Outline Linear Regression and Nearest Neighbors method Statistical Decision.

Empirical Research Methods in Computer Science Lecture 7 November 30, 2005 Noah Smith.

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.

Chapter 7 Point Estimation of Parameters. Learning Objectives Explain the general concepts of estimating Explain important properties of point estimators.

Lecture3 – Overview of Supervised Learning Rice ELEC 697 Farinaz Koushanfar Fall 2006.

Classification Problem GivenGiven Predict class label of a given queryPredict class label of a given query

Chapter 6 – Three Simple Classification Methods © Galit Shmueli and Peter Bruce 2008 Data Mining for Business Intelligence Shmueli, Patel & Bruce.

1Ellen L. Walker Category Recognition Associating information extracted from images with categories (classes) of objects Requires prior knowledge about.

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.

Chapter1: Introduction Chapter2: Overview of Supervised Learning

Chapter 20 Classification and Estimation Classification – Feature selection Good feature have four characteristics: –Discrimination. Features.

Chapter 13 (Prototype Methods and Nearest-Neighbors )

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition Objectives: Occam’s Razor No Free Lunch Theorem Minimum.

Regress-itation Feb. 5, Outline Linear regression – Regression: predicting a continuous value Logistic regression – Classification: predicting a.

Information Retrieval and Organisation Chapter 14 Vector Space Classification Dell Zhang Birkbeck, University of London.

Eick: kNN kNN: A Non-parametric Classification and Prediction Technique Goals of this set of transparencies: 1.Introduce kNN---a popular non-parameric.

LECTURE 05: CLASSIFICATION PT. 1 February 8, 2016 SDS 293 Machine Learning.

Debrup Chakraborty Non Parametric Methods Pattern Recognition and Machine Learning.

Instance-Based Learning Evgueni Smirnov. Overview Instance-Based Learning Comparison of Eager and Instance-Based Learning Instance Distances for Instance-Based.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Computational Intelligence: Methods and Applications Lecture 14 Bias-variance tradeoff – model selection. Włodzisław Duch Dept. of Informatics, UMK Google:

Chapter 7. Classification and Prediction

Classification 3 (Nearest Neighbor Classifier)

CSSE463: Image Recognition Day 11

Information Retrieval

The Elements of Statistical Learning

Overview of Supervised Learning

Nearest-Neighbor Classifiers

Mathematical Foundations of BME Reza Shadmehr

Instance Based Learning

Contact: Machine Learning – (Linear) Regression Wilson Mckerrow (Fenyo lab postdoc) Contact:

Parametric Methods Berlin Chen, 2005 References:

Nonparametric density estimation and classification

Multivariate Methods Berlin Chen

Multivariate Methods Berlin Chen, 2005 References:

CSSE463: Image Recognition Day 11

CSSE463: Image Recognition Day 11

Memory-Based Learning Instance-Based Learning K-Nearest Neighbor

Presentation transcript:

Flexible Metric NN Classification based on Friedman (1995) David Madigan

Nearest-Neighbor Methods k-NN assigns an unknown object to the most common class of its k nearest neighbors Choice of k? (bias-variance tradeoff again) Choice of metric? Need all the training to be present to classify a new point (“lazy methods”) Surprisingly strong asymptotic results (e.g. no decision rule is more than twice as accurate as 1-NN)

Suppose a Regression Surface Looks like this: Flexible-metric NN Methods try to capture this idea… want this not this

FMNN Predictors may not all be equally relevant for classifying a new object Furthermore, this differential relevance may depend on the location of the new object FMNN attempts to model this phenomenon

Local Relevance Consider an arbitrary function f on R p If no values of x are known, have: Suppose x i =z, then:

Local Relevance cont. The improvement in squared error provided by knowing x i is: I 2 i (z) reflects the importance of the ith variable on the variation of f(x) at x i =z

Local Relevance cont. Now consider an arbitrary point z=(z 1,…,z p ) The relative importance of x i to the variation of f at x=z is: R 2 i (z)=0 when f(x) is independent of x i at z R 2 i (z)=1 when f(x) depends only on x i at z

Estimation Recall:

On To Classification For J-class classification have {y j }, j=1,…,J output variables, y j  {0,1},  y j =1. Can compute: Technical point: need to weight the observations to rectify unequal variances

The Machete Start with all data points R 0 Compute Then: Continue until R i contains K points M 1 th order statistic

Results on Artificial Data

Results on Real Data