Jeremy Wyatt Thanks to Gavin Brown

Slides:

Advertisements

Similar presentations

Naïve Bayes Classification

Advertisements

Evaluating Classifiers

Chapter 4 Pattern Recognition Concepts: Introduction & ROC Analysis.

Caroline Rougier, Jean Meunier, Alain St-Arnaud, and Jacqueline Rousseau IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 21, NO. 5,

Learning Algorithm Evaluation

Evaluation of segmentation. Example Reference standard & segmentation.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

Curva ROC figuras esquemáticas Curva ROC figuras esquemáticas Prof. Ivan Balducci FOSJC / Unesp.

Cos 429: Face Detection (Part 2) Viola-Jones and AdaBoost Guest Instructor: Andras Ferencz (Your Regular Instructor: Fei-Fei Li) Thanks to Fei-Fei Li,

The University of Texas at Austin, CS 395T, Spring 2008, Prof. William H. Press 1 Computational Statistics with Application to Bioinformatics Prof. William.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

Model Evaluation Metrics for Performance Evaluation

Cost-Sensitive Classifier Evaluation Robert Holte Computing Science Dept. University of Alberta Co-author Chris Drummond IIT, National Research Council,

1 Learning to Detect Objects in Images via a Sparse, Part-Based Representation S. Agarwal, A. Awan and D. Roth IEEE Transactions on Pattern Analysis and.

1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.

Cbio course, spring 2005, Hebrew University (Alignment) Score Statistics.

1 The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller MI – 25. oktober 2007 Kresten Toftgaard Andersen.

Determine whether each curve below is the graph of a function of x. Select all answers that are graphs of functions of x:

INTRODUCTION TO Machine Learning 3rd Edition

Evaluating Classifiers

Identifying Computer Graphics Using HSV Model And Statistical Moments Of Characteristic Functions Xiao Cai, Yuewen Wang.

1 CSI 5388: ROC Analysis (Based on ROC Graphs: Notes and Practical Considerations for Data Mining Researchers by Tom Fawcett, (Unpublished) January 2003.

Chapter 4 Pattern Recognition Concepts continued.

Basic statistics 11/09/13.

Error estimation Data Mining II Year Lluís Belanche Alfredo Vellido.

Performance measurement. Must be careful what performance metric we use For example, say we have a NN classifier with 1 output unit, and we code ‘1 =

Copyright © 2003, SAS Institute Inc. All rights reserved. Cost-Sensitive Classifier Selection Ross Bettinger Analytical Consultant SAS Services.

Data Analysis 1 Mark Stamp. Topics  Experimental design o Training set, test set, n-fold cross validation, thresholding, imbalance, etc.  Accuracy o.

Sensitivity Sensitivity answers the following question: If a person has a disease, how often will the test be positive (true positive rate)? i.e.: if the.

GA-Based Feature Selection and Parameter Optimization for Support Vector Machine Cheng-Lung Huang, Chieh-Jen Wang Expert Systems with Applications, Volume.

Evaluating What’s Been Learned. Cross-Validation Foundation is a simple idea – “ holdout ” – holds out a certain amount for testing and uses rest for.

Experiments in Machine Learning COMP24111 lecture 5 Accuracy (%) A BC D Learning algorithm.

Classification Performance Evaluation. How do you know that you have a good classifier? Is a feature contributing to overall performance? Is classifier.

A Comparative Evaluation of Three Skin Color Detection Approaches Dennis Jensch, Daniel Mohr, Clausthal University Gabriel Zachmann, University of Bremen.

MEASURES OF TEST ACCURACY AND ASSOCIATIONS DR ODIFE, U.B SR, EDM DIVISION.

BAGGING ALGORITHM, ONLINE BOOSTING AND VISION Se – Hoon Park.

F. Provost and T. Fawcett. Confusion Matrix 2Bitirgen - CS678.

Stable Multi-Target Tracking in Real-Time Surveillance Video

Evaluating Results of Learning Blaž Zupan

Computational Intelligence: Methods and Applications Lecture 16 Model evaluation and ROC Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

Model Evaluation l Metrics for Performance Evaluation –How to evaluate the performance of a model? l Methods for Performance Evaluation –How to obtain.

Preventing Overfitting Problem: We don’t want to these algorithms to fit to ``noise’’ Reduced-error pruning : –breaks the samples into a training set and.

Machine Learning Tutorial-2. Recall, Precision, F-measure, Accuracy Ch. 5.

Classification Evaluation. Estimating Future Accuracy Given available data, how can we reliably predict accuracy on future, unseen data? Three basic approaches.

1 Performance Measures for Machine Learning. 2 Performance Measures Accuracy Weighted (Cost-Sensitive) Accuracy Lift Precision/Recall –F –Break Even Point.

Evaluating Classification Performance

Professor William H. Press, Department of Computer Science, the University of Texas at Austin1 Opinionated in Statistics by Bill Press Lessons #50 Binary.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis.

Chapter 5: Credibility. Introduction Performance on the training set is not a good indicator of performance on an independent set. We need to predict.

Timothy Wiemken, PhD MPH Assistant Professor Division of Infectious Diseases Diagnostic Tests.

Supervise Learning. 2 What is learning? “Learning denotes changes in a system that... enable a system to do the same task more efficiently the next time.”

Performance Evaluation 02/15/17

Evaluating Results of Learning

9. Credibility: Evaluating What’s Been Learned

Machine Learning Week 10.

Data Mining Classification: Alternative Techniques

Features & Decision regions

Improved Rooftop Detection in Aerial Images with Machine Learning

Notes Over 2.1 Function {- 3, - 1, 1, 2 } { 0, 2, 5 }

Experiments in Machine Learning

Learning Algorithm Evaluation

INTRODUCTION TO Machine Learning

Model Evaluation and Selection

Computational Intelligence: Methods and Applications

Roc curves By Vittoria Cozza, matr

Presentation transcript:

Jeremy Wyatt Thanks to Gavin Brown ROC Analysis Jeremy Wyatt Thanks to Gavin Brown

Evaluating vision algorithms You have designed a new edge detection technique. You give it to me, and I try it on my image dataset where the task is to predict whether the scene contains a chair or not. I tell you that it achieved 95% accuracy on my data. Is your technique a success?

Types of errors But suppose that The 95% is the correctly classified pixels Only 5% of the pixels are actually edges It misses all the edge pixels How do we count the effect of different types of error?

Types of errors Prediction Edge Not edge True Positive False Negative Not Edge Edge Ground Truth True Positive False Negative True Negative

True Positive False Negative Two parts to each: whether you got it correct or not, and what you guessed. For example for a particular pixel, our guess might be labelled… True Positive Did we get it correct? True, we did get it correct. What did we say? We said ‘positive’, i.e. edge. or maybe it was labelled as one of the others, maybe… False Negative What did we say? We said ‘negative, i.e. not edge. Did we get it correct? False, we did not get it correct.

Sensitivity and Specificity Count up the total number of each label (TP, FP, TN, FN) over a large dataset. In ROC analysis, we use two statistics: TP Can be thought of as the likelihood of spotting a positive case when presented with one. Or… the proportion of edges we find. Sensitivity = TP+FN Can be thought of as the likelihood of spotting a negative case when presented with one. Or… the proportion of non-edges that we find TN Specificity = TN+FP

1 1 60 30 80 20 TP TN Sensitivity = = ? Specificity = = ? TP+FN TN+FP Prediction 1 60+30 = 90 cases in the dataset were class 1 (edge) 1 60 30 Ground Truth 80+20 = 100 cases in the dataset were class 0 (non-edge) 80 20 90+100 = 190 examples (pixels) in the data overall

The ROC space Note Sensitivity 1 - Specificity 1.0 This is edge detector A This is edge detector B Sensitivity 0.0 1.0 1 - Specificity Note

The ROC Curve Draw a ‘convex hull’ around many points: Sensitivity This point is not on the convex hull. 1 - Specificity

ROC Analysis All the optimal detectors lie on the convex hull. Which of these is best depends on the ratio of edges to non-edges, and the different cost of misclassification Any detector on this side can lead to a better detector by flipping its output. sensitivity 1 - specificity Take-home point : You should always quote sensitivity and specificity for your algorithm, if possible plotting an ROC graph. Remember also though, any statistic you quote should be an average over a suitable range of tests for your algorithm.