Cross Validation False Negatives / Negatives

Slides:

Advertisements

Similar presentations

Machine Learning: Methodology Chapter Some material adopted from notes by Chuck Dyer.

Advertisements

...visualizing classifier performance in R Tobias Sing, Ph.D. (joint work with Oliver Sander) Modeling & Simulation Novartis Pharma AG 3 rd BaselR meeting.

...visualizing classifier performance Tobias Sing Dept. of Modeling & Simulation Novartis Pharma AG Joint work with Oliver Sander (MPI for Informatics,

Authorship Verification Authorship Identification Authorship Attribution Stylometry.

Learning Algorithm Evaluation

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

The University of Texas at Austin, CS 395T, Spring 2008, Prof. William H. Press 1 Computational Statistics with Application to Bioinformatics Prof. William.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

Model Evaluation Metrics for Performance Evaluation

Classification by Machine Learning Approaches - Exercise Solution Michael J. Kerner – Center for Biological Sequence.

© Vipin Kumar CSci 8980 Fall CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance Computing Research Center Department of Computer.

1 I256: Applied Natural Language Processing Marti Hearst Sept 27, 2006.

Tutorial 2 LIU Tengfei 2/19/2009. Contents Introduction TP, FP, ROC Precision, recall Confusion matrix Other performance measures Resource.

Jeremy Wyatt Thanks to Gavin Brown

ROC & AUC, LIFT ד"ר אבי רוזנפלד.

Evaluation – next steps

CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 07: Cost-Sensitive Measures.

Evaluating Classifiers

SVMLight SVMLight is an implementation of Support Vector Machine (SVM) in C. Download source from :

SPH 247 Statistical Analysis of Laboratory Data May 19, 2015SPH 247 Statistical Analysis of Laboratory Data1.

Evaluation – next steps

Classification II. 2 Numeric Attributes Numeric attributes can take many values –Creating branches for each value is not ideal The value range is usually.

1 Evaluating Model Performance Lantz Ch 10 Wk 5, Part 2 Right – Graphing is often used to evaluate results from different variations of an algorithm. Depending.

Performance measurement. Must be careful what performance metric we use For example, say we have a NN classifier with 1 output unit, and we code ‘1 =

הערכת טיב המודל F-Measure, Kappa, Costs, MetaCost ד " ר אבי רוזנפלד.

Data Analysis 1 Mark Stamp. Topics  Experimental design o Training set, test set, n-fold cross validation, thresholding, imbalance, etc.  Accuracy o.

איחזור מידע אלגוריתמי חיפוש PageRank ד " ר אבי רוזנפלד.

Data Mining: Classification & Predication Hosam Al-Samarraie, PhD. Centre for Instructional Technology & Multimedia Universiti Sains Malaysia.

Evaluating What’s Been Learned. Cross-Validation Foundation is a simple idea – “ holdout ” – holds out a certain amount for testing and uses rest for.

Experiments in Machine Learning COMP24111 lecture 5 Accuracy (%) A BC D Learning algorithm.

There are many kinds of animals. Animals can be put into groups based upon their characteristics. Some animals do not have backbones.

Classification Performance Evaluation. How do you know that you have a good classifier? Is a feature contributing to overall performance? Is classifier.

Animal Groups Your Name.

Computational Intelligence: Methods and Applications Lecture 16 Model evaluation and ROC Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

Model Evaluation l Metrics for Performance Evaluation –How to evaluate the performance of a model? l Methods for Performance Evaluation –How to obtain.

Practical Issues of Classification Underfitting and Overfitting –Training errors –Generalization (test) errors Missing Values Costs of Classification.

WEKA Machine Learning Toolbox. You can install Weka on your computer from

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

Machine Learning Tutorial-2. Recall, Precision, F-measure, Accuracy Ch. 5.

Classification Evaluation. Estimating Future Accuracy Given available data, how can we reliably predict accuracy on future, unseen data? Three basic approaches.

1 Performance Measures for Machine Learning. 2 Performance Measures Accuracy Weighted (Cost-Sensitive) Accuracy Lift Precision/Recall –F –Break Even Point.

Quiz 1 review. Evaluating Classifiers Reading: T. Fawcett paper, link on class website, Sections 1-4 Optional reading: Davis and Goadrich paper, link.

Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis.

CS 2750: Machine Learning The Bias-Variance Tradeoff Prof. Adriana Kovashka University of Pittsburgh January 13, 2016.

ROC curve estimation. Index Introduction to ROC ROC curve Area under ROC curve Visualization using ROC curve.

Performance Indices for Binary Classification 張智星 (Roger Jang) 多媒體資訊檢索實驗室台灣大學資訊工程系.

Classification Cheng Lei Department of Electrical and Computer Engineering University of Victoria April 24, 2015.

Evaluating Classifiers. Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)

Information Organization: Evaluation of Classification Performance.

Evaluating Classifiers

Evaluation – next steps

Machine Learning: Methodology Chapter

Performance Evaluation 02/15/17

Measuring Success in Prediction

Machine Learning Week 10.

Data Mining Classification: Alternative Techniques

Features & Decision regions

Variation and Classification

Variation and Classification

Experiments in Machine Learning

Variation and Classification

Model Evaluation and Selection

Evaluating Models Part 1

Variation and Classification

Assignment 1: Classification by K Nearest Neighbors (KNN) technique

Variation and Classification

Machine Learning: Methodology Chapter

ECE – Pattern Recognition Lecture 8 – Performance Evaluation

Presentation transcript:

Cross Validation False Negatives / Negatives ד"ר אבי רוזנפלד

מוטיבציה– מה רואים פה??? Correctly Classified Instances 93 92.0792 % Incorrectly Classified Instances 8 7.9208 % === Detailed Accuracy By Class === TP Rate FP Rate Precision Recall F-Measure ROC Area Class 1 0 1 1 1 1 mammal 1 0 1 1 1 1 bird 0.6 0.01 0.75 0.6 0.667 0.793 reptile 1 0.011 0.929 1 0.963 0.994 fish 0.75 0 1 0.75 0.857 0.872 amphibian 0.625 0.032 0.625 0.625 0.625 0.92 insect 0.8 0.033 0.727 0.8 0.762 0.986 invertebrate === Confusion Matrix === a b c d e f g <-- classified as 41 0 0 0 0 0 0 | a = mammal 0 20 0 0 0 0 0 | b = bird 0 0 3 1 0 1 0 | c = reptile 0 0 0 13 0 0 0 | d = fish 0 0 1 0 3 0 0 | e = amphibian 0 0 0 0 0 5 3 | f = insect 0 0 0 0 0 2 8 | g = invertebrate

הגדרות – נניח שיש 2 קטגוריות (P,N) נוסחה משמעות מדד TP / (TP + FP) בכמה אחוזים דייקתי בקטגוריה מסוימת (מתוך אלו שאמרתי הם בקטגוריה) Precision TP / (TP + FN) בכמה אחוזים דייקתי בכלל במופעים בקטגוריה Recall (TP + TN) / (הכול) הדיוק הכללי Accuracy TP / P Recall(P) True Positive Rate (כמה מצאתי בצד החיובי) Sensitivity Also TPR (True Positive Rate) TN / N Recall(N) True Negative Rate (כמה מצאתי בצד השלילי) Specificity FP / (TN + FP) 1 – Recall (N) FP = False Positive Rate = 1 - Specificity

Confusion Matrix פה עם 2 קטגוריות Machine Learning methods usually minimize FP+FN TPR (True Positive Rate): TP / (TP + FN) = Recall(Positive) FPR (False Positive Rate): FP / (TN + FP) = 1 – Recall(Negative) Predicted class Yes No Actual class TP: True positive FN: False negative FP: False positive TN: True negative

False Positives / Negatives איך אני מסווג Confusion matrix 1 Confusion matrix 2 P N 20 10 30 90 P N 10 20 15 105 FN Actual Actual FP Predicted Predicted Precision (P) = 20 / 50 Recall (P) = 20 / 30 FP = False Positive = Type 1 Error (False Alarm) FN = False Negative = Type 2 Error (Miss) מה הRECALL וPRECISION פה בצד של P? בצד של N?

Example Precision (A) = 50% (500/1000) Recall (A) = 83% (500/600) Predicted Label Positive (A) Negative (B) Known Label 500 100 10,000 Precision (A) = 50% (500/1000) Recall (A) = 83% (500/600) Accuracy = 95% (10500/11100) Sensitivity = 83% Specificity = Recall (B) = 10,000/10,500

הגדרות נוסחה משמעות מדד TP / (TP + FP) בכמה אחוזים דייקתי בקטגוריה מסוימת (מתוך אלו שאמרתי הם בקטגוריה) Precision TP / (TP + FN) בכמה אחוזים דייקתי בכלל במופעים בקטגוריה Recall 2*precision*recall/ precision+recall Harmonic mean of precision and recall F-Measure (TP + TN) / (הכול) הדיוק הכללי Accuracy

False Positives / Negatives Confusion matrix 1 Confusion matrix 2 P N 20 10 30 90 P N 10 20 15 105 FN Actual Actual FP Predicted Predicted Precision (P) = 20 / 50 = 0.4 Recall (P) = 20 / 30 = 0.666 F-measure=2*.4*.666/1.0666=.5

דוגמא עם כמה קטגוריות 27 animals — 8 cats, 6 dogs, and 13 rabbits Confusion Matrix: Predicted class Actual class Cat Dog Rabbit Cat 5 3 0 Dog 2 3 1 Rabbit 0 2 11 בקטגוריית Cat יש 3 False Negatives (שהם מסויגים ככלבים), ועוד 2 False Positives של כלבים המסווגים כמו כחתולים. בקטגוריית Dog יש 3 False Negatives (2 חתולים וארנבת), ועוד 5 False Positives (3 חתולים ו2 ארנבות). Recall (Dog) = 3/6, Precision(Dog) = 3/8 What are the Recall and Precision for the other categories?

דוגמא פשוטה מWEKA === Summary === Correctly Classified Instances 320 66.39 % Incorrectly Classified Instances 162 33.61 % === Detailed Accuracy By Class === Precision Recall 0.664 1 0 0 === Confusion Matrix === a b <-- classified as 320 0 | a = FALSE -> Precision (A) = 320/582), Recall = 320/320 162 0 | b = TRUE -> Precision (B) = Recall (B) = 0 שימו לב! זה חוק של ZERO בWEKA (לסווג הכול לפי הרוב) שנראה בתרגיל

דוגמא נוספת (מהחיות – (zoo.arff Correctly Classified Instances 93 92.0792 % Incorrectly Classified Instances 8 7.9208 % === Detailed Accuracy By Class === TP Rate FP Rate Precision Recall F-Measure ROC Area Class 1 0 1 1 1 1 mammal 1 0 1 1 1 1 bird 0.6 0.01 0.75 0.6 0.667 0.793 reptile 1 0.011 0.929 1 0.963 0.994 fish 0.75 0 1 0.75 0.857 0.872 amphibian 0.625 0.032 0.625 0.625 0.625 0.92 insect 0.8 0.033 0.727 0.8 0.762 0.986 invertebrate === Confusion Matrix === a b c d e f g <-- classified as 41 0 0 0 0 0 0 | a = mammal 0 20 0 0 0 0 0 | b = bird 0 0 3 1 0 1 0 | c = reptile 0 0 0 13 0 0 0 | d = fish 0 0 1 0 3 0 0 | e = amphibian 0 0 0 0 0 5 3 | f = insect 0 0 0 0 0 2 8 | g = invertebrate Recall (Invertebrate) = 8/10 = 0.8, Precision = 8/11 = 0.727 תרגיל: לחשב את TP Rate, FP Rate, Precision, Recall, F-Measure לכל הקטגוריות.