Presentation is loading. Please wait.

Presentation is loading. Please wait.

Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C.

Published byElaine Griffith Modified over 8 years ago

Similar presentations

Presentation on theme: "Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C."— Presentation transcript:

1 Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C. Computer Vision Chapter 4 Statistical Pattern Recognition Yeh Jin Long Dr. Fuh Chiou Shann

2 DC & CV Lab. CSIE NTU Summary Introduction. Bayesian approach. Maximin decision rule. Misidentification and false-alarm error rates. Nearest neighbor rule. Construction of decision trees. Estimation of decision rules error. Neural network.

3 DC & CV Lab. CSIE NTU Introduction

4 DC & CV Lab. CSIE NTU Introduction(cont.) Units: Image regions and projected segments Each unit has an associated measurement vector data set : { circular arc, a hole, … } Using decision rule to assign unit to class or category optimally

5 DC & CV Lab. CSIE NTU Introduction (Cont.) Feature selection and extraction techniques Have no hole Have hole Decision rule construction techniques word O - number 0 Techniques for estimating decision rule error

6 DC & CV Lab. CSIE NTU Simple Pattern Discrimination Also called pattern identification process A unit is observed or measured A category assignment is made that names or classifies the unit as a type of object The category assignment is made only on observed measurement (pattern)

7 DC & CV Lab. CSIE NTU Simple Pattern Discrimination (cont.) a: assigned category from a set of categories C t: true category identification from C d: observed measurement from a set of measurements D (t, a, d): event of classifying the observed unit P(t, a, d): probability of the event (t, a, d)

8 DC & CV Lab. CSIE NTU e(t, a): economic gain/utility with true category t and assigned category a A mechanism to evaluate a decision rule Identity gain matrix Economic Gain Matrix

9 DC & CV Lab. CSIE NTU An Instance

10 DC & CV Lab. CSIE NTU Another Instance P(g, g): probability of true good, assigned good, P(g, b): probability of true good, assigned bad,... e(g, g): economic consequence for event (g, g), … e positive: profit consequence e negative: loss consequence

11 DC & CV Lab. CSIE NTU Another Instance (cont.)

12 DC & CV Lab. CSIE NTU Another Instance (cont.)

13 DC & CV Lab. CSIE NTU Another Instance (cont.) Fraction of good objects manufactured P(g) = P(g, g) + P(g, b) Fraction of bad objects manufactured P(b) = P(b, g) + P(b, b) Expected profit per object E =

14 DC & CV Lab. CSIE NTU Conditional Probability P(b|g): false-alarm rate P(g|b): misdetection rate

15 DC & CV Lab. CSIE NTU Conditional Probability (cont.) Another formula for expected profit per object E = =P(g|g)P(g)e(g,g)+P(b|g)P(g)e(g,b) + P(g|b)P(b)e(b,g)+P(b|b)P(b)e(b,b)

16 DC & CV Lab. CSIE NTU Example 4.1 P(g) = 0.95, P(b) = 0.05 Table 4.4: Machine performance P(g) = 0.95P(b) = 0.05Detected State GoodBad True State GoodP(g|g) = 0.8P(b|g) = 0.2 BadP(g|b) = 0.1P(b|b) = 0.9 E = Table 4.5: Economic consequence Detected State GoodBad True State Goode(g|g) = $2000 e(g|b) = - $100 Bad e(b|g) = - $10.000e(b|b) = - $100

17 DC & CV Lab. CSIE NTU Example 4.1 (cont.)

18 DC & CV Lab. CSIE NTU Example 4.2 P(g) = 0.95, P(b) = 0.05 Table 4.6: Machine performance P(g) = 0.95P(b) = 0.05Detected State GoodBad True State GoodP(g|g) = 0.85P(b|g) = 0.15 BadP(g|b) = 0.12P(b|b) = 0.88 E = Table 4.7: Economic consequence Detected State GoodBad True State Goode(g|g) = $2000 e(g|b) = - $100 Bad e(b|g) = - $10.000e(b|b) = - $100

19 DC & CV Lab. CSIE NTU Example 4.2 (cont.)

20 DC & CV Lab. CSIE NTU Decision Rule Construction (t, a): summing (t, a, d) on every measurements d Therefore, Average economic gain

21 DC & CV Lab. CSIE NTU Decision Rule Construction (cont.)

22 DC & CV Lab. CSIE NTU Decision Rule Construction (cont.) We can use identity matrix as the economic gain matrix to compute the probability of correct assignment:

23 DC & CV Lab. CSIE NTU Fair Game Assumption Decision rule uses only measurement data in assignment; the nature and the decision rule are not in collusion In other words, P(a| t, d) = P(a| d)

24 DC & CV Lab. CSIE NTU Fair Game Assumption (cont.) From the definition of conditional probability

25 DC & CV Lab. CSIE NTU P(t, a, d) = P(a| t, d)*P(t,d) //By conditional probability = P(a| d)*P(t,d) //By fair game assumption By definition, = Fair Game Assumption (cont.)

26 DC & CV Lab. CSIE NTU Deterministic Decision Rule We use the notation f(a|d) to completely define a decision rule; f(a|d) presents all the conditional probability associated with the decision rule A deterministic decision rule: Decision rules which are not deterministic are called probabilistic/nondeterministic/stochastic

27 DC & CV Lab. CSIE NTU Previous formula By // By conditional probability and //By p.23 => Expected Value on f(a|d)

28 DC & CV Lab. CSIE NTU Expected Value on f(a|d) (cont.)

29 DC & CV Lab. CSIE NTU Bayes Decision Rules Maximize expected economic gain Satisfy Constructing f

30 DC & CV Lab. CSIE NTU Measurement P(c,d)d1d1 d2d2 d3d3 True categoryc1c1 0.120.180.3 Identificationc2c2 0.20.160.4 Measurement f(a|d)d1d1 d2d2 d3d3 Assigned categoryc1c1 Identificationc2c2 e Assigned c1c1 c2c2 Truec1c1 c2c2 E[e; f] = Figure 4.2 Calculation of the Bayes decision rule and calculation of the expected gain. E[e; f] = Σ {Σ f(a | d) [Σ e(t, a)P(t,d)] } d € Da € Ct € C

31 DC & CV Lab. CSIE NTU Bayes Decision Rules (cont.)

32 DC & CV Lab. CSIE NTU Assigned c1c1 c2c2 True c1c1 c2c2

33 DC & CV Lab. CSIE NTU Bayes Decision Rules (cont.) + +

34 DC & CV Lab. CSIE NTU Continuous Measurement For the same example, try the continuous density function of the measurements: and Measurements lie in the close interval [0,1] Prove that they are indeed density function

35 DC & CV Lab. CSIE NTU Continuous Measurement (cont.) Suppose that the prior probability of is and the prior probability of is = When, a Bayes decision rule will assign an observed unit to t1, which implies =>

36 DC & CV Lab. CSIE NTU DC & CV Lab. CSIE NTU Continuous Measurement (cont.) E[e;f] =.805 >.68, the continuous measurement has larger expected economic gain than discrete

37 DC & CV Lab. CSIE NTU Continuous Measurement (cont.).805 >.68, the continuous measurement has larger expected economic gain than discrete

38 DC & CV Lab. CSIE NTU Prior Probability The Bayes rule: Replace with The Bayes rule can be determined by assigning any categories that maximizes

39 DC & CV Lab. CSIE NTU Economic Gain Matrix Identity matrix Incorrect loses 1 A more balanced instance

40 Economic Gain Matrix Suppose are two different economic gain matrix with relationship According to the construction rule. Given a measurement d, Because We then got DC & CV Lab. CSIE NTU

41 DC & CV Lab. CSIE NTU Maximin Decision Rule Maximizes average gain over worst prior probability

42 DC & CV Lab. CSIE NTU Example 4.3

43 DC & CV Lab. CSIE NTU Example 4.3 (cont.) P(d | c)d1d1 d2d2 d3d3 c1c1.2.3.5 c2c2.4.1

44 DC & CV Lab. CSIE NTU Example 4.3 (cont.)

45 E[e;f] P(c1) Maximin gain for deterministic rules Maximin gain Conditional gains E[e|c1;f]E[e|c2;f] Decision rule f110 f2.5.1 f3.7.4 f4.2.5 f5.8.5 f6.3.6 f7.5.9 f801

46 DC & CV Lab. CSIE NTU Example 4.3 (cont.)

47 DC & CV Lab. CSIE NTU Example 4.3 (cont.) The lowest Bayes gain is achieved when The lowest gain is 0.6714

48 DC & CV Lab. CSIE NTU Example 4.3 (cont.)

49 DC & CV Lab. CSIE NTU P( d|c )d1d1 d2d2 c1c1 3/41/4 c2c2 1/87/8 ec1c1 c2c2 c1c1 c2c2 020/7 Example 4.4 d1d1 d2d2 E[e|c1;f]E[e|c2;f] f1c1c1 c1c1 f2c1c1 c2c2 f3c2c2 c1c1 f4c2c2 c2c2

50 DC & CV Lab. CSIE NTU Example 4.4 P( d|c )d1d1 d2d2 c1c1 3/41/4 c2c2 1/87/8 ec1c2 c1c1 c2c2 020/7 d1d1 d2d2 E[e|c1;f]E[e|c2;f] f1c1c1 c1c1 0 f2c1c1 c2c2 f3c2c2 c1c1 1/65/14 f4c2c2 c2c2 20/7

51 DC & CV Lab. CSIE NTU Example 4.4 (cont.)

52 DC & CV Lab. CSIE NTU Example 4.4 (cont.)

53 DC & CV Lab. CSIE NTU Example 4.5

54 DC & CV Lab. CSIE NTU Example 4.5 (cont.)

55 DC & CV Lab. CSIE NTU Example 4.5 (cont.)

56 f1 and f4 forms the lowest Bayes gain Find some p that eliminate P(c1) p = 0.3103 DC & CV Lab. CSIE NTU

57 DC & CV Lab. CSIE NTU Example 4.5 (cont.)

58 DC & CV Lab. CSIE NTU Decision Rule Error The misidentification error α k The false-identification error β k

59 DC & CV Lab. CSIE NTU An Instance

60 DC & CV Lab. CSIE NTU Reserving Judgment The decision rule may withhold judgment for some measurements Then, the decision rule is characterized by the fraction of time it withhold judgment and the error rate for those measurement it does assign. It is an important technique to control error rate.

61 Reserving Judgment Let be the maximum Type I error we can tolerate with category k Let be the maximum Type II error we can tolerate with category k Measurement that will not be rejected (acceptance region) DC & CV Lab. CSIE NTU

62 DC & CV Lab. CSIE NTU Nearest Neighbor Rule Assign pattern x to the closest vector in the training set The definition of “closest”: where is a metric or measurement space Chief difficulty: brute-force nearest neighbor algorithm computational complexity proportional to number of patterns in training set

63 DC & CV Lab. CSIE NTU Binary Decision Tree Classifier Assign by hierarchical decision procedure

64 DC & CV Lab. CSIE NTU Major Problems Choosing tree structure Choosing features used at each non-terminal node Choosing decision rule at each non-terminal node

65 DC & CV Lab. CSIE NTU Decision Rules at the Non-terminal Node Thresholding the measurement component Fisher’s linear decision rule Bayes quadratic decision rule Bayes linear decision rule Linear decision rule from the first principal component

66 DC & CV Lab. CSIE NTU Error Estimation An important way to characterize the performance of a decision rule Training data set: must be independent of testing data set Hold-out method: a common technique construct the decision rule with half the data set, and test with the other half

67 DC & CV Lab. CSIE NTU Neural Network A set of units each of which takes a linear combination of values from either an input vector or the output of other units

68 DC & CV Lab. CSIE NTU Neural Network (cont.) Has a training algorithm Responses observed Reinforcement algorithms Back propagation to change weights

69 DC & CV Lab. CSIE NTU Summary Bayesian approach Maximin decision rule Misidentification and false-alarm error rates Nearest neighbor rule Construction of decision trees Estimation of decision rules error Neural network

Download ppt "Digital Camera and Computer Vision Laboratory Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R.O.C."

Similar presentations

Principles of Density Estimation

Principles of Density Estimation
Naïve Bayes. Bayesian Reasoning Bayesian reasoning provides a probabilistic approach to inference. It is based on the assumption that the quantities of.

Naïve Bayes. Bayesian Reasoning Bayesian reasoning provides a probabilistic approach to inference. It is based on the assumption that the quantities of.
Component Analysis (Review)

Component Analysis (Review)
ECE 8443 – Pattern Recognition LECTURE 05: MAXIMUM LIKELIHOOD ESTIMATION Objectives: Discrete Features Maximum Likelihood Resources: D.H.S: Chapter 3 (Part.

ECE 8443 – Pattern Recognition LECTURE 05: MAXIMUM LIKELIHOOD ESTIMATION Objectives: Discrete Features Maximum Likelihood Resources: D.H.S: Chapter 3 (Part.
Data Mining Classification: Alternative Techniques

Data Mining Classification: Alternative Techniques
Data Mining Classification: Alternative Techniques

Data Mining Classification: Alternative Techniques
2 – In previous chapters: – We could design an optimal classifier if we knew the prior probabilities P(wi) and the class- conditional probabilities P(x|wi)

2 – In previous chapters: – We could design an optimal classifier if we knew the prior probabilities P(wi) and the class- conditional probabilities P(x|wi)
Pattern Classification. Chapter 2 (Part 1): Bayesian Decision Theory (Sections 2.1-2.2) Introduction Bayesian Decision Theory–Continuous Features.

Pattern Classification. Chapter 2 (Part 1): Bayesian Decision Theory (Sections ) Introduction Bayesian Decision Theory–Continuous Features.
Pattern recognition Professor Aly A. Farag

Pattern recognition Professor Aly A. Farag
Visual Recognition Tutorial

Visual Recognition Tutorial
Bayesian Decision Theory Chapter 2 (Duda et al.) – Sections

Bayesian Decision Theory Chapter 2 (Duda et al.) – Sections
Pattern Recognition: Readings: Ch 4: , , 4.13

Pattern Recognition: Readings: Ch 4: , , 4.13
0 Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.

0 Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John.
1 MACHINE LEARNING TECHNIQUES IN IMAGE PROCESSING By Kaan Tariman M.S. in Computer Science CSCI 8810 Course Project.

1 MACHINE LEARNING TECHNIQUES IN IMAGE PROCESSING By Kaan Tariman M.S. in Computer Science CSCI 8810 Course Project.
Chapter 4 (part 2): Non-Parametric Classification

Chapter 4 (part 2): Non-Parametric Classification
Pattern Recognition. Introduction. Definitions.. Recognition process. Recognition process relates input signal to the stored concepts about the object.

Pattern Recognition. Introduction. Definitions.. Recognition process. Recognition process relates input signal to the stored concepts about the object.
© Prentice Hall1 DATA MINING Introductory and Advanced Topics Part II Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist.

© Prentice Hall1 DATA MINING Introductory and Advanced Topics Part II Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist.
Chapter 3 (part 1): Maximum-Likelihood & Bayesian Parameter Estimation  Introduction  Maximum-Likelihood Estimation  Example of a Specific Case  The.

Chapter 3 (part 1): Maximum-Likelihood & Bayesian Parameter Estimation  Introduction  Maximum-Likelihood Estimation  Example of a Specific Case  The.
Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)

Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)
METU Informatics Institute Min 720 Pattern Classification with Bio-Medical Applications PART 2: Statistical Pattern Classification: Optimal Classification.

METU Informatics Institute Min 720 Pattern Classification with Bio-Medical Applications PART 2: Statistical Pattern Classification: Optimal Classification.

Similar presentations

Ads by Google