1. Ch.9 Pattern Recognition
Slides are from KAIST J. Kim

2. What is Pattern Recognition?
A pattern is an object, process or event that can be given a name
Pattern Recognition
The assignment of a physical object or event to one of several prespecified categories -- Duda & Hart
A subfield of Artificial Intelligence
The bulk of human intelligence is based on pattern recognition: the quintessential example of self-organization 2

3. Resources Professional Association Text Books Journals
International Association for Pattern Recognition (IAPR)
Text Books
Pattern Classification by Richard O. Duda, Pater E. Hart, and David G, Storks
Journals
IEEE Transactions on Pattern Analysis and Machine Intelligence
Pattern Recognition
Pattern Recognition Letters
Artificial Intelligence and Pattern Recognition …
Conference and Workshops
International Conference for Pattern Recognition
IEEE Computer Vision and Pattern Recognition
Int’l Conference on Document Analysis and Recognition
Int’l Workshop in Frontiers of Handwriting Recognition 3

4. Examples of Patterns 4

5. Related fields and Application areas of PR
Adaptive signal processing
Machine learning
Artificial neural networks
Robotics and vision
Cognitive science
Mathematical statistics
Nonlinear optimization
Exploratory data analysis
Fuzzy and genetic algorithm
Detection and estimation theory
Formal languages
Structural modeling
Biological cybernetics
Computational neuroscience …
Image processing /segmentation
Computer vision
Speech recognition
Automated target recognition
Optical character recognition
Seismic analysis
Man and machine dialogue
Fingerprint identification
Industrial inspection
Medical diagnosis
ECG signal analysis
Data mining
Gene sequence analysis
Protein structure anlaysis
Remote sensing
Aerial reconnaissance 5

6. 패턴인식의 응용 Computer aided diagnosis 영상인식 음성인식 Genome Sequence Analysis
Medical imaging, EEF, ECG, X- ray mammography
영상인식
공장 자동화, Robot Navigation
얼굴식별, Gesture Recognition
Automatic Target Recognition
음성인식
Speaker identification
Speech recognition
Genome Sequence Analysis 6

7. 생체 인식 불변의 생체 특징을 이용한 사람 식별 정적 패턴 동적 패턴 출입통제 전자상거래 인증 지문, 홍채, 얼굴, 장문
손등의 정맥
동적 패턴
Signature, 성문
Typing pattern
출입통제
전자상거래 인증 7

8. Gesture Recognition Object manipulation in Virtual Reality
Tele-operations
Control remote by gesture input
TV control by hand motion
Text editing on Pen Computers
Sign language Interpretation 8

9. 데이터로부터 패턴의 추출 Data Mining
데이타
의사결정 정보
인구통계
Point of Sale
ATM
금융통계
신용정보 문헌
첩보자료
진료기록
신체검사기록
A상품 구매자의 80%가 B상품도 구매한다 (CRM)
미국시장의 자동차 구매력이 6개월간 감소
A상품의 매출 증가가 B상품의 2배
탈수 증상을 보이면 위험
광고전략은 ?
상품의 진열
최적의 예산 할당은 ?
시장점유의 확대방안은 ?
고객의 이탈 방지책은 ?
처방은 ?
국내 사례: 신용카드 사용 패턴의 학습에 의한 분실 카드 사용 방지 9

10. e-Book, Tablet PC, PDA, M-phone10

11. Input Devices PDA with Camera or Scanner Pen Scanner & Computer
Motion_Sensing Pen 11

12. Freehand Equation Input
수식 입력 시스템 : 문서 편집기와 연계
전자 펜으로 수식 입력
수식 인식 12

13. 古文書 : 承政院 日記 13

14. Verification & Correction Interface14

15. Mail Sorter 15

16. Autonomous Land Vehicle (DARPA’s GrandChallenge contest)16

17. Protein Structure Analysis17

18. Types of PR problems Classification Clustering Regression Description
Assigning an object to a class
Output: a label of class
Ex: classifying a product as ‘good’ or ‘bad’ in quality control
Clustering
Organizing objects into meaningful groups
Output: (hierarchical) grouping of objects
Ex: taxonomy of species
Regression
Predict value based on observation
Ex: predict stock price, future prediction
Description
Representing an object in terms of a series of primitives
Output: a structural or linguistic description
Ex: labeling ECG signals, video indexing, protein structure indexing 18

19. Pattern Class
A collection of “similar” (not necessarily identical) objects
Inter-class variability
Intra-class variability
Pattern Class Model
descriptions of each class/population (e.g., a probability density like Gaussian) 19

20. Classification vs Clustering
Classification (known categories)
Clustering (creation of new categories)
Category “A”
Category “B”
Clustering
(Unsupervised Classification)
Classification (Recognition)
(Supervised Classification) 20

21. Pattern Recognition : Key Objectives
1) Process the sensed data to eliminate noise
Data vs Noise
2) Hypothesize the models that describe each class population
Then we may recover the process that generated the patterns.
3) Choose the best-fitting model for given sensed data to
assign the class label associated with the model. 21

22. 일반적인 Classification 과정
Sensor
signal
Sensor
Feature
Extractor
Feature
Classifier
Class
Membership 22

23. Example : Salmon or Sea Bass
Sort incoming fish on a belt according to two classes:
Salmon or
Sea Bass
Steps:
Preprocessing (segmentation)
Feature extraction (measure features or properties)
Classification (make final decision) 23

24. Sea bass vs Salmon Discrimination
Possible features to be used:
Length
Lightness
Width
Number and shape of fins
Position of the mouth
Etc … 24

25. Salmon vs. Sea Bass (by length)25

26. Salmon vs. Sea Bass (by lightness)
Best Decision Strategy
with lightness 26

27. Cost of Misclassification
There are two possible classification errors.
(1) deciding a sea bass into a salmon.
(2) deciding a salmon into a sea bass.
Which error is more important ?
Generalized as Loss function
Then, look for the decision of minimun Risk
Risk = Expected Loss
Loss Function
Salmon
Sea Bass
-10
Sea bass
-20
decision
truth 27

28. Classification with more features (by length and lightness)
It is possibly better.
Really ?? 28

29. How Many Features and Which?
Choice of features determines success or failure of classification task
For a given feature, we may compute the best decision strategy from the (training) data
Is called training, parameter adaptation, learning
Machine Learning Issues
Issues with feature extraction:
Correlated features do not improve performance.
It might be difficult to extract certain features.
It might be computationally expensive to extract many features.
“Curse” of dimensionality … 29

30. 30

31. 31

32. Components of PR system
Sensors and preprocessing
Feature extraction
Class
assignment
Pattern
Classifier
Learning algorithm
Teacher
Sensors and preprocessing.
A feature extraction aims to create discriminative features good
for classification.
A classifier.
A teacher provides information about hidden state
A learning algorithm sets PR from training examples. 32

33. PR Approaches Template matching Statistical PR: Structural PR:
The pattern to be recognized is matched against a stored template
Works only simple problems
Statistical PR:
based on underlying statistical model of patterns(features) and pattern classes.
Class-conditional probability Pr(X|Ci)
Structural PR:
pattern classes represented by means of formal structures as grammars, automata, strings, etc.
Based on measures of structural similarity
Not only for classification but also description
Often called Syntactic pattern recognition
Neural networks
classifier is represented as a network of cells modeling neurons of the human brain (connectionist approach).
Knowledge is stored in the connectivity and strength of synaptic weights
Trainable, requires minimum a priori knowledge, works for complex decision boundaries
Statistical structure Analysis
Combining Structure and statistical analysis
Bayesian Network, MRF 등의 Probabilistic framework을 활용 33
Modified From Vojtěch Franc

34. Template Matching
Template 34
Input scene

35. Deformable Template Matching: Snake
Example : Corpus Callosum Segmentation
Prototype registration to the low-level segmented image
Shape training set
Prototype and variation learning
Prototype warping 35

36. 36

37. Classifier
The task of classifier is to partition feature space into class-labeled decision regions
Borders between decision regions  decision boundaries
Determining decision region of a feature vector X 37

38. Representation of classifier
A classifier is typically represented as a set of discriminant functions …
The classifier assigns a feature vector x to the i-the class if
Class identifier ….
Feature vector 38
Discriminant function

39. Classification of Classifiers by Form of Discriminant Function
A posteriori Probability
P( yi | X)
Bayesian
Linear Function
Linear Discrinant Analysis, Support Vector Machine
Non-Linear Function
Non-Linear Discrinant Analysis
Artificial Neuron
Artificial Neural Network 39

40. Bayesian Decision Making
Statistical approach
the optimal classifier with Minimum error
Assume that complete statistical model is known.
Decision given the posterior probabilities
X is an observation :
if P(1 | x) > P(2 | x) decide state of nature = 1
if P(1 | x) < P(2 | x) decide state of nature = 2 40

41. Searching Decision Boundary41

42. Bayesian Rule : P(x|1) P(1|x)42

43. Limitations of Bayesian approach
Statistical model p(x|y) is mostly not known
learning to estimate p(x|y) from training examples {(x1,y1),…,(x,y)}
Usually p(x|y) is assumed to be a parametric form
Ex: multivariate normal distribution
Non-parametric estimation of p(x|y) requires a large set of training samples
Non-Bayesian methods offers equally good (??) 43

44. Discriminative approaches
Assume that G(x) is a polynomial function
Quadratic function
Linear function – Linear Discriminant Analysis (LDA)
Classifier design is to determination of separating hyperplane. 44

45. LDA Example height Task: jockey-hoopster recognition.
The set of hidden state is
The feature space is
weight
Training examples …
Linear classifier: 45

46. Artificial Neural Network
For a given structure, find best weight sets which minimizes sum of square error, J(w), from training examples {(x1,y1),…,(x,y)} 46

47. PR design cycle Data collection Feature choice Model choice and design
Probably the most time-intensive component of project
How many examples are enough ?
Feature choice
Critical to the success of the PR project
Require basic prior knowledge, engineering sense
Model choice and design
Statistical, neural and structural
Parameter settings
Training
Given a feature set and ‘blank’ model, adapt the model to explain the training data
Supervised, unsupervised, reinforcement learning
Evaluation
How well does the trained model do ?
Overfitting vs. generalization 47

48. Learning for PR system Learning algorithm
Sensors and preprocessing
Feature extraction
Class
assignment
Pattern
Classifier
Learning algorithm
Teacher
Which Feature is good for classifying given classes ?
Feature analysis
Can we get required probabilities or boundaries ?
Learning from training Data 48

49. Learning
Change of contents and organization of system’s knowledge enabling to improve to its performance on task - Simon
Data Mining
Learning rules from large set of data
Availability of large database allows application of machine learning to real problems 49

50. Learning Algorithm Categorization (Depending on Available Feedback)
supervised learning
examples of correct input/output pair is available
unsupervised learning
No hint at all about the correct outputs.
Clustering or consistent interpretation.
reinforcement learning
Receives no examples in advance, but rewards or punishments at the end
Transduction : Semi-supervised learning
Training with labeled training examples and unlabeled examples 50

51. Issues on Learning Algorithm
Prior Knowledge
Prior knowledge can help in learning.
Assumptions on parametric forms and range of values
Incremental learning
Update old knowledge whenever new example arrives
Batch learning
Apply learning algorithm to the entire set of examples
Analytic approach : find the optimal parameter values by analysis
Iterative adaptation : improve parameter values from initial guess 51

52. Learning Algorithms General Ideas
Tweak parameters so as to optimize performance criterion
In the course of learning, the parameter vector traces a path that (hopefully) ends at the best parameter vector 52

53. Inductive Learning
For given training examples (correct input-output pairs),
Recover unknown underlying function from which the training data generated
generalization ability for unseen data is required
Learning can be seen as learning the representation of a function
Forms of the Function
Logical sentences / Polynomials / Set of weights (Neural Networks), …
Given form of polynomial function (structure of neural network), adjust parameters (synaptic weights) to minimize error 53

54. Model Complexity B A Decision Boundary of Salmon and Sea bass
Which is better ? A or B A B 54

55. Model Complexity
We can get perfect classification performance on the training data by choosing complex models.
Complex models are tuned to the particular training samples, rather than on the characteristics of the true model.
Issue of generalization 55

56. Generalization
The main goal of pattern classification system is to generalize or suggest the class or action of objects yet unseen.
Some complex decision boundaries are not good at generalization.
Some simple boundaries are not good either.
One must look for a tradeoff between performance and simplicity
This is at the core of statistical pattern recognition 56

57. Generalization Strategy
How can we improve generalization performance ?
More training examples (i.e., better pdf estimates).
Simpler models (i.e., simpler classification boundaries) usually yield better performance. 57
Simplify the decision boundary!

58. Overfitting and underfitting
From Vojtěch Franc
Overfitting and underfitting
underfitting
good fit
overfitting
Problem of generalization: a small emprical risk Remp does not imply small true expected risk R. 58

59. Curse of Dimensionality
Adding too many features can, paradoxically, lead to a worsening of performance.
If each input feature is divided into M divisions, then the total number of cells is Md (d: # of features) which grows exponentially with d.
With fixed training data, adding more features worsening the quality of generalization
Because too many cell causes small number of training samples per cell
Small training data  poor generalization
Training data is always not sufficient !! 59

60. Curse of Dimensionality
Function의 수를 늘려면 error 감소
훈련데이더에 대한Classifier 성능 향상
제한된 양의 training Data로 훈련 시에 Feature 수를 늘 리면 일반화 능력 감소
적절한 일반화 능력 향상을 위하여 요구되는 훈련데이 터의 양은 feature dimension에 따라 급격히 증가
For a finite set of training data, Finding Optimal set of Features
is a difficult problem 60

61. Optimal Number of Cells (example)61

62. Implication of Curse of Dimensionality to PR system design
With finite training samples, be cautious of adding features
Features of high Discrimination power first
Feature analysis is mandatory 62

63. Empirical risk minimization principle
From Vojtěch Franc
Empirical risk minimization principle
The true expected risk R(q) is approximated by empirical risk
with respect to a given labeled training set {(x1,y1),…,(x,y)}.
The learning based on the empirical minimization principle is defined as
Examples of algorithms: Perceptron, Back-propagation, etc. 63

64. Cross-Validation
Validate learned model on different set to assess the generalization performance
guarding against overfitting
Partition Training set into two subsets
Estimation subset for learning parameters
validation subset
Leave-one-out validation method
N-1 for training, 1 for validation, takes turn
Overcome Small training set 64

65. Unsupervised learning
Input: training examples {x1,…,x} without information about the hidden state.
Clustering: goal is to find clusters of data sharing similar properties.

66. Example of unsupervised learning algorithm
Goal is to minimize
k-Means clustering: …
Classifier
Learning algorithm … 66

67. Other Issues in Pattern Recognition

68. Difficulty of Class Modeling68

69. 인식에는 Context Processing이 필수69

70. 문자인식에서의 Context Processing70

71. Local decision is not enough
Global Consistency
Local decision is not enough 71

72. Combining Multiple Classifiers
Approaches for improving the performance of the group of experts
Best single classifier vs Combining multiple classifiers
Two heads (experts, classifiers) are better than one
Classifier output is either
Best (single) class
Ranking
Score as each class
Method for generating multiple classifiers
Co-related classifiers would not be a help
Method for combining multiple classifiers
Majority rules, borda count, decorelated combination, etc.
A study on combining multiple classifiers has been investigated in pattern recognition during the last decade.
For a classifier, it is well known that it makes a decision in the form of one of the three classification results; measurement score, ranking, and single choice.
We consider only classifiers producing single choice level decision.
For a combination type of classifiers according to the arrangements of them, only parallel combination type is chosen in this paper. 72

73. 패턴 인식 성능의 평가 실제 (정) 인식률 = (p+q)/(p+q+r+s) 오인식률 = (r+s)/(p+q+r+s)
Miss detection = r/(p+r)
False alarm = s/(p+s)
Recall = p/p+r
Precision = p/p+s
기각율 (refuse to make decision)
처리율 A
not A
인식결과 p a s q
not a r
A case: 20% 기각했는데 결과에는 0.5% error
B case : 10% 기각했는데 결과에는 1.0% error
Which is better ? 73

74. 패턴인식의 성능 향상
100% 를 향하여 성능
시간, 노력 74