1. Statistical Decision Theory, Bayes Classifier
Lecture Notes for CMPUT 466/551
Nilanjan Ray

2. Supervised Learning Problem
Input (random) variable: X
Output (random) variable: Y
A toy example:
unknown
A set of realizations of X and Y are available in (input, output) pairs:
This set T is known as the training set
Supervised Learning Problem: given the training set, we want to estimate
the value of the output variable Y for a new input variable value, X=x0 ?
Essentially here we try to learn the unknown function f from T.

3. Basics of Statistical Decision Theory
We want to attack the supervised learning problem from the viewpoint of
probability and statistics. Thus, let’s consider X and Y as random variables
with a joint probability distribution Pr(X,Y).
Assume a loss function L(Y,f(x)), such as a squared loss (L2):
Choose f so that the expected prediction error is minimized:
Known as regression
function
Also known as
(non-linear) filtering
in signal processing
The minimizer is the conditional expectation :
So, if we knew Pr(Y|X), we would readily estimate the output
See [HTF] for the derivation of this minimization;
Also see [Bishop] for a different derivation– know about a heavier machinery, called
calculus of variations

4. Loss Functions and Regression Functions
L1 loss:
RF is conditional median:
0-1 loss:
RF is conditional mode:
Exponential loss function
Used in boosting…
Huber loss function
Robust to outliers…
Your loss function
Own student award in …
Observation: the estimator/regression function is always
defined in terms of the conditional probability Pr(Y|X)
Pr(Y|X) is Typically unknown to us, so what can we do??

5. Create Your Own World Example Strategies:
Try to estimate regression function directly from training data
Assume some models, or structure on the solution,….so that the EPE minimization becomes tractable
Your own award winning paper
Etc…
Structure free
Highly structured

6. Nearest-Neighbor Method
Use those observations to predict that are neighbors of the (new) input:
Nk(x) is the neighborhood of x defined by k-closest points xi in the training sample
Estimates the regression function directly from the training set:
Nearest neighbor method is also (traditionally) known as a nonparametric method

7. Nearest-Neighbor Fit Major limitations:
Very in inefficient in high dimension
Could be unstable (wiggly)
If training data is scarce, may not be the right choice
For data description see
[HTF], Section 2.3.3

8. How to Choose k: Bias-Variance Trade-off
Let’s try to answer the obvious question: how can we choose k?
One way to get some insight about k is the bias-variance trade-off
Let’s say input-output model is
Test error:
Irreducible error
(the regressor
has no control)
Will decrease
as k increases
Will most likely
increase as k increases
So we can find a trade-off between bias and variance to choose k

9. Bias Variance Trade-off
Bias variance trade-off: Graphical representation

10. Assumption on Regression Function: Linear Models
Structural assumption: Output Y is linear in the inputs X=(X1, X2, X3,…, Xp)
Predict the output by:
Vector notation, 1 included in X
We estimate β from the training set (by least square):
Can be shown that EPE minimization
leads to this estimate with the linear model
(see the comments in [HTF] section 2.4)
Where,
So, for a new input
The regression output is

11. Example of Linear Model Fit
Classification rule: X2
Very stable (not wiggly), and
computationally efficient.
However, if the linear model
assumption fails, the regression
fails miserably X1
X = (X1, X2)
Y = (Red, Green), coded as Red=1, Green=0

12. More Regressors
Many regression techniques are based on modifications of linear models and nearest-neighbor techniques
Kernel methods
Local regression
Basis expansions
Projection pursuits and neural networks ….
Chapter 2 of [HTF] gives their quick overview

13. Bayes Classifier
The key probability distribution Pr(Y|X) in the Bayesian paradigm
called posterior distribution
EPE is also called Risk in the Bayesian terminology; however Risk is more
general than EPE
We can show that EPE is the average error of classification under 0-1 penalty
in the Risk
So the Bayes’ classifier is minimum error classifier
Corresponding error is called Bayes’ risk / Bayes rate etc.
Materials mostly based on Ch 2 of [DHS] book

14. Bayes’ Classifier…
We can associate probabilities to the categories (output classes) of the classifier
This probability is called prior probability or simply the prior R1 R2 R3
We assign a cost (Cij) to each possible
decision outcome: Hi is true, we choose Hj
M=3
2D input space, X
We know the conditional probability densities p(X|Hi)
Also called likelihood of the hypothesis Hi, or simply the likelihood
Total risk

15. Bayes’ Classifier…
Choose the Rj’s in such as way that this risk is minimized
Since Rj partitions the entire region, any x will belong to exactly one such Rj
So we can minimize the risk via the following rule to construct the partitions:
You should realize that the reasoning is similar to that in [HTF]
while minimizing EPE in Section 2.4.

16. Minimum Error Criterion
We are minimizing the Risk with 0-1 criterion
This is also known as minimum error criterion
In this case the total risk is the probability of error
Note that
Posterior probability
This is same result if you minimize the EPE with 0-1 loss function
(see [HTF] Section 2.4)

17. Two Class Case: Likelihood Ratio Test
We can write Risk as:
We can immediately see the rule of classification from above:
Assign x to H1 if
else assign x to H2
Likelihood ratio
Special case: 0-1 criterion

18. An Example: Bayes Optimal Classifier
We exactly know the posterior distribution of the
two distribution, from which these boundaries are created

19. Minimax Classification
Bayes rule:
In many real life applications, prior probabilities may be unknown
So we cannot have a Bayes’ optimal classification.
However, one may wish to minimize the worst case overall risk
We can write the Risk as a function of P1 and R1:
Observation 1: If we fix R1 then Risk is a linear function of P1
Observation 2: The function is concave in P1

20. Minimax… Let’s arbitrarily fix P1 = a, and compute By observation 1,
is the straight line AB
Q: why should it be a tangent to g(P1)?
Claim:
is not a good classification when P1 is not known. Why?
So, what could be a better classification here?
The classification corresponding to MN. Why?
This is the minimax solution
Why the name? We can reach the solution by
An aside Q: when can we reverse the order and get the same result?
Another way to solve R1 in minimax is from:
If you get multiple solutions, choose one that gives you the minimum Risk

21. Neyman-Pearson Criterion
Consider a two class problem
Following four probabilities can be computed:
Probability of detection (hit)
Probability of false alarm
Probability of miss
Probability of correct rejection R1 R2
We do not know the prior probabilities, so Bayes’s optimum classification
is not possible
However we do know that Probability of False alarm must be below 
Based on this constraint (Neyman-Pearson criterion) we can design a classifier
Observation: maximizing probability of detection and minimizing probability of
false alarm are conflicting goals (in general)

22. Receiver Operating Characteristics
ROC is a plot: probability of false alarm vs. probability of detection
Classifier 1
Probability of detection
Classifier 2
Probability of false alarm
Area under ROC curve is a measure of performance
Used also to find a operating point for the classifier