1. Tutorial #3 by Ma’ayan Fishelson
Parameter Estimation
Using Likelihood Functions.
Tutorial #3
by Ma’ayan Fishelson
This presentation has been cut and slightly edited from Nir Friedman’s full course of 12 lectures which is available at Changes made by Dan Geiger, Ydo Wexler and Ma’ayan Fishelson.

2. Example: Binomial Experiment
Head
Tail
When tossed, it can land in one of two positions: Head or Tail.
We denote by  the (unknown) probability P(H).
Estimation task:
Given a sequence of toss samples x[1], x[2],…,x[M] we want to estimate the probabilities P(H) =  and P(T) = 1 - .

3. Statistical Parameter Fitting
Consider instances x[1], x[2], …, x[M] such that:
The set of values x can take is known.
Each one is sampled from the same distribution.
Each one is sampled independently of the rest.
The task is to find the vector of parameters  that have generated the given data. This parameter vector  can be used to predict future data.
i.i.d.
samples
i.I.d - idependent, identically distributed

4. The Likelihood Function
How good is a particular  ? It depends on how likely it is to generate the
observed data:
The likelihood for the sequence H,T, T, H, H is:
0.2
0.4
0.6
0.8 1 
L()

5. Sufficient Statistics
To compute the likelihood in the thumbtack example we only require NH and NT (the number of heads and the number of tails).
NH and NT are sufficient statistics for the binomial distribution.
For example: H,T,T,H,H is just like T,H,H,H,T

6. Sufficient Statistics
A sufficient statistic is a function of the
data that summarizes the relevant information
for the likelihood.
Formally, s(D) is a sufficient statistics if for
any two datasets D and D’:
s(D) = s(D’ )  LD() = LD’()
Datasets
Statistics

7. Maximum Likelihood Estimation
MLE Principle: Choose parameters that maximize the likelihood function
One of the most commonly used estimators in statistics.
Intuitively appealing.
One usually maximizes the log-likelihood function defined as lD() = logeLD().

8. Example: MLE in Binomial Data
Applying the MLE principle we get
(Which coincides with what one would expect)
0.2
0.4
0.6
0.8 1
L()
Example:
(NH,NT ) = (3,2)
MLE estimate is 3/5 = 0.6

9. From Binomial to Multinomial
For example, suppose that X can have the values 1,2,…,K (for example, a die has 6 faces).
We want to learn the parameters 1, 2. …, K.
Sufficient statistics:
N1, N2, …, NK - the number of times each
outcome is observed.
Likelihood function:
MLE:
A case of a die where 1….6 are the possible values

10. Example: Multinomial Let x1,x2,…,xn be a protein sequence.
We want to learn the parameters q1, q2,…,q20
corresponding to the frequencies of the 20 amino-acids.
Let N1, N2,…,N20 be the number of times each amino-acid is observed in the sequence.
Likelihood function:
MLE:
A case of a die where 1….6 are the possible values

11. Is MLE all we need? Suppose that after 10 observations,
ML estimate is P(H) = 0.7 for the thumbtack.
Would you bet on heads for the next toss?
Suppose now that after 10 observations,
ML estimate is P(H) = 0.7 for a coin.
Would you place the same bet?
Solution: The Bayesian approach that incorporates your subjective prior knowledge. E.g., you may know a priori that some amino acids have high frequencies and some have low frequencies. How would one use this information ?

12. Bayes’ rule
where,
Bayes’ rule:
It holds because:

13. Example: Dishonest Casino
A casino uses 2 kind of dice:
99% are fair.
1% is loaded: 6 comes up 50% of the times
We pick a die at random and roll it 3 times.
We get 3 consecutive sixes.
What is the probability the die is loaded?

14. Dishonest Casino (2) The solution is based on using Bayes rule and the
fact that while P(loaded | 3 sixes) is not known, the
other three terms in Bayes rule are known, namely:
P(3 sixes | loaded)=(0.5)3
P(loaded)=0.01
P(3 sixes) =
P(3 sixes|loaded) P(loaded)+P(3 sixes|fair) P(fair)

15. Dishonest Casino (3)

16. Biological Example: Proteins
Extra-cellular proteins have a slightly different amino acid composition than intra-cellular proteins.
From a large enough protein database (SWISS-PROT), we can get the following:
p(ext) - the probability that any new sequence is extra-cellular
p(ai|int) - the frequency of amino acid ai for intra-cellular proteins
p(ai|ext) - the frequency of amino acid ai for extra-cellular proteins
p(int) - the probability that any new sequence is intra-cellular

17. Biological Example: Proteins (2)
Q: What is the probability that a given new
protein sequence x=x1x2….xn is extra-cellular?
A: Assuming that every sequence is either
extra-cellular or intra-cellular (but not both),
we can write:
Thus,

18. Biological Example: Proteins (3)
Using conditional probability we get ,
By Bayes’ theorem
The probabilities p(int), p(ext) are called the prior
probabilities.
The probability P(ext|x) is called the posterior probability.