Introduction to Estimation Theory: A Tutorial Volkan Cevher

Georgia Institute of Technology Center for Signal and Image Processing Outline Introduction Terminology and Preliminaries Bayesian (Random) Parameter Estimation Nonrandom Parameter Estimation Questions Georgia Institute of Technology Center for Signal and Image Processing

Georgia Institute of Technology Center for Signal and Image Processing Introduction Classical detection problem: Design of optimum procedures for deciding between possible statistical situations given a random observation: The model has the following components: Parameter Space (for parametric detection problems) Probabilistic Mapping from Parameter Space to Observation Space Observation Space Detection Rule There is also non parametric detection problems using rough information about the data such as signs ranks rather than the exact values of the data. Georgia Institute of Technology Center for Signal and Image Processing

Georgia Institute of Technology Center for Signal and Image Processing Introduction Parameter Space: Completely characterizes the output given the mapping. Each hypothesis corresponds to a point in the parameter space. This mapping is one-to-one. Probabilistic Mapping from Parameter Space to Observation Space: The probability law that governs the effect of a parameter on the observation. Example 1: Probabilistic mapping Parameter Space Georgia Institute of Technology Center for Signal and Image Processing

Georgia Institute of Technology Center for Signal and Image Processing Introduction Observation Space: Finite dimensional, i.e. Y n, where n is finite. Detection Rule Mapping of the observation space into its parameters in the parameter space is called a detection rule. Georgia Institute of Technology Center for Signal and Image Processing

Georgia Institute of Technology Center for Signal and Image Processing Introduction Classical estimation problem: Interested in not making a choice among several discrete situations, but rather making a choice among a continuum of possible states. Think of a family of distributions on the observation space, indexed by a set of parameters. Given the observation, determine as accurately as possible the actual value of the parameter. In this example, given the observations, parameter  is being estimated. Its value is not chosen among a set of discrete values, but rather is estimated as accurately as possible. Example 2: Georgia Institute of Technology Center for Signal and Image Processing

Georgia Institute of Technology Center for Signal and Image Processing Introduction Estimation problem also has the same components as the detection problem. Parameter Space Probabilistic Mapping from Parameter Space to Observation Space Observation Space Estimation Rule Detection problem can be thought of as a special case of the estimation problem. There are a variety of estimation procedures differing basically in the amount of prior information about the parameter and in the performance criteria applied. Estimation theory is less structured than detection theory. “Detection is science, estimation is art.” Array Signal Processing by Johnson, Dudgeon. Georgia Institute of Technology Center for Signal and Image Processing

Georgia Institute of Technology Center for Signal and Image Processing Introduction Based on the a priori information about the parameter, there are two basic approaches to parameter estimation: Bayesian Parameter Estimation Nonrandom Parameter Estimation Bayesian Parameter Estimation: Parameter is assumed to be a random quantity related statistically to the observation. Nonrandom Parameter Estimation: Parameter is a constant without any probabilistic structure. Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Estimation theory relies on jargon to characterize the properties of estimators. In this presentation, the following definitions are used: The set of n observations are represented by the n-dimensional vector y (observation space). The values of the parameters are denoted by the vector  (parameter space). The estimate of this parameter vector is denoted by : . Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Definitions (continued): The estimation error (y) ( in short) is defined by the difference between the estimate and the actual parameter: The function C[a,]: + is the cost of estimating a true value of  as a. Given such a cost function C, the Bayes risk (average risk) of the estimator is defined by the following: Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Example 3: Suppose we would like to minimize the Bayes risk defined by for a given cost function C. By inspection, one can see that the Bayes estimate of  can be found (if it exists) by minimizing, for each y, the posterior cost given Y=y: Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Definitions (continued): An estimate is said to be unbiased if the expected value of the estimate equals the true value of the parameter . Otherwise the estimate is said to be biased. The bias b() is usually considered to be additive, so that: An estimate is said to be asymptotically unbiased if the bias tends to zero as the number of observations tend to infinity. An estimate is said to be consistent if the mean-squared estimation error tends to zero as the number of observations becomes large. Lack of bias should not be considered a desirable property. This property does not imply that the estimate has less error than the biased one; there exists biased estimates whose mean-squared errors are less than unbiased ones. In such cases, the estimate is usually asymptotically unbiased. Hence, lack of bias may be good. Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Definitions (continued): An efficient estimate has a mean-squared error that equals a particular lower bound: the Cramer-Rao bound. If an efficient estimate exists, it is optimum in the mean-squared sense: No other estimate has a smaller mean-squared error. Following shorthand notations will also be used for brevity: For many problems found in practice, no efficient estimate may exist. However, the Cramer-Rao bound is still used to compare the performance of the estimators. More on Cramer-Rao bound in the upcoming sections. Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Following definitions and theorems will be useful later in the presentation: Definition: Sufficiency Suppose that  is an arbitrary set. A function T:  is said to be a sufficient statistic for the parameter set  if the distribution of y conditioned on T(y) does not depend on  for . If knowing T(y) removes any further dependence on  of the distribution of y, one can conclude that T(y) contains all the information in y that is useful for estimating . Hence, it is sufficient. Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Definition: Minimal Sufficiency A function T on  is said to be minimal sufficient for the parameter set  if it is a function of every other sufficient statistic for . A minimal sufficient statistic represents the furthest reduction in the observation without destroying information about . Minimal sufficient statistic does not necessarily exist for every problem. Even if it exists, it is usually very difficult to identify it. Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries The Factorization Theorem: Suppose that the parameter set  has a corresponding families of densities p. A statistic T is sufficient for  iff there are functions g and h such that for all y and . Refer to the supplement for a proof. Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Example 4: (Poor) Consider the hypothesis-testing problem ={0,1} with densities p0 and p1. Noting that the factorization is possible with Thus the likelihood ratio L is a sufficient statistic for the binary hypothesis-testing problem. Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries The Rao-Blackwell Theorem: Suppose that ĝ(y) is an unbiased estimate of g() and that T is sufficient for . Define Then is also an unbiased estimate of g(). Furthermore, with equality iff Refer to the supplement for a proof. The Rao-Blackwell theorem tells us that with a sufficient statistic T we can improve any unbiased estimator that is not already a function of T by conditioning it on T(Y). Furthermore, this theorem implies that if T is sufficient for theta and if there is only one function of T that is an unbiased estimate of g(theta), that function is a minimum variance unbiased estimate of g(theta). Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Definition: Completeness The parameter family  is said to be complete if the condition E{f(Y)}=0 for all  implies that P(f(Y)=0)=1 for all . (Poor) Suppose that ={0,1,…,n}, ={0,1}, and For any function f on , we have The condition E{f(Y)}=0 for all  implies that However, an nth order polynomial has at most n zeros unless all of its coefficients are zero. Hence,  is complete. Example 5: y is the number of zeros or ones in n independent binomial trials depending on how you assign theta as the probability. Georgia Institute of Technology Center for Signal and Image Processing

Terminology and Preliminaries Definition: Exponential Families A class of distributions with parameter set  is said to be an exponential family if there are real-valued functions C,Q1,…,Qm,T1,…,Tm, and h such that T(y)=[T1(y),…,Tm(y)]T is a complete sufficient statistic. Exponential families are commonly encountered in problems involving Gaussian random variables. Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation For the random observation Y , indexed by a parameter m, our goal is to find a function such that is the best guess of the true value of  given Y=y. Bayesian estimators are the estimators that minimize the Bayesian risk function. The following estimators are commonly used in practice and can be distinguished by their cost functions. Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation Minimum-Mean-Squared-Error (MMSE): Euclidian Cost function: The posterior cost given Y=y is given by Minimizing this cost function also minimizes the Bayes risk . Hence, on differentiating with respect to , one can obtain the Bayes estimate Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation Minimum-Mean-Absolute-Error (MMAE): Absolute Error Cost function: The posterior cost given Y=y is given by Here we used the fact that with P(X0)=1, then MMAE 1of3 Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation Further simplification is also possible: MMAE 2of3 Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation Taking the derivative with respect to each , one can see that This derivative is a nondecreasing function of that approaches –1 as and +1 as . Thus achieves its minimum where its derivative changes sign: MMAE 3of3 Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation Maximum A Posteriori Probability (MAP): Uniform Error Cost function: The posterior cost given Y=y is given by Within some smoothness conditions, the estimator that maximizes this cost function is given by Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation Observations: MMSE Estimator: The MMSE estimate of  given Y=y is the conditional mean of  given Y=y . MMAE Estimator: The MMAE estimate of  given Y=y is the conditional median of  given Y=y . MAP Estimator: The MMAE estimate of  given Y=y is the conditional mode of  given Y=y . Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation Example 6: (Poor) Given the following conditional probability density function hence y has an exponential density with parameter . Suppose  is also exponential random variable with density Then, the posterior distribution of  given Y=y is given by for 0 and y0, and w(|y)=0 otherwise. Georgia Institute of Technology Center for Signal and Image Processing

Bayesian Parameter Estimation Example 7: (Continued.) The MMSE is the mean of this distribution: The MMAE is the median of this distribution: The MAP estimate is the mode of this distribution (where it is maximum): To decide which one to use, one must decide which three of the cost functions best suits the problem at hand. Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation Our goal is the same in Bayesian parameter estimation problem. Find . Assume that the parameter set  is real valued. In the nonrandom parameter estimation problem, we do not know anything about the true value of  other than the fact that it lies in . Hence, given the observation Y=y, what is the best estimate of  is the question we would like to answer. Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation The only average performance cost that can be done is with respect to the distribution of Y given , given a cost function C. A reasonable restriction to place on an estimate of  is that its expected value is equal to the true parameter value: For its tractability, the Euclidian norm squared cost function will be used. Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation When the squared-error cost is used, the risk function is the following: One can not generally expect to minimize this risk function uniformly for all . This is easily seen for the squared error cost since for any particular value of , say 0 the conditional mean-squared error can be made zero by choosing the estimate to be identically 0 for all observations y. However, if  is not close to 0, such an estimate would perform poorly. Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation With the unbiased-ness restriction, the conditional mean-squared error becomes the variance of the estimate. Hence, these estimators are termed minimum-variance unbiased estimators (MVUEs). The procedure for seeking MVUEs: Find a complete sufficient statistics T for . Find any unbiased estimator ĝ(y) of g(). Then, is an MVUE of g(). Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation Example 8: (Poor) Consider the model where N1,…,Nn are i.i.d. N(0,2) noise samples, and sk is a known signal for k=1,…,n. Our objective is to estimate  and 2. 1. The density of Y is given by where =[ 1 2 ]T and Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation Example 9: (Continued.) Note that T= [ T1 T2 ]T is a complete sufficient statistic for . 2. We wish to estimate Assuming that s10, the estimate ĝ1(y)=y1/s1 is an unbiased estimator of g1(). Moreover, note that and that Hence, is an unbiased estimate of g2(). Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation Example 10: (Continued.) 3. Since T1 and T2 are complete, the estimates are MVUEs of . Note that ĝ1(y) and T1 (y) are both linear functions of Y and are jointly Gaussian. Hence, MVUEs are Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation Maximum-Likelihood (ML) Estimation: For many problems arising in practice, it is not usually feasible to find MVUEs. Another method for seeking good estimators are needed. ML is one of the most commonly used methods in signal processing literature. Consider MAP estimation for : In the absence of any prior information about the parameter , we can assume that it is uniformly distributed (w() becomes a uniform distribution) since this represents the worst case scenario. Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation ML Estimation: (Continued.) Hence, the MAP estimate for a given y is any value of  that maximizes p(y) over . p(y) is usually called the likelihood ratio. Hence, the ML estimate is Maximizing p(y) is the same as maximizing log p(y) (log-likelihood function). Therefore, a necessary condition for the maximum-likelihood estimate is The above condition is also known as the likelihood equation. There are 2 things wrong with the MAP argument. First it is not always possible to construct a uniform distribution on Lambda, since Lambda may not be a bounded set. Second, assuming a uniform prior for the parameter is different from assuming that the prior is unknown or that the parameter is not a random variable. However, the ML turns out to be very useful. Georgia Institute of Technology Center for Signal and Image Processing

Nonrandom Parameter Estimation Cramer-Rao Bound: Let be some unbiased estimator of  Then the error covariance matrix is bounded by the Cramer-Rao bound (refer to the supplement). If the Cramer-Rao bound can be satisfied with equality, only the maximum likelihood estimate achieves it. Hence, if an efficient estimate exists, it is the maximum likelihood estimate. refer to the attached paper: “The Stochastic CRB for Array Processing: A Textbook Derivation” by Stoica, Larsson, and Gershman. Example 11: Georgia Institute of Technology Center for Signal and Image Processing

Georgia Institute of Technology Center for Signal and Image Processing Questions Georgia Institute of Technology Center for Signal and Image Processing