1. STATISTICAL INFERENCE PART III
EXPONENTIAL FAMILY, FISHER INFORMATION, CRAMER-RAO LOWER BOUND (CRLB)

2. EXPONENTIAL CLASS OF PDFS
X is a continuous (discrete) rv with pdf f(x;), . If the pdf can be written in the following form
then, the pdf is a member of exponential class of pdfs of the continuous (discrete) type. (Here, k is the number of parameters)

3. REGULAR CASE OF THE EXPONENTIAL CLASS OF PDFS
We have a regular case of the exponential class of pdfs of the continuous type if
Range of X does not depend on .
c() ≥ 0, w1(),…, wk() are real valued functions of  for .
h(x) ≥ 0, t1(x),…, tk(x) are real valued functions of x.
If the range of X depends on , then it is called irregular exponential class or range-dependent exponential class.

4. EXAMPLES
X~Bin(n,p), where n is known. Is this pdf a member of exponential class of pdfs? Why?
Binomial family is a member of exponential family of distributions.

5. EXAMPLES
X~Cauchy(1,). Is this pdf a member of exponential class of pdfs? Why?
Cauchy is not a member of exponential family.

6. EXPONENTIAL CLASS and CSS
Random Sample from Regular Exponential Class
is a css for .
If Y is an UE of , Y is the UMVUE of .

7. EXAMPLES Let X1,X2,…~Bin(1,p), i.e., Ber(p).
This family is a member of exponential family of distributions.
is a CSS for p.
is UE of p and a function of CSS.
is UMVUE of p.

8. EXAMPLES
X~N(,2) where both  and 2 is unknown. Find a css for  and 2 .

9. FISHER INFORMATION AND INFORMATION CRITERIA
X, f(x;), , xA (not depend on ).
Definitions and notations:

10. FISHER INFORMATION AND INFORMATION CRITERIA
The Fisher Information in a random variable X:
The Fisher Information in the random sample:
Let’s prove the equalities above.

11. FISHER INFORMATION AND INFORMATION CRITERIA

12. FISHER INFORMATION AND INFORMATION CRITERIA

13. FISHER INFORMATION AND INFORMATION CRITERIA
The Fisher Information in a random variable X:
The Fisher Information in the random sample:
Proof of the last equality is available on Casella & Berger (1990), pg

14. CRAMER-RAO LOWER BOUND (CRLB)
Let X1,X2,…,Xn be sample random variables.
Range of X does not depend on .
Y=U(X1,X2,…,Xn): a statistic; does’nt contain .
Let E(Y)=m().
Let prove this!

15. CRAMER-RAO LOWER BOUND (CRLB)
-1Corr(Y,Z)1
0 Corr(Y,Z)21  
Take Z=′(x1,x2,…,xn;)
Then, E(Z)=0 and V(Z)=In() (from previous slides).

16. CRAMER-RAO LOWER BOUND (CRLB)
Cov(Y,Z)=E(YZ)-E(Y)E(Z)=E(YZ)

17. CRAMER-RAO LOWER BOUND (CRLB)
E(Y.Z)=mʹ(), Cov(Y,Z)=mʹ(), V(Z)=In()
The Cramer-Rao Inequality
(Information Inequality)

18. CRAMER-RAO LOWER BOUND (CRLB)
CRLB is the lower bound for the variance of an unbiased estimator of m().
When V(Y)=CRLB, Y is the MVUE of m().
For a r.s., remember that In()=n I(), so,

19. LIMITING DISTRIBUTION OF MLEs
: MLE of 
X1,X2,…,Xn is a random sample.

20. EFFICIENT ESTIMATOR T is an efficient estimator (EE) of  if
T is UE of , and,
Var(T)=CRLB
Y is an efficient estimator (EE) of its expectation, m(), if its variance reaches the CRLB.
An EE of m() may not exist.
The EE of m(), if exists, is unique.
The EE of m() is the unique MVUE of m().

21. ASYMPTOTIC EFFICIENT ESTIMATOR
Y is an asymptotic EE of m() if

22. EXAMPLES A r.s. of size n from X~Poi(θ). Find CRLB for any UE of θ.
Find UMVUE of θ.
Find an EE for θ.
Find CRLB for any UE of exp{-2θ}. Assume n=1, and show that is UMVUE of exp{-2θ}. Is this a reasonable estimator?

23. EXAMPLE
A r.s. of size n from X~Exp(). Find UMVUE of , if exists.

24. Summary We covered 3 methods for finding UMVUE:
Rao-Blackwell Theorem (Use a ss T, an UE U, and create a new statistic by E(U|T))
Lehmann-Scheffe Theorem (Use a css T which is also UE)
Cramer-Rao Lower Bound (Find an UE with variance=CRLB)