NASSP Masters 5003F - Computational Astronomy Lecture 7 – chi squared and all that Testing for goodness-of-fit continued. Uncertainties in the fitted parameters. Confidence intervals. The Null Hypothesis.

Hypothesis testing continued. Procedure: 1.“Suppose the model is a perfect fit.” 2.Calculate survival function for χ 2 of pure noise of N-M degrees of freedom.. 3.Draw vertical at point of measured χ 2. 4.Y value where this vertical intercepts the SF is the probability that a perfect model would have this χ 2 value by random fluctuation. NASSP Masters 5003F - Computational Astronomy Survival function

Questions answered so far: In fitting a model, we want: 1.The best fit values of the parameters; 2.Then we want to know if these values are good enough! Ie if the model is a good fit to the data. 3. If the model passes, we want uncertainties in the best-fit parameters. Number 1 is accomplished. √ Number 2 is accomplished. √ NASSP Masters 5003F - Computational Astronomy

Uncertainties in the best-fit parameters Usually what one gets is a covariance matrix (mentioned in lecture 4): This is a symmetric matrix: σ ij 2 =σ ji 2 for all i,j. For U= χ 2, E=2(H bestfit ) -1, where H bestfit is the Hessian, evaluated at the best-fit values of the θ i. For U=-L, E=F -1, where F is the “Fisher Information Matrix”: These definitions are equivalent! –For Gaussian data, identical. NASSP Masters 5003F - Computational Astronomy

The Hessian or curvature matrix The contours are ellipses in the limit as the minimum is approached. –Ellipsoidal hypercontours in the general case that M>2. Semiaxes aligned with the eigenvectors of H. Small semiaxis:  large curvature;  small uncertainty in that direction. NASSP Masters 5003F - Computational Astronomy Contours of U: Arrows show the eigenvectors.

1-parameter example 1)Gaussian data, U= χ 2. –For this simple model, we can find the best fit θ without numerical minimization: –Setting this to zero gives: NASSP Masters 5003F - Computational Astronomy

Sidebar – optimum weighted average A weighted average is: Since the y i are random variables, so is μ ^. Therefore it will have a PDF and an uncertainty σ μ. The smallest uncertainty is given for –Exactly what we have from the χ 2 fit. NASSP Masters 5003F - Computational Astronomy

Back to the1-parameter example. –Again, because this model is so simple, we can calculate σ θ by direct propagation of uncertainties. θ^ is a function of N uncorrelated random variables y i, so It is fairly easy to show that: NASSP Masters 5003F - Computational Astronomy

What does the standard approach give? Hessian is a 1-element matrix: Hence QED. NASSP Masters 5003F - Computational Astronomy

1-parameter example continued 2)Poisson data, U= -L. (No point in using -L for gaussian data, it’s then mathematically the same as chi squared.) –Again it is simple to calculate the position of the minimum directly: –Setting this to zero gives Ie, the average of the ys. NASSP Masters 5003F - Computational Astronomy

Uncertainties in the Poisson/L case. –With our present simple model it is very easy by propagation of uncertainties to show that –Following the formal procedure for comparison: –Inverting this gives the same result. NASSP Masters 5003F - Computational Astronomy

1-parameter example continued 3)Poisson data, U=“chi squared”. –There are two flavours of “chi squared” for Poisson data! –Note that the following is simply incorrect: NASSP Masters 5003F - Computational Astronomy

Don’t use Pearson’s for fitting. It is not hard to prove it is biased. –Eg, keeping our simple model, –Setting this to zero gives –In his paper, Mighell calculates the limiting value of θ^ Pearson as N->∞ and shows it is not θ. NASSP Masters 5003F - Computational Astronomy

The Mighell formula is unbiased. –For this statistic, –Setting this to zero gives –Some not-too-hairy algebra shows that the limiting value of θ^ Mighell as N->∞ is equal to θ. NASSP Masters 5003F - Computational Astronomy

Goodness-of-fit: 1.The Gaussian/ χ 2 case has been covered already. 2.The Poisson/L case is a problem, because no general PDF for L is known for this noise distribution. –If we insist on using this, have to estimate SF via a Monte Carlo. Messy, time-consuming. 3.For the Poisson/”chi squared” case, where we have 2 competing formulae, we should do: –Use Mighell to fit; –Use Mighell for uncertainties; –But use Pearson (with the best-fit values of θ i ) for goodness-of-fit hypothesis testing. Because it has the same PDF (thus also SF) as χ 2. NASSP Masters 5003F - Computational Astronomy

Confidence intervals There is a hidden assumption behind frequentist model fitting: namely that it is meaningful to talk about p(θ i ^). NASSP Masters 5003F - Computational Astronomy

We already have some hints about its shape… and a Monte Carlo seems to offer a way to map it as accurately as we want. Confidence intervals NASSP Masters 5003F - Computational Astronomy

Bayesians think this is nonsense. Such a MC is like pretending that θ ^ is the ‘true’ value, and then generating lots of hypothetical experimental data. But all we really know is the single set of data which we measure in the real experiment. –Plus possibly some ‘prior knowledge’. We don’t want p( θ ^), we want p( θ ). But we’ll continue with the frequentist way for the time being. NASSP Masters 5003F - Computational Astronomy

Confidence intervals We also assume that p(θ i ) is approximately Gaussian (which may be entirely unwarranted!!) –We interpret this to mean that there is a 68% chance that the interval contains the truth value θ. NASSP Masters 5003F - Computational Astronomy

Confidence intervals Note that this is not the only interval which contains 68% of the probability. We can move the interval up and down the θ axis as we please. The –σ to +σ version is just a convention. FYI erf() is called the error function. NASSP Masters 5003F - Computational Astronomy

Confidence intervals For more than 1 parameter the q% confidence interval is the (hyper)contour within which the probability of the truth value occuring =q. Again, by convention, symmetrical contours are used. NASSP Masters 5003F - Computational Astronomy

When m=s+b (which is not always appropriate) It is of interest to ask (probably before we attempt to fit the parameters of s!): –Is there any signal present at all? In frequentist statistics this is again done via hypothesis testing. The hypothesis now is called the null hypothesis (‘null’ from Latin for ‘nothing’): –“Suppose there is no signal at all.” –and test what follows from this. NASSP Masters 5003F - Computational Astronomy

Testing the Null Hypothesis - details 1)Gaussian data, U= χ 2 : –Construct the survival function (SF). Degrees of freedom? –Depends whether we fit the background or not. –Suppose we have M b and M s. –If background fitted, υ =N-M b. –If not (in this case need to know the background from other information), υ =N. –From the set of measurements y i, calculate –From the SF read off that value of probability which corresponds to U meas. That is the probability that background alone would generate >=U meas. NASSP Masters 5003F - Computational Astronomy Note ONLY include background!

Testing the Null Hypothesis – details cont. 2)Poisson data, U=“ χ 2 ”: –The PDF, therefore the SF, are not known for the Mighell statistic. –However the PDF and SF for the Pearson statistic are identical to χ 2. –  Use Pearson statistic for Poisson hypothesis testing. 3)Poisson data, U=-L: –PDF and SF not known. –But one can compare two models via the Cash statistic. (Cash W, Ap J 228, 939 (1979). NASSP Masters 5003F - Computational Astronomy

The Cash statistic This is only valid providing the null model can be obtained by some combination of signal parameters. –This implies that one of the signal parameters will be an amplitude (ie, a scalar multiplying the whole signal function). –It also ensures that NASSP Masters 5003F - Computational Astronomy hence

The Cash statistic Cash showed that the PDF of C was the same shape as that of χ 2, but with υ =M fitted. Note that this is rather different from the usual p( χ 2 ), for which υ is approx. equal to the number of data values N. NASSP Masters 5003F - Computational Astronomy

Incomplete gamma functions - advice Recall the survival function for χ 2 is –The incomplete gamma function can be calculated via scipy.special.gammainc. It is very small values of P that we are interested in however – ie where Г ( υ /2,U/2)/ Г ( υ /2) becomes close to 1. In this regime it is better to use the complementary (means, 1 minus) incomplete gamma function: –scipy.special.gammaincc <– note 2 cs. –But NOTE the definition carefully. NASSP Masters 5003F - Computational Astronomy

General problems with fitting: When some of the θs are ‘near degenerate’. –Solution: avoid this. When several different models fit equally well (or poorly). –Solution: F-test (sometimes). Supposedly restricted to the case in which 2 models differ by an additive component.

NASSP Masters 5003F - Computational Astronomy Degenerate θs Data Model: two close gaussians – 2 par- ameters: the amp of each gaussian. Valley in U is long and narrow. Many combinations of θ 1 and θ 2 give about as good fit; parameters strongly correlated.

Overview of the grand plan NASSP Masters 5003F - Computational Astronomy Frequentist Bayesian T B D… Bayesian T B D… χ2υ=N-Mχ2υ=N-M χ2υ=N-Mχ2υ=N-M Null H Poisson -LPoisson χ 2 Gaussian ( χ 2  -L) GOF Uncert Fit No formula (MC) No formula (MC) U Pearson υ =N-M U Pearson υ =N-M χ2υ=Nχ2υ=N χ2υ=Nχ2υ=N Cash υ =M Cash υ =M U Pearson υ =N U Pearson υ =N Minimize χ 2 Minimize -L Minimize U Mighell E=2H -1 E=-F -1 E=2H -1

Flowchart to disentangle the uses of χ 2 : NASSP Masters 5003F - Computational Astronomy Minimize χ 2 to get best-fit θ. Test the hypothesis that it is.Test the Null Hypothesis: Compare to theoretical χ 2 surviv- al function (num deg free = N). P<P cut ? Yes – there is a signal No – no signal. P<P cut ? No – model is good. Yes – model is bad. Compare to theoretical χ 2 surviv- al function (num deg free = N-M). Is there any signal at all? Is the model an accurate des- cription? Decide on a cutoff probability P cut. Calculate χ 2 for the best fit θ. Decide on a cutoff probability P cut. Calculate χ 2 for θ= bkg values.