1. An Introduction to the Statistics of Uncertainty
Tom LaBone
April 17, 2009
SRCHPS Technical Seminar
MJW Corporation
University of South Carolina

2. Introduction
All physical measurements must be reported with some quantitative measure of the quality of the measurement
needed to decide if the measurement is suitable for a particular purpose
The concept of “uncertainty” was developed in metrology to partially fill this need
The US Guide to the Expression of Uncertainty in Measurement ANSI/NCSL Z (R2007) provides guidance on calculating and reporting the uncertainty in a measurement
US version of the ISO guide
referred to as the “GUM”

3. Overview
Illustrate the use of the GUM methodology using a relatively simple physical system
Combined Standard Uncertainty
Type A uncertainty only
Probability Distributions
Expanded Uncertainty
Monte Carlo Methods
Type B Uncertainty
Student’s t Distribution

4. Example
The SRS Health Physics Instrument Calibration Laboratory “sells” its radiation fields as a product
The uncertainty attached to a radiation field helps the customer decide if the “product” is suitable for their application
This is a rather involved case for such a short talk, so let us work with a less complex example

5. What is the density of the cube?
Measure the height, width, length, and mass of the cube
Calculate the density r using this formula
Single measurements

6. Measurand
The measurands we directly measure (mass and dimensions) are called input quantities
The measurand we calculate (the density) is called the output quantity
In this discussion the input quantities are assumed to be uncorrelated
e.g., the measurement of the height does not influence the measurement of the length

7. Variability
If we repeated the measurements again would we expect to see exactly the same result?
Our measurements of dimension and mass will exhibit variability
if we measure the “same thing” repeatedly we are likely get a range of answers that vary in a seemingly random fashion

8. Why Do Measurements Vary?
Every measurement is influenced by a multitude of quantities that are not under our control and of which we may not even be aware (influence quantities)
Random effects
Measurements also vary because the measurand is not and cannot be specified in infinite detail
For example, I did not specify how the linear measurements of the cube should be made

9. Errors
Using the input and output quantities we have defined the “true” value of the density
The error in a measurement is defined as
The true value and hence the error are unknowable, but errors can be classified by how they influence the measurement
random and systematic errors
error = measured value of density – “true” value of density

10. Types of Errors
Random errors result from random effects in the measurement
the magnitude and sign of a random error changes from measurement to measurement
measurements cannot be corrected for random errors
…but random errors can be quantified and reduced
Systematic errors results from systematic effects in the measurement
the magnitude and sign of a systematic error is constant from measurement to measurement
measurements can be corrected for known systematic errors
…but the correction introduces additional random errors

11. What can we do about random errors?
Law of Large Numbers
If you repeat measurements many times and take the mean, this sample mean is a good estimator of the true population mean and is taken to be the best estimate of the thing we defined as the measurand
Plug the sample means into the equation to obtain the best estimate of r
sample means

12. Repeated Measurements
Sample Mean
central tendency

13. Precision of Result
Precision is the number of digits with which a value is expressed
The calculations here were performed to the internal precision of the computer (~16 digits)
The density is arbitrarily presented with 9 digits of precision
In which digit do we lose physical significance? ?

14. Uncertainty
“…parameter associated with the result of a measurement that characterizes the dispersion of the values that could reasonably be attributed to the measurand….”
an interval that we are reasonably confident contains the true value of the measurand
the terms “random” and “systematic” are used with the term “error” but not with the term “uncertainty”
associated with the measurement, not the measurement process

15. Evaluation of Uncertainty
Type A evaluation of uncertainty
evaluation of uncertainty by the statistical analysis of repeated measurements
called Type A uncertainty
Type B evaluation of uncertainty
evaluation of uncertainty by any other method
called Type B uncertainty

16. Repeated Measurements
dispersion
central tendency
Sample mean
sample standard
deviation

17. Standard Uncertainty of Inputs
The sample standard deviation s is a term in statistics with a precise meaning
In metrology the analogous term is standard uncertainty u
For Type A evaluations the standard deviation is the standard uncertainty
This may not true for Type B evaluations

18. Significant Digits Report uncertainty to 2 digits
round to even number if the last digit is 5
Round the measurement to agree with the reported uncertainty

19. Uncertainty in Density
We have calculated the standard uncertainty in the input quantities (length, mass, etc)
How do we get the standard uncertainty in the output quantity (density)?
the combined standard uncertainty
Propagation of uncertainty

20. Combined Standard Uncertainty
sensitivity coefficient (often abbreviated as “c”)
Given a small change in the length of
the cube how much does the density change?
Units must match up properly!

21. Standard Deviation of the Mean
describes how repeated estimates
of the mean are scattered around
their grand mean (mean of the means)
describes how individual
measurements are scattered
around their mean

22. Which Standard Deviation Should We Use?
Sample standard deviation
If you want to describe how individual measurements are scattered about their mean
Standard deviation of the mean
If you want to describe how multiple estimates of the mean are scattered about their grand mean
also called the standard error of mean
We need to use the standard deviation of the mean in the error propagation

23. Combined Standard Uncertainty
Type A uncertainty only
r = g/cm3 with a combined
standard uncertainty uc = 8.8 x 10-4 g/cm3

24. Where We Are
We have calculated the density and its combined standard uncertainty (Type A uncertainty only)
Next, we want to
calculate the expanded uncertainty and
address the Type B uncertainty
But, we need to discuss probability distributions and other such things first

25. Probability Distributions
Up to this point we have described our data with
the mean (central tendency)
the standard deviation (dispersion)
The mean and standard deviation do not uniquely specify the data
Use a mathematical model that defines the probability of observing any given result
probability density function (pdf)

26. Uniform (Rectangular) PDF
otherwise
a = 1 b = 9
m-s
m+s
a=1
b=9

27. Rectangular PDF Notation
f(x) is the rectangular probability density function
the value of the pdf is not the probability
the area under the pdf is probability
note that f(x) has units – probability has no units
m is the population mean
s is the population standard deviation
a is the lower bound of the distribution
a is a parameter in the pdf
the probability of observing a value of x less than a is zero
b is the upper bound of the distribution
b is a parameter in the pdf
the probability of observing a value of x greater than b is zero

28. Probability P(x < m-1s) =0.2113249 P(x < m+1s) =0.7886751
The area under the curve

29. Normal PDF m=5 m-s m+s The population parameters
are the parameters in the
pdf – this is unusual

30. Probability P(x < m-1s) =0.1586553 P(x < m+1s) =0.8413447
The area under the pdf curve

31. Normal vs Rectangular P(x < m+1s) =0.7886751
same mean and
standard deviation

32. Sample Statistics and Population Parameters
No matter what the probability distribution is, the sample mean and standard deviation are the best estimates (based on the observed data) of the population mean and standard deviation
Sample Statistics
Population Parameters

33. Random Numbers 1000 numbers drawn at random
from the rectangular distribution
1000 numbers drawn at random
from the normal distribution

34. Uses of PDFs
We use the rectangular pdf to describe a random variable that is bounded on both sides and has the equal probability of appearing anywhere between the bounds
The normal distribution has a special place in statistics because of the Central Limit Theorem

35. Central Limit Theorem
As the sample size N gets “large”, the mean of a sample will be normally distributed regardless of how the individual values are distributed
Theorem provides no guidance on what “large” is
The standard deviation of the mean (aka the standard error of the mean) is equal to

37. So What?
No matter what probability distribution you start with, if the sample is large enough the means of data drawn from that distribution are normally distributed
What are the practical implications of this?
All the input quantities (length, etc) are means
The input quantities are normally distributed
The output quantity (density) is normally distributed

38. Normal Probabilities The area under the normal curve
m-1s
m+1s
m-1.96s
m+1.96s
The area under the normal curve
between m-1s and m+1s =
The area under the normal curve
between m-1.96s and m+1.96s = 0.95

39. Expanded Uncertainty
It is often desirable to express the uncertainty as an interval around the measurement result that contains a large fraction of results that might reasonably be observed
This is accomplished by using multiples of the standard uncertainty
the multiplier is called the coverage factor

40. Intervals Confidence interval Coverage interval
interval constructed with standard deviations from known probability distributions
the interval has an exact probability of covering the mean value of the measurand
Coverage interval
interval constructed with uncertainties
the interval does not have an exact probability of covering the mean value of the measurand
only an approximation
uses a coverage factor rather than a standard normal quantile (e.g., the 1.96)
coverage factor of 2 (~95%) or 3 (~99%) is typically used

41. Expanded Uncertainty for Density
Type A uncertainty only
68% Confidence Interval
95% Confidence Interval
95% Coverage Interval

42. Monte Carlo Methods
Evaluation of measurement data – Supplement 1 to the “Guide to the expression of uncertainty in measurement” – Propagation of distributions using a Monte Carlo method OIML G (2008)
Run a statistics experiment using random numbers

43. Calculate the sample mean and
standard deviations of the 106 densities
95% empirical CI

44. Implementation in R for (i in 1: (10^6)) {
d[i] <- rnorm(1,M,s.M) / (rnorm(1,L,s.L) * rnorm(1,W,s.W) * rnorm(1,H,s.H)) }
quantile(d,probs=c(0.025,0.975))
mean(d)
sd(d)
draw a random length,
width, and height
draw a random mass
calculate a density
calculate the empirical 95% confidence interval, mean and standard deviation

45. Advantages of Monte Carlo
Intuitive
Set up the experiment in the computer just like it occurs in the lab
Able to handle
very complex problems
asymmetric probability distributions
No need to mess with the t distribution or effective degrees of freedom
you will see what I am talking about shortly

46. Type B Uncertainty Assessment
Calipers
Used to measure length, height, and width
“Accuracy” of ± 0.02 mm (± cm) for measurements <100 mm
Scale
Used to measure “mass”
“Accuracy” of ± gram
What do they mean by “accuracy” and how do I use this information?

47. Calipers
The “accuracy” of ± cm is taken to mean that if I moved the calipers from cm to cm the reading could be anywhere from cm to cm
Assume rectangular distribution with an upper limit of X cm and a lower limit of X – cm
The standard uncertainty of this distribution is

48. Scale
The “accuracy” of ± gram is taken to mean that if the weight increased from grams to grams the reading could be anywhere from grams to grams
Assume rectangular distribution with an upper limit of X grams and a lower limit of X – grams
The standard uncertainty of this distribution is

49. Standard Uncertainty for Length
Combine the Type A and Type B uncertainties in quadrature (i.e., add the variances)
Includes Type A and
Type B uncertainties
The notation uc(L) is used here to
indicate the uncertainty includes
Type A and B uncertainties

50. Combined Standard Uncertainty for Density
Type A and B uncertainty
r = g/cm3 with a combined
standard uncertainty uc = 2.1 x 10-3 g/cm3

51. Type B Uncertainties
Remember, once Type B uncertainties are included in the error propagation the equivalence of standard deviations and standard uncertainties is usually lost
The assessment of Type B uncertainties usually requires some degree of professional judgment and experience
Once you decide what the Type B uncertainty is, it is treated the same as a Type A uncertainty
One goal of GUM is to make Type B uncertainties easier to handle by having people report the right information along with the uncertainty itself

52. Summary of Jargon Standard uncertainty Combined standard uncertainty
uncertainty of the result of a single type of measurement (e.g., length)
includes Type A and/or Type B uncertainties
Combined standard uncertainty
standard uncertainties from multiple types of measurements used to calculate an output quantity (e.g., density)
Expanded uncertainty
the standard uncertainty multiplied by a coverage factor
combined standard uncertainty multiplied by a coverage factor

53. Where We Are For the density, we have calculated
combined standard uncertainty
expanded uncertainty
(including both Type A and Type B uncertainties)
The only remaining issue is to account for the impact “small” samples have on the expanded uncertainty

54. Student’s t Distribution
If s is given, i.e., not determined from the
data, then k is a standard normal quantile,
e.g., 1.96 gives a 95% confidence interval
If s is not given and we use s, which is
determined from the data, then k is a not
standard normal quantile, it is a quantile
from a Student’s t distribution

55. Student’s t PDF n = degrees of freedom (df) t
The t distribution looks a lot like
the normal distribution but has
“fatter” tails
normal

56. So What?
A 95% coverage interval for a normal distribution is given by 1.96s
A 95% coverage interval for a t distribution might be more like 3s (depending the df)
The t distribution converges to the normal distribution as the degrees of freedom (the size of the sample) gets large
The t distribution is needed to calculate coverage intervals for small samples taken from normally distributed data
for large samples you can use normal quantiles

57. Expanded Uncertainty for Length (Type A uncertainty only)
mean of N = 15 measurements (see slide 21)
standard error of the mean length (see slide 21)
standard normal quantile for p = 0.95
95% confidence interval
t quantile for p = 0.95 (14 df)
95% confidence interval

58. Degrees of Freedom
Data can be used to estimate parameters or estimate variance
degrees of freedom is the number of data available to estimate variance after the parameters are estimated
one degree of freedom was used to calculate the mean which leaves = 14 left to calculate variance
The GUM views degrees of freedom as an indication of the uncertainty in the uncertainty
large degrees of freedom = small uncertainty
small degrees of freedom = large uncertainty

59. Expanded Uncertainty for Length (Type A and Type B uncertainty)
The Type A uncertainty of the length has 14 degrees of freedom
How many degrees of freedom does the Type B uncertainty have?
unless “they” tell you the degrees of freedom, you end up doing a bit of hand waving on the df for Type B uncertainties
the GUM gives a way to get an approximate df
How many degrees of freedom does the combined uncertainty of Length have?
There is no exact solution to this problem
An approximate solution is given by the Welch-Satterthwaite equation

60. Degrees of Freedom for Type B Uncertainty
Where (Du/u) is the relative uncertainty in
the uncertainty
25% relative uncertainty in the uncertainty
10% relative uncertainty in the uncertainty
Zero uncertainty in the uncertainty

61. Welch Satterthwaite
Gives the effective degrees of freedom associated with the expanded uncertainty in the length
The sensitivity coefficients c
are equal to 1 here
The Type B uncertainty is assumed
to have an infinite degrees of freedom,
i.e., the uncertainty has no uncertainty
Use this to calculate the t quantile

62. Expanded Uncertainty for Length (Type A and Type B uncertainty)
mean of N = 15 measurements
standard uncertainty in length (see slide 49)
standard normal quantile for p = 0.95
95% coverage interval
t quantile for p = 0.95 (323 df)
95% coverage interval

63. Expanded Uncertainty for Density (Type A Uncertainty Only)

64. standard uncertainty uc = 8.8 x 10-4 g/cm3
r = g/cm3 with a combined
standard uncertainty uc = 8.8 x 10-4 g/cm3
(see slide 23)
(see slide 23)
standard normal quantile for p = 0.95
95% confidence interval
t quantile for p = 0.95 (41 df)
95% confidence interval

65. Expanded Uncertainty for Density (Type A and Type B Uncertainty)
The Type B uncertainty is assumed
to have an infinite degrees of freedom,
i.e., the uncertainty has no uncertainty
Same as the normal distribution

66. standard uncertainty uc = 2.1 x 10-3 g/cm3
r = g/cm3 with a combined
standard uncertainty uc = 2.1 x 10-3 g/cm3
(see slide 50)
(see slide 50)
standard normal quantile for p = 0.95
95% coverage interval
t quantile for p = 0.95 (1200 df)
95% coverage interval – the final answer!

67. Why Bother with Student?
It is important not to overstate your confidence in a number
if you make an error calculating the coverage interval, try to make it too big
When you start to include all known sources of uncertainty, some are very likely to have small degrees of freedom
the weakest link determines the strength of the chain
same goes for uncertainty calculations

68. Monte Carlo
R Code
for (i in 1:(10^6)) {
r.L <- (L + rt(1,14)*s.L)
r.L <- runif(1,r.L-0.002,r.L+0.002)
r.W <- (W + rt(1,14)*s.W)
r.W <- runif(1,r.W-0.002,r.W+0.002)
r.H <- (H + rt(1,14)*s.H)
r.H <- runif(1,r.H-0.002,r.H+0.002)
r.M <- (M + rt(1,14)*s.M)
r.M <- runif(1,r.M ,r.M )
d[i] <- r.M / (r.L * r.W * r.H) }
quantile(d,probs=c(0.025,0.975))
mean(d)
sd(d)
Expanded Uncertainty for Density (Type A and Type B Uncertainty)
MC 95% coverage interval = (1.4626, )
GUM 95% coverage interval = (1.4625, ) (see slide 66)

69. Summary Combined Standard Uncertainty Probability Distributions
Type A uncertainty only
Probability Distributions
rectangular, normal
Expanded Uncertainty
Monte Carlo Methods
Type B Uncertainty
uncertainty in calipers and scale
Student’s t Distribution
degrees of freedom in Type B uncertainty
degrees of freedom in combined uncertainty
Welch-Satterthwaite

70. Recommended Reading
The GUM and its Monte Carlo supplement already cited
An Introduction to Uncertainty in Measurement by Les Kirkup and Bob Frenkel (Cambridge University Press:2006)
An Introduction to Error Analysis by John Taylor (University Science Book: 1982)