Why The Bretz et al Examples Failed to Work In their discussion in the Biometrical Journal, Bretz et al. provide examples where the implementation of the methodology in SAS failed to yield confidence intervals. This led to more documentation of the programs to help enable statisticians solve similar problems if encountered with applications. Greater documentation will also show a general approach to solving computationally difficult problems in SAS IML that seems to yield fairly satisfactory results. Examining these cases will also show how to modify the programs so that desired results can be obtained. It is important to remember that the methodology in the manuscript submitted to the Biometrical Journal assumes that σ is known and that the implementation also makes this assumption. Therefore, if values are changed without regard to this fact, it is easy to create datasets that are highly unlikely with real data.

In their first example, they notice that setting σ = 1 causes the program to fail. If we look at the first stage data with and and note that each random variable now has a standard deviation of 0.1, each of which is assumed to be from essentially the same population, we realize that we have a highly unusual observation, i.e. the two observations are greater than 10 standard deviations apart. Because the program assumes that both variables are coming from the same population, it uses the unbiased estimate of the difference along with the standard deviation of this estimate to determine the range of values for ∆ that are likely to yield a valid confidence interval… * If the lower and upper bounds are not set to specific values by the user, then the program assigns them to values that are likely to work; if lower=. then lower=y-y0-4#s#sqrt(1/nB+1/n0); if upper=. then upper=y-y0+4#s#sqrt(1/nB+1/n0); inc=(upper-lower)/ninter; Delta=do(lower,upper,inc); The variables, lower and upper, define the range of values used to search for the C.I. for ∆. This range depends on the Stage B mean and the control mean.

However, in the example provided, the range of values will not work. This is indicated by a plot provided by the program. σ = 10 σ = 1 The example provided on the left is given in the manuscript whereas the example on the right is the one provided by Bretz et al.. When σ = 10, we get the expected, reversed ‘S’ curve, and the program works fine, but when σ = 1, the probability does not approach 0 as ∆ becomes “large” in the plot. Instead, the plot abruptly ends as the probability nears Because the probability does not approach 0, the program fails to yield a confidence interval. However, the plot does give us a better idea where the confidence interval is likely to be (based on rough symmetry). By trial and error, the bounds are set from 1.5 to 2.0.

****************************************************; lower=1.5; upper=2.0; * "lower" and "upper" are variables defining the lower and upper bounds of Delta used to search for the C.I. See instructions for when it is necessary to manually define them.; **************************************************** Main Part of IML Program Starts Below ****************************************************; Values set by user by trial and error

Finally, when lower is set to 1.5 and upper is set to 2.5, the full, reversed ‘S’ curve is seen and the program provides an estimate of the confidence interval as [1.64, 2.03]. Because the difference between and is now statistically so great, the confidence interval is essentially the naive confidence interval. A similar problem is encountered when the selected mean is set to 7. Setting upper = 8 and lower = 2, an estimated confidence interval is found as [2.43, 6.35]. In cases involving real datasets, this kind of a problem is expected to be rare even when heavy selection is taking place from many treatments.