1. Item Parameter Estimation: Does WinBUGS Do Better Than BILOG-MG?
Bayesian Statistics, Fall 2009
Chunyan Liu & James Gambrell

2. Introduction 3 Parameter IRT Model
Assigns each item a logistic function with a variable lower asymptote.

3. Purpose
Compare BILOG-MG and WinBUGS estimation of item parameters under the 3 parameter logistic (3PL) IRT model
Investigate the effect of sample size on the estimation of item parameters

4. BILOG – MG (Mislevy & Bock 1985)
Propriety software
Uses unknown estimation shortcuts
Sometimes gives poor results
“Black Box” program
Very fast estimation
Provides only point estimates and standard errors for model parameters
Estimation method
Marginal Maximum Likelihood
Expectation-Maximization algorithm (Bock and Aitkin, 1981)
“Black Box” nature makes it hard to assess what is going on when things go wrong. Also hard to justify suspicious or unexpected results.

5. WinBUGS More open-source (related to OpenBugs) More widely studied
Might give more robust results
Much more flexible
Provides full posterior densities for model parameters
More output to evaluate convergence
Very slow estimation!

6. Literature Review
Most researchers have used custom-built MCMC samplers using Metropolis-Hastings- within-Gibbs algorithm
as recommended by Cowles, 1996!
Patz and Junker (1999a & b)
Wrote MCMC sampler in S plus
Found that their sampler produced estimates identical to BILOG for the 2PL model, but had some trouble with 3PL models.
Found MCMC was superior at handling missing data.

7. Literature Review Jones and Nediak (2000)
Developed “commercial grade” sampler in C++
Improved the Patz and Junker algoritm
Compared MCMC results to BILOG using both real and simulated data
Found that item parameters varied substantially, but the ICCs described were close according to the Hellinger deviance criterion
MCMC and BILOG were similar for real data
MCMC was superior for simulated data
Note that MCMC provides much more diagnostic out to assess convergence problems

8. Literature Review
Proctor, Teo, Hou, and Hsieh (2005 project for this class!)
Compared BILOG to WinBUGS
Fit a 2PL model
Only simulated a single replication
Did not use deviance or RMSE to assess error

9. Data Test: 36-item multiple choice
Item parameters (a, b and c) come from Chapter 6 of Equating, Scaling and Linking (Kolen and Brennan)
Treated as true item parameters (See Appendix)
Item responses simulated using 3PL model
a – slope b – difficulty
c – guessing – examinee ability

10. Methods
N (N=200, 500, 1000, 2000) θ values were generated from N(0,1) distribution.
N item responses were simulated based on the θ’s generated in step 1 and the true item parameters using the 3PL model.
Item parameters (a, b, c for the 36 items) were estimated using BILOG-MG based on the N item responses.
Item parameters (a, b, c for the 36 items) were estimated using WinBUGS based on the N item responses using the same prior as specified by BILOG-MG.
Repeat steps two and four 100 times. For each item, we have 100 estimated parameter sets from both programs

11. Priors a[i] ~ dlnorm(0, 4) b[i] ~ dnorm(0, 0.25) c[i] ~ dbeta(5,17)
Same priors used in BILOG and WinBUGS

12. Criterion-Root Mean Square Error (RMSE)
For each item, we computed the RMSE for a, b, and c using the same formula
where and
Here could be , , or and x could be the parameter of a, b or c

13. Results
1. Deciding the number of Burn-in Iterations- History Plots

14. Results-cont.
1. Deciding the number of Burn-in Iterations- Autocorrelation and BGR plots

15. Results-cont. 1. Deciding the number of Burn-in Iterations- Statistics
node mean sd MC error 2.5% median 97.5% start sample
a[1]
a[2]
a[3]
a[4]
a[5]
b[1]
b[2]
b[3]
b[4]
b[5]
c[1]
c[2]
c[3]
c[4]
c[5]

16. 1. Running conditions for WinBUGS
Results-cont.
1. Running conditions for WinBUGS
Adaptive phase: 1000 iterations
Burn-in: 1500 iterations
For computing the Statistics: 3500 iterations
Using 1 chain
Using bugs( ) function to run WinBUGS through R
Need BRugs and R2WinBUGS packages

17. Results-cont.
2. Effect of Sample Size

18. BILOG-MG vs. WinBUGS – a parameter
Results-cont.
BILOG-MG vs. WinBUGS – a parameter

19. BILOG-MG vs. WinBUGS - b parameter
Results-cont.
BILOG-MG vs. WinBUGS - b parameter

20. BILOG-MG vs. WinBUGS - c parameter
Results-cont.
BILOG-MG vs. WinBUGS - c parameter

21. Discussion & Conclusions
Larger sample size decreased RMSE for all parameters under both programs.
For N=200, there was a significant convergence problem for BILOG-MG. No problem with WinBUGS.

22. Discussion & Conclusions-cont.
Slope parameter “a”
WinBUGS was superior to BILOG when N = 500 or less
More accurately estimated for items without extreme a or b parameters by both programs.
Difficulty parameter “b”
BILOG was superior to WinBUGs when N = 500 or less
Both programs had larger error for items either too difficult or too easy
Guessing parameter “c”
WinBUGs was superior to BILOG at all sample sizes, but especially at N = 1,000 or less
More accurately estimated for difficult items by both programs.
Both programs had larger error for items with shallow slopes.

23. Limitations Only one chain is used in the simulation study.
Some of the MC errors are not less than 1/20 of the standard deviation, could use more iterations in MCMC sampler
Simulated data
Conforms to the 3PL model much more closely than real data would
No missing responses
No omit problems
Fewer low scores

24. WinBUGS code for running 3PL
model 3PL; { for (i in 1:N) { for (j in 1:n) { e[i,j]<-exp(a[j]*(theta[i]-b[j])) p[i,j] <- c[j]+(1-c[j])*(e[i,j]/(1+e[i,j])) resp[i,j] ~ dbern(p[i,j]) } theta[i] ~ dnorm(0,1) for (i in 1:n) { a[i] ~ dlnorm(0, 4) b[i] ~ dnorm(0, 0.25) c[i] ~ dbeta(5,17)

25. True Item Parmaeters item a b c 1 0.5496 -1.796 0.1751 19 0.6562
0.3853
0.1201 2
0.7891
0.1165 20
1.0556
0.9481
0.2036 3
0.4551
0.2087 21
0.3479
2.2768
0.1489 4
1.4443
0.4833
0.2826 22
0.8432
1.0601
0.2332 5
0.974
-0.168
0.2625 23
1.1142
0.5826
0.0644 6
0.5839
0.2038 24
1.4579
1.0241
0.2453 7
0.8604
0.4546
0.3224 25
0.5137
1.379
0.1427 8
1.1445
0.2209 26
0.9194
1.0782
0.0879 9
0.7544
0.0212
0.16 27
1.8811
1.4062
0.1992 10
0.917
1.0139
0.3648 28
1.5045
1.5093
0.1642 11
0.9592
0.7218
0.2399 29
0.9664
1.5443
0.1431 12
0.6633
0.0506
0.124 30
0.702
2.2401
0.0853 13
1.2324
0.4167
0.2535 31
1.2651
1.8759
0.2443 14
1.0492
0.7882
0.1569 32
0.8567
1.714
0.0865 15
1.069
0.961
0.2986 33
1.408
1.5556
0.0789 16
0.9193
0.6099
0.2521 34
0.5808
3.4728
0.1399 17
0.8935
0.5128
0.2273 35
0.9257
3.1202
0.109 18
0.9672
0.195
0.0535 36
1.2993
2.1589
0.1075

26. Acknowledgement
Professor Katie Cowles

27. Questions?