1. School Performance Variation on COMLEX Level 1 Linjun Shen, Ph.D., MPH National Board of Osteopathic Medical Examiners

2. Purposes n Provide more information to school administrators and educators n Help school administrators and educators to interpret and analyze COMLEX data n Demonstrate analytical approaches individual schools can adopt in digesting their own COMLEX data

3. Limitations n Analysis is limited to Level 1 June 2002 administration n Variables available for this analysis is limited, so as the results

4. A model of the medical school- licensing examination relationship

5. Scope of This Analysis n Interest of the analysis: school mean scores and school passing rates on Level 1. n Potentially influential factors: gender of students, sector of school, school mean MCAT. n Analytical approach: mainly descriptive and graphical with a brief statistical modeling.

6. Summary of June 2002 Level 1

7. June 2002 Level 1

8. Level 1 School Mean

9. Highest and Lowest on Level 1

10. School Mean Score Distribution

11. School Failing Rate Distribution

12. Failing Rate by Ranking of Means

13. Passing Rate by Ranking of Means

14. School Mean Performance n Two ways to rank schools: by mean score and passing rate n Numerical ranking is not as important as qualitative grouping n Variation among school mean scores is substantial (53 point between the highest and lowest, 40 point between above and below average groups)

15. Significant Gender Differences on June 2002 Level 1

16. Gender Difference in Mean Scores by School

17. Gender Difference in Failing Percentage by School

18. Gender Difference in Osteopathic Medical Schools n There are substantial gender differences in terms of individual student Level 1 performance n For a few schools, the differences in both mean scores and passing rates are alarming n Gender differences can also be an opportunity for schools to improve if efforts are made to narrow the gap

19. Sector Differences on June 2002 Level 1

20. Gender and Sector Interaction on Mean Score

21. Gender and Sector Interaction on Failing Rate

22. Effects of Sector of Osteopathic Medical Schools n There is no evidence that sector significantly affects Level 1 performance n According to a study 10 years ago, sector was a powerful factor in determining osteopathic Board exam performance n It is a very positive improvement of osteopathic medical education n More formal study is needed

23. School Mean MCAT Distribution

24. School Mean MCAT of Class 2000

25. Mean MCAT by Rank

26. Mean MCAT and Mean Level 1 (r =.30, not significant)

27. Mean MCAT and Mean Passing Rate (r =.27)

28. MCAT and Level 1 Performance n The implication of correlation of school means of MCAT and a single Level 1 administration is highly limited n Individual student MCAT and Level 1 score relationship can be very different from school mean MCAT and school mean Level 1

29. Conventional Regression

30. Results of Conventional Regression n The model is significant. Both gender and sector are significant predictor. n This regression model with two variables only explained 3% of the student. performance variation. Most of the variation remains unexplained. n More information is needed to explain student performance.

31. Hierarchical Linear Regression

32. Results of HLM Analysis n Difference of students Level 1 performance across school is significant n Gender is a predictor for Level 1 scores but the difference of gender is not significant across schools n Neither Sector and school mean MCAT predict student Level 1 performance n More data will help: student MCAT, race, curriculum type, etc.

33. Thank You!