1. You got WHAT on that test? Using SAS PROC LOGISTIC and ODS to identify ethnic group Differential Item Functioning (DIF) in professional certification exam questions Steve Grilli Life Office Management Association

2. PRESENTATION OUTLINE Introduce LOMA Intro to Educational Stats LOMA’s SAS Item Analysis Program “DIF” Defined Logistic Regression LOMA’s SAS DIF Identification Program Conclusions

3. About LOMA   Founded in 1924... an international association of insurance and financial services companies   Located in Atlanta, GA... with local partners around the world   Purpose: to facilitate information sharing, improve company operations and management, provide industry-specific employee development

4. LOMA By the Numbers  80+ years of experience  1,200+ members in 80 countries  13 professional education programs  Courses available in 7 languages  100,000+ annual examination enrollments  More than 10,000 attendees to conferences & meetings each year  1,200 individuals serve on more than 50 LOMA committees

5. Educational Statistics: “Classical” Item Analysis vs. IRT Item Response Theory – three parameter Rasch model with discrimination parameter a, difficulty b, and “pseudo-guessing” parameter c. Used in Computer Adaptive Testing (CAT)

6. Classical Item Analysis  Biserial correlation between performance on a dicotomous test item (X=1 if student got it correct; 0 otherwise), and a continuous variable – score on the entire exam.

7. ITEM ANALYSIS – SAS CODE  /* CALCULATE BISERIAL CORRELATIONS FOR AN ARREA OF EXAM QUESTIIONS */  DATA NEXT;  SET PXDAT;  SET ADD;  SET YI;  ARRAY P PX1-PX&R;  ARRAY ZCAL 3 Z1-Z&R;  ARRAY BISA 3 BISA1-BISA&R;  ARRAY BIS 3 BIS1-BIS&R;  ARRAY YI YIMEAN1-YIMEAN&R;  DO OVER P;  ZCAL=PROBIT(P);  BISA=.39894/EXP((ZCAL*ZCAL)/2);  END;  DO OVER BIS;  BIS=((YI-YMEAN)/YSTD)*(P/BISA);  END;  PROC TRANSPOSE DATA=NEXT OUT=BIS PREFIX=BIS;  VAR BIS1-BIS&R;

8. ITEM ANALYSIS – SAS OUTPUT  ITEM ANALYSIS  PAPER EXAMS  COURSE 290  FORM 1265  04M  COURSE: 290  ITEM: 1 1 2* 3 4 5 6 OMIT  1,180 UPPER 3RD 0.6 99.2 0.1 0.2 0.0 0.0 0.0  1,181 MIDDLE 3RD 2.6 96.3 0.2 0.9 0.0 0.0 0.0  1,180 LOWER 3RD 14.4 72.7 4.1 8.7 0.0 0.0 0.1  3,541 TOTAL 5.9 89.4 1.4 3.3 0.0 0.0 0.0  BISERIAL CORRELATION: 0.855 CONFIDENCE: 100.0  COURSE: 290  ITEM: 2 1 2 3 4* 5 6 OMIT   1,180 UPPER 3RD 1.5 0.8 0.2 97.5 0.0 0.0 0.0  1,181 MIDDLE 3RD 7.5 3.6 0.2 88.8 0.0 0.0 0.0  1,180 LOWER 3RD 21.1 9.1 3.8 65.8 0.0 0.0 0.2  3,541 TOTAL 10.0 4.5 1.4 84.1 0.0 0.0 0.1  BISERIAL CORRELATION: 0.625 CONFIDENCE: 100.0

9. ITEM ANALYSIS – SAS OUTPUT  PAPER ITEM ANALYSIS EXCEPTION REPORT  COURSE 290  FORM 1265  04M  ERROR CODES  E1: BISERIAL CORRELATION LESS THAN.200  E2: FEWER THAN 50% OF THE UPPER GROUP CHOSE RIGHT ANSWER  E3: 25% OR MORE OF UPPER GROUP CHOSE A SPECIFIC DISTRACTOR  E4: DISCRIMINATION CONFIDENCE LESS THAN 90% (50 OR MORE STUDENTS)  (NOTE PROBLEM ANSWERS IN PARENTHESIS FOR E2 AND E3)  ITEM PROBLEMS  53 E1 E4  71 E3(1)

10. DIFFERENTIAL ITEM FUNCTIONING (DIF) “ an item displays DIF if examinees from different groups have differing probabilities or likelihoods of success on the item after conditioning or matching on the ability the item is intended to measure” -- NCME “ an item displays DIF if examinees from different groups have differing probabilities or likelihoods of success on the item after conditioning or matching on the ability the item is intended to measure” -- NCME DIF is a necessary but not a sufficient condition for item bias DIF is a necessary but not a sufficient condition for item bias Item bias exists when members of one group are less likely to answer an item correctly because of some aspect of the item or the testing situation that in not relevant to the purpose of the testing. Item bias exists when members of one group are less likely to answer an item correctly because of some aspect of the item or the testing situation that in not relevant to the purpose of the testing.

11. TYPES OF DIF  Two types of DIF: Uniform and Non-Uniform.  Uniform DIF is when one group’s advantage is roughly constant across the ability scale.  Non-Uniform DIF occurs when the advantage varies at different ability levels; i.e., ability and group membership interact

12. DIF DETECTION  Experts recommend the use of logistic regression to detect DIF  LOMA chose this method for its conceptual clarity, its ability to detect non- uniform DIF, and the ease with which existing SAS software could be employed in its detection

13. LOGISTIC REGRESSION

14. LOMA DIF LOGISTIC MODEL Theta is ability measure (score on the exam) E is education, 1 if BA or higher; 0 otherwise G is group membership – generally US vs China Theta x G is the interaction term to test for non-uniform DIF G x E is interaction of group and education

15. DIF LOGISTIC MODEL: SAS CODE  PROC LOGISTIC DESCENDING ;  ODS OUTPUT TypeIII=MODEL&I GlobalTests=GT&I;  CLASS EDCODE (PARAM=REF REF='A')  GRP (PARAM=REF REF='US');  MODEL RES&I=GRADE EDCODE GRP GRP*GRADE GRP*EDCODE/  SELECTION=STEPWISE INCLUDE=1 SLE=.01 SLS=.01 HIER=MULTIPLE;

16. SAS DIF PROGRAM: OUTPUT  DIFFERENTIAL ITEM FUNCTIONING REPORT  COURSE 290 04M  REFERENCE GROUP: UNITED STATES  FOCAL GROUP: CHINA  ITEM MODEL PREDICTORS DIF TYPE LR CHI SQ CONFIDENCE  1 SCORE 57.57 100.00%   2 SCORE 114.53 100.00%   3 SCORE 22.40 100.00%  4 SCORE, GROUP UNIFORM 77.33 100.00%  5 SCORE, GROUP UNIFORM 166.31 100.00%  6 SCORE, GROUP UNIFORM 79.02 100.00%  7 SCORE, GROUP UNIFORM 143.04 100.00%  8 SCORE 216.45 100.00%  9 SCORE, GROUP, GROUP*SCORE NON-UNIFORM 152.34 100.00%

17. SAS DIF PROGRAM: FILE OUTPUT  [290,M04,17]  1=US v CH -- S -- NONE  [290,M04,18]  1=US v CH -- S -- NONE  [290,M04,19]  1=US v CH -- S, G, G*S -- NON-U  [290,M04,20]  1=US v CH -- S, ED, G -- U (2)

18. Item: S0420-E (QID=9780)CR: 4DiffiEst: 81Codes: mc; 0, r TextRef: O&S, c. 11, pp. 259-260Mandatory: Y Most of an insurer's customers can be characterized as either external or internal. However, some customers have characteristics of both internal and external customers. One example of an insurance customer who has characteristics of both internal and external customers is (1) a third-party administrator (2)a policy beneficiary (3)an individual policyowner (4)a general agent ItemRegionGroupN1234*56OmitCRR,Bis.,Conf.,E,DIF S0420 05/04 All Upper 3rd1,18010.91.11.086.90.0 69.7, 0.466, 100.0, DIF: US v CH -- S, ED, G -- U (2) Middle 3rd1,18122.32.74.170.90.0 0.1 Lower 3rd1,18024.711.212.751.20.0 0.2 Total3,54119.35.05.969.70.0 0.1 S0420 05/04 US/Can Upper 3rd20623.81.52.971.80.0 61.9, 0.264, 100.0, E4 Middle 3rd20529.83.94.961.50.0 Lower 3rd20625.210.711.252.40.0 0.5 Total61726.35.36.361.90.0 0.2 S0420 05/04 Int'l Upper 3rd1,0207.01.10.691.40.0 72.7, 0.527, 100.0 Middle 3rd1,01919.42.63.274.70.0 0.1 Lower 3rd1,02024.510.712.552.20.0 0.1 Total3,05917.04.85.572.70.0 0.1 pp. 288, 289

19. CONCLUSIONS Need to monitor DIF due to increasing globalization SAS PROC LOGISTIC and ODS feature simple and effective means of DIF detection