Testing Standards Chris Gruber, Barbara Plake & Wayne Camara

Session Part 1: ATP Experiences Chris Gruber ATP Liaison to the APA & Member of the ATP Test Standards Committee during the time covered.

What are “The Standards?” A book: American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association. With a history: APA, “Technical Standards…”, 1954 NEA, “Technical Standards…” 1955 APA, AERA & NCME, “Standards…”, 1966, 1974, 1985

What are the areas of concern? Part 1: Test Construction, Evaluation & Documentation—6 chapters, 113 standards Part 2: Fairness in Testing—4 chapters, 48 standards Part 3: Testing Applications—5 chapters, 93 standards

What is a standard? Standard 1.4: If a test is used in a way that has not been validated, it is incumbent on the user to justify the new use, collecting new evidence, if necessary. Comment: Professional judgment is required to evaluate the extent to which…

ATP experience on the 6 th Edition Call for Comments—1995 ATP Proposed New Standard Test users have the responsibility to respect copyright.

ATP experience on the 6 th Edition Draft 1—1996 ATP Comments cover 14 single-spaced pages. One comment reflected continuing concern that there was still no mention of Copyright

ATP experience on the 6 th Edition Draft 2—1997, ATP comments extend to 25 single spaced pages. Cover memo cites involvement of 12 member organizations—virtually all of our largest members at the time—with comments from scores of staff research and development professionals. A standard on copyright appears, rewritten, restricted in application, and qualified by “...to the extent that developers enjoin users to do so...”

ATP experience on the 6 th Edition Draft 3—1998, ATP comments extend to 26 single spaced pages. Cover memo cites involvement of 12 member organizations—virtually all of our largest members at the time—with comments from scores of staff research and development professionals. A standard on copyright? See next slide…

ATP experience on the 6 th Edition Published version—1999 Standard 11.8 Test users have the responsibility to respect copyright.

What works? Clear, respectful disagreement where appropriate and necessary Great science! Teamwork Perseverance

Part 2: Revision of Test Standards Wayne Camara Management Cmte: –David Frisbie (NCME) –Suzanne Lane (AERA) –Wayne Camara (APA) Staff support AERA Joint Cmte on the Revision of the Testing Standards: –Co-Chairs: Lauress Wise and Barbara Plake –Cmte may number members Staff support APA

Management Cmte Develop agreement (SOW) for Technical Cmte. Approve budget and expenses for the Standards and the Tech Cmte. Oversee finances and Development Fund Appoint members to Joint Cmte Determine if/when revision is required Establish timeline for revision Establish new agreement for 3 sponsoring assoc Technical Cmte Follow separate agreement developed by Management Cmte and approved by sponsoring associations Decide technical issues Determine overall purpose of standards and general guidelines on use Establish communication structures with sponsors for revision effort Develop draft for review and final document

COMMENTS RECEIVED ON STANDARDS – 12/07 There were 71 participants who provided responses to the web surveys –30 from the Standards web page –41 from the APA web page Of these 71, only 45 provided specific comments (1 or more) on the standards (the remaining 26 filled in some demographic items but not comments). –21 from the Standards web page –24 from the APA web page Note – individuals not submitting any comments are not included in remaining analyses or spreadsheets Total Number of Participants Type of ParticipantNo. of Participant Non APA Participants30 APA Participants41 Total Number of Participants71 Total Number of Participants Who Submitted Comments ParticipantsNumber Non APA Participants Who Submitted Comments21 APA Participants Who Submitted Comments24 Total Number of Participants45

From the 45 participants who submitted comments: –265 comments were provided by respondents on the Standards web page –158 comments were provided by APA respondents Total Number of Comments Submitted by Survey Group Survey GroupNo. of Comments Non APA Survey Group265 APA Survey Group158 Total423

The 45 submitted 423 comments for review The comments were submitted under three main categories: Additions Deletions Revisions The largest percent of the comments were revisions (52%) and additions (41%) Type of Comment Submitted Type of CommentNo. of Comments % of Total Comments Total Revisions22252% Total Additions17541% Total Deletions235% Total Other Submissions31% Total Number of Comments423100%

The total number of comments was also analyzed by chapter number Chapters One, Three, Five, Seven and Ten had the largest percent of comments submitted

Affiliation for APA Respondents (N=42) Text Affiliations Represented by the APA ParticipantsNo. of Participants APA Individual Member (No Affiliation)26 APA member + member of SIOP1 Committee on Aging1 Committee on Children, Youth, and Families1 Committee on Ethnic Minority Affairs1 District of Columbia Psychological Association1 Division 12 - Society of Clinical Psychology1 Division 14 - The Society for Industrial and Organizational Psychology2 Division 17 - Society of Counseling Psychology1 Division 5 - Evaluation, Measurement, and Statistics1 Individual Member1 Individual member (myself)1 member1 member of APA1 Pearson VUE psychometricians1 State Psychological Association1 Total APA Participants (One Participant is a member of Two Organizations)42

Affiliation Type By Number of Participants Affiliation TypeNo. of Participants% of Participants APA4158% Testing Organization1014% College or University811% Other710% R&D Organization34% State Agency11% Federal Agency11% School System00% Industry00% Total71100% The participants were asked to select a type of affiliation 58% of the total group were affiliated with the APA 35% of the total group were affiliated with either a Testing Organization (14%), a College or University (11%) or “other” type of affiliation (10%)

Comments and Next Steps Major Presentation at NCME – AERA 2008 Major Presentation at APA 2008 Develop SOW – Scope of Revision and Timeline Appoint Members to Tech Cmte Negotiate Timeline and Revisions to SOW with Tech Cmte Provide Updates to Sponsoring Associations

Questions?