1. AP Physics Comparability Study Conducted periodically to compare AP and college students –College students take part of the AP exam, and it counts toward their course grade –Instructors grade the exam by their own standards –Instructors send ETS both exam grade and final course grade for students, as well as comments on student motivation and match of study test to their course content

2. Helps set cut points for AP grades –College students’ responses scored as AP students’ would be –ETS statisticians analyze the data, to see how college grades of A, B etc, compare to AP grades of 5, 4, etc. –AP grades are adjusted as appropriate, based on data and judgment of Chief Reader. Behind that judgment are curriculum surveys used to set course outline and exam specs, input of committee as exam questions are developed, and information about student performance from the Reading leadership. –Goal is that 4/5 cut is at college mid-A’s, 3/4 cut at mid- B’s etc. Comparability Study

3. Provides some information to colleges that relates their students’ performance to the AP students and other participating colleges –College instructors receive data that allows them to compare their students as a group to the AP population and the total group of college students participating. Comparability Study

4. The Study Evaluated both the B and C exams Used multiple-choice sections of 2006 exams Four study tests –Questions in B divided roughly in half Since B exam covers the whole survey course, questions divided into “typical” semesters --- mechanics, fluids, thermal E&M, waves & optics, atomic & nuclear –C Mechanics and C E&M already separate exams, and were given in their entirety

5. Invitations sent to colleges and universities receiving 20 or more AP Physics B 2006 exam grades –Letter sent to department chairs, college AP Physics Readers and present/past Development Committee members –Instructors gave tests at appropriate time near the end of their semester courses The Study

6. The Participants Arkansas State University New Jersey Institute of Technology University of Arkansas Oklahoma State University Baylor University University of Rhode Island University of British ColumbiaUniversity of Tennessee California State University Northridge Texas Christian University The College of New JerseyTexas Tech University Grove City College Virginia Polytechnic Institute and State University James Madison UniversityWheaton College University of Mississippi

7. Not all colleges gave every study test Physics B first semester had 11 colleges, Physics B second semester had 4 (all these also gave first semester) Physics C Mechanics and E&M each had 8 Desire 20 colleges minimum – since study was added late in summer, many colleges couldn’t work it into their schedules The Participants

8. 2007 exam analysis done as usual, using standard equating performed each year, yielding cut scores based on carryover of same score scale Data from colleges analyzed – through method similar to regular equating, a set of “new” cut scores for 2006 generated based on college standards The Analysis

9. Chief Reader, ETS staff, and CB staff review individual college data to identify any “suspect” data, e.g. –Students not motivated –Test content not good match to course content –Low correlation between student course grade and study test grade Again using equating, equivalent adjusted cut scores for 2007 exams are generated The Analysis

10. The Data Physics B Number correct on 2006 Multiple-choice (68 questions) ABCD College Average32241914 5/44/33/22/1 Average 2006 AP student MC score at cut points 40301913 Did not have data on both parts for many students. College scores on semester tests converted to scores on complete exam, taking into account difference in difficulty of the two parts. 2006 AP students getting 5 or 4 score higher than college average.

11. Physics B Cut points for 2007 exam (180 points total) 5/44/33/22/1 AP 2007 unadjusted cut106825337 AP 2007 study cut86614125 AP final 2007 cut106825333 Cut scores suggested by study lower than present standards – but not prudent to make large adjustments based on the small number of colleges participating. Large differences warrant a larger study that also includes free-response questions.

12. Physics C: Mechanics Number correct on 2006 Multiple-choice (35 questions) ABCD College Average211595 5/44/33/22/1 Average 2006 AP student MC score at cut points 2014106 College and AP student performance fairly similar

13. Physics C: Mechanics Cut points for 2007 exam (90 points total) 5/44/33/22/1 AP 2007 regular cut59463525 AP 2007 study cut6047/483223 AP final 2007 cut59463525 Since the sample of colleges is small there is no compelling reason to adjust cut points

14. Physics C: Electricity & Magnetism Number correct on 2006 Multiple-choice (35 questions) ABCD College Average181174 5/44/33/22/1 Average 2006 AP student MC score at cut points 161184 Again, college and AP student performance fairly similar

15. Physics C: Electricity & Magnetism Cut points for 2007 exam (90 points total) 5/44/33/22/1 AP 2007 regular cut49342716 AP 2007 study cut543422-2314 AP final 2007 cut51342716 Greater discrepancies at 5/4 and 3/2 cuts than for Mechanics Small sample of colleges warrants caution, so 5/4 cut was adjusted up only slightly Less prudent to ease standards based on small sample, so 3/2 cut not adjusted

16. Physics B standards might be high compared to colleges, but further evidence is needed before making major adjustments. No evidence that Physics C standards have been far from college standards. A larger study on all exams is needed to further confirm results – likely to be conducted when redesign is implemented Conclusions