1. EAP and Student Placement Karen Borglum, Ed.D. Assistant Vice President Curriculum and Articulation

2. Walmart “GO” Grant Spring of 2010: EAP program receives a two-year grant from Walmart Outcomes of grant Develop an EAP/Gen Ed curriculum designed to increase student academic success Establish a common course repository Implement an aligned curriculum during two consecutive semesters Engage faculty in data and curriculum evaluation Engage targeted students in community-building activities Disseminate the curriculum model

3. Next Steps Dollars were then used to address placement issues A UCF Statistician was hired to review data and make suggestions

4. Present Placement Process Four tests given: essay writing (graded by a faculty member) worth 50%; reading test, grammar test, and usage/mechanics test graded by Accuplacer system for the remaining 50%

5. Goals of Statistical Analysis To determine if faculty assessments of student essays were reliable across graders. To determine which of the computerized essay evaluation software packages, most closely agree with assessment of three panel graders

6. Methodology Sixty students were given entrance essays and were graded by 11 faculty members collegewide. The three faculty scores were averaged to produce a composite score for comparison in Accuplacer (present system) and the Compass (new system)

7. Findings Inter-rater reliability when using our faculty to grade essays is.66 (n=60 essays) Spearman’s Rho coefficients comparing faculty and computer scoring: Accuplacer-.747 Compass-.851 There is substantial variability between faculty graders (especially across campuses), but the average scores produced by groups of graders match up with computerized grading

8. Meetings Faculty and Deans met over the course of about a year to discuss results, review data, run tests, and determine next steps

9. Meeting History May 3, 2012: Discussion of history of EAP, review of statistical analysis, preliminary discussion of systems. Need to discuss Compass testing with Frank Potter from Compass Need only 3 tests for placement instead of 4: Essay, Reading, and Listening May 24, 2012: Sent e-mail to Campus Presidents explaining EAP Placement Issue June 4, 2012: Met with Frank Potter to look at possible cut scores Indian River, Jacksonville, Miami Dade and Palm Beach use Compass

10. Meeting History Continued September 10, 2012: Discussion of how to establish cut scores in Compass 300 units were provided to run a Beta test in reading similar to writing November 12, 2012: Reviewed data from Beta tests Determined that we needed linear regression done from Compass February 6, 2013: James May reviewed his work with Compass Statistician to determine cut-scores

11. Cut-Score Decision Two issues emerged from Compass data: 1. Restriction of range issue: not a true population since our students can only place with a score of 66. 2. Equitable forms issue: the line cut scores are an average of 4 tests, so the placement is obscured

12. Final Decision Take an average of reading and essay scores to place students into reading, grammar and writing Collect data on the listening tests to make a determination on a separate placement for the speaking course

13. Cut Scores Course NumberReading ScoreWriting Score EAP 0281 (combined skills) 15-291-2 EAP 0320 (reading)30-39 EAP 0340 (writing)3 EAP 0420 (reading)40-494 EAP 1520 (reading)50-69 EAP 1540 (writing)5-6 EAP 1620 (reading)70-94 EAP 1640 (writing)7-9 Test out of EAP95-10010-11

14. Cost Previous year: # of students tested – 1,100 # of units used – 3,300 Price per unit - $ 1.75 (Accuplacer ESL, AKA LOEP) Total cost - $ 5,775 Projection (with Listening test): # of students to be tested for 1 year – 1,200 # of units needed – 8,320* Price per unit - $ 1.50 (ACT Compass ESL) Total cost – 12,480 Projection (without Listening test): # of students to be tested for 1 year – 1,200 # of units needed – 7,120* Price per unit - $ 1.50 (ACT Compass ESL) Total cost – 10,680 *Based on projection that 1/3 of our students are retaking the test – there is a slightly lower cost for a retake for the listening and reading sections. E-Write uses 3.5 units independently of whether it is a first time or retake.

15. Cost Analysis The cost of Compass is more than Accuplacer; however, it is a better placement tool. There will be a more accurate and consistent placement for the essay portion of the test. The Director of Standardized Testing is seeking approval to implement a retake fee of $10 or $15 to offset cost. The retake fee is NOT being explored because of the possible new placement test; it was already in the works with the current system. We already have a retake fee for students retaking the PERT.

16. Additional Information Valencia’s Continuing Ed Division is also looking to use Compass for their placement work (we can get a pricing discount) Faculty and Deans have voted to approve cut scores Deans have discussed results with their Campus Presidents The Director of Standardized Testing is supportive of this switch Agreement to review placement information January 2014 to determine if placement needs to be tweaked There are mechanisms in place if students/faculty feel that a student is improperly placed upon initial testing

17. Voting Supportive of new placement structure Non-supportive of new placement structure