1. Eng I Fall 2010: Diagnostics & Results David Minugh English Department, SU May, 2011

2. Diagnostic Tests: Key aspects Intake hurdle or not: a basis for admission? If students are already accepted, is the test mandatory or not? Costs (no “not” alternative here!) Information to students or a placement test? Computerized or not? Reusable or not?

3. Intake Hurdle: A basis for admissions? If we limit intake to those with scores better than X, – We need a new test every time (or a large pool of items) – It needs to be legally acceptable (including student IDs) – What about our intake profile (broader recruitment?) – Teacher candidates will have to be treated separately – We need to have good validity – What about motivation and test-taking skills? – Time: a summer test?! – Retakes for those who were sick, abroad, etc.? – Is it relevant if we can’t afford to turn away students? – Current status: offered to those who actually register

4. Mandatory or not? Our paper-based test (up to 2006) was “mandatory,” – no sanctions for those who didn’t take it – identical from year to year (over a 30-year span) – make-ups were offered on an ad-hoc basis – dropped to save marking and room costs Our most recent (voluntary) net-based diagnostic test was taken by 139 of about 300 students – No way of checking which students took it – The process took several weeks The new, scheduled diagnostic test (Sept. 3) reached 300 of about 330 students – On their schedule (three 100-student waves), Fri 9-11 am, 11-1, 1-3 pm

5. Information or Placement Test? Any diagnostic test contains potential information for students – however, they may not know what it means – we may not know what it means, either – motivation and commitment are not easily and cheaply measured It could be used for group placements – SU groups are not differentiated: 10 regular + 2 future- teacher groups – Students choose groups by schedule or teacher or books; 11 different combinations of teachers Group placement implies non-recyclable tests

6. Costs Test-making: – One-time development cost for our test at SU – Higher for Q pools, much higher for new tests each semester – Oxford’s test (A1-C2 rating) runs about 20 cr/student – DiaLang (an EU project) is free, but now unsupported Room costs: regular exam halls v ones with computers – At SU the cost is the same (!) – 15-20 TKR, c. 3000 cr/hr for a 140-person hall + 5 computer- literate proctors, 3 waves – IT support costs and level of support? Marking: – Free for computerized tests – Student/admin markers for paper-based version – Admin time is at a premium when the semester starts

7. Computerized or Not? Volume: some 300 new students per semester Advantages: – no mountain of papers – instant feedback – easy administration of results and student contacts – students get into the VLE system quickly – error analysis (practical or research) Disadvantages: – test limited to electronic marking and relatively simplistic questions; very limited productive component – student unfamiliarity with the system – costs to install and run

8. Starting Points for SU 285 new English I students took our net-based diagnostic test on Sept. 3, 2010, at the start of the Fall Semester 2010 – A few others from other courses were excluded from this discussion The test was strictly proficiency-oriented: Grammar (60), vocabulary (60), collocations (20), phonemes (10), word stress (10) The maximum score was 160; actual scores ranged from 39 to 152, a bell curve peaking in the 97-128 range (see next slide) Similar results were obtained in the Spring 2011 diagnostic test Those who had 80 or less received an individual e-mail about this, but were not called to mandatory counseling For the results shown here, all tests have been offered twice, and all papers graded, i.e. a full semester had been completed when the results were calculated (March 2011)

9. Eng I Diagnostic Test HT10

10. Credits Obtained (max 30) v. Diagnostic Results

11. Credits Achieved (max 30) and Diagnostic Results

13. Percentage Course Dropouts per Diagnostic Results (n = 25)

14. Course Drop-outs/Finishers per Diagnostic Results

15. Course Completers/Partials/ Drop-outs per Diagnostic Results

16. Percentage Course Completers/Partials/ Drop-outs per Diagnostic Results

17. Literature (12 cr): Completers/Partials/ Drop-outs per Diagnostic Results

18. % Literature (12 cr): Completers/Partials/ Drop-outs per Diagnostic Results

19. Linguistic Analysis (6cr): Completers/Partials/ Drop-outs per Diagnostic Results

20. % Linguistic Analysis (6 cr): Completers/Partials/ Drop- outs per Diagnostic Results

21. Linguistics Test (6 cr): Completers/Partials/ Drop-outs per Diagnostic Results

22. % Linguistics Test (6 cr): Completers/Partials/ Drop-outs per Diagnostic Results

23. Eng I Final Grade v Diagnostic Score

24. Tentative Conclusions Proficiency alone will not guarantee success Drop-outs (25  10%): – Half had scored < 80 on the diagnostic test – Almost 30% had scored above 100 – Another 18 took 0 credits (40% of them scored < 80) Results: – No student scoring under 72 finished within one semester – The first linguistic test (lectures based on an introductory linguistics textbook (Yule, The Study of Language) is the biggest stumbling block – Quite a few lack only one or two components – Very few A grades were granted, and no Es (this may change as further make-up papers and spring retakes come in) – Both the literary and the linguistic course seem accessible even to quite weak students (60% in the 65-80 range passed the literary and/or linguistic course) – The diagnostic test is not a strong predictor for most students, but is relatively reliable at the lower and upper ends of the scale – The tipping point (50% chance of succeeding within a semester) appears to be around 100 on the diagnostic test