1. The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee

2. Outline The design and implementation of the questionnaire Some overall themes Comparing results by regions Implications

3. The timetable First meeting: Preliminary design: Web-based questionnaire (roll-out date): Data collection: Data preparation (for analysis): Analysis + preparation for conference: 24 – 25 January 2004 1 February 2004 1 March 2004 1 March – 15 April 2004 18 – 21 April 2004 23 April - 12 May 2004

6. Part 2: Questions for teacher trainers/educators Part 3: Questions for people who work, full time or part time, for language testing/examination boards or organisations that design school-external, regional or nationwide tests/examinations

7. Summary of responses 914 respondents to at least one part of the questionnaire –Part 1: 741 –Part 2: 709 –Part 3: 689 501 respondents have provided contact details

8. Summary of responses Point to Note –Respondents did not necessarily answer every item in each section for which they submitted data. Responses for parts 2 & 3 are particularly low –Part 1: 522 – 614 responses per item –Part 2: 181 – 212 responses per item –Part 3: 198 – 228 responses per item

9. Where do the respondents work?

10. ’Teachers only’: degree of formal training in assessment 1= none 2= short 3= medium 4= long

11. Correlation between assessment-related activities teachers are engaged in and absence of professional training 1Classroom-focussed assessment

12. Teachers’ perceived needs: do you need training in this area? 1Classroom-focussed assessment

13. Correlation between assessment-related activities teachers are engaged in and absence of professional training 2Purposes of assessment

14. Teachers’ perceived needs: do you need training in this area? 2Purposes of assessment

15. Correlation between assessment-related activities teachers are engaged in and absence of professional training 3Content and concepts

16. Teachers’ perceived needs: do you need training in this area? 3Content and concepts

17. Correlation between assessment-related activities teachers are engaged in and absence of professional training 4External tests and exams

18. Teachers’ perceived needs : do you need training in this area? 4External tests and exams

19. Tentative conclusions on teachers’ needs Teachers generally perceive a need for training across the board Areas of particular need: –ELP/other portfolios, preparing own tests, peer/self assessment, interpreting results and giving feedback –To award grades, to find out what needs to be taught, placing students, awarding certificates –Aspects of culture, integrated skills, establishing reliability and validity, statistics –Reviewing and writing items, statistics, defining criteria, rating, interviewing

20. Teacher trainers: what they train 1Classroom-based

21. Correlation between assessment-related activities teacher trainers give training in, and absence of their own professional training 1Classroom-based

22. Teacher trainers: what they train 2Purposes

23. Correlation between assessment-related activities teacher trainers give training in and absence of their own professional training 2Purposes

24. Teacher trainers: what they train 3Contents and concepts

25. Correlation between assessment-related activities teacher trainers give training in and absence of their own professional training 3Contents and concepts

26. Tentative conclusions from teacher trainer data Teacher trainers perceive their own needs similarly to teachers Much of what they train in is in areas in which they have no professional training themselves (generally 30%-50%) Areas of greatest need, where teacher training is not given, or given by people with little professional training, largely coincide with the areas of greatest need in teachers: – ELP/other portfolios, preparing own tests, peer/self assessment, interpreting results and giving feedback + informal continuous assessment –To award grades, to find out what needs to be taught, placement, awarding certificates –Aspects of culture, integrated skills, establishing reliability and validity, statistics + productive skills

27. Correlation between activities experts are engaged in and absence of professional training 1Item writing and rating

28. Experts’ perceived needs : do you need training in this area? 1Item writing and rating

29. Correlation between which activities experts are engaged in and absence of professional training. 2 Developing tests and assessment systems

30. Experts’ perceived needs : do you need training in this area? 2 Developing tests and assessment systems

31. Correlation between which activities experts are engaged in and absence of professional training 3CEF related

32. Experts’ perceived needs : do you need training in this area? 3CEF related

33. Tentative conclusions on experts’ needs Experts generally perceive need for training across the board Areas of particular need: –Item writing, reviewing items or tests, making decisions about composition of tests –Statistical analysis, setting pass marks, creating/maintaining item banks, doing validation research –Relating existing tests to the CEF, using CEF as basis for new test systems

34. Regions compared 1. Southern Europe –Portugal, Spain, Andorra, Italy, Malta, Greece, Turkey 2. Western Europe –Ireland, UK, France, Belgium, Netherlands, Germany, Austria, Switzerland 3. Northern Europe –Iceland, Norway, Sweden, Denmark, Finland 4. Eastern Europe –Estonia, Latvia, Lithuania, Poland, Russia, Czech Rep., Slovakia, Hungary, Bulgaria, Romania, Ukraine, Slovenia, Croatia, Serbia and Montenegro, Macedonia, Azerbaijan 5. non-European countries Questionnaire 1 / Teachers

35. Teachers’ previous training 2 = A little (1-2 days)

36. Teachers’ need for training - average number of activities, etc where need for assessment training was reported (out of max 25)

37. Teachers’ reported need for advanced training 1 (activities out of max 25)

38. Teachers’ reported need for advanced training 2 (activities out of max 25)

39. Statistically significant differences Teachers’ previous training no differences between regions Teachers’ need for advanced training: Western Europe < Eastern Europe Western Europe < Southern Europe Teachers’ need for basic training: no statistically significant differences small n size made it impossible to calculate differences between individual countries

40. Top 5 needs (teachers / regions) 1st2nd3rd4th5th South Portfolio 73% Interpretation 65% Statistics ext. 64% Continuo us 62% Criteria 61% West Reliability 55% Validity 54% Statistics 50% Portfolio 45% Self / peer 45% North Portfolio 70% Self / peer 63% Reliability 62% Validity 62% Statistics 61% East Portfolio 72% Reliability 72% Statistics 71% Validity 71% Interpretation 65%

41. Implications for training event(s) Common training events could be designed for both teachers and teacher trainers These should aim to cover the needs revealed here as pressing; however, it is likely that more ’core’ needs may exist for grass roots teachers (further investigation needed) A separate, specialist, training event would be necessary to cater for experts’ needs Regional events may be adapted to reflect different priorities

42. And finally.. Beneficial to belong to EALTA?