1. The future of online diagnosis and feedback J Charles Alderson, Lancaster University Former Scientific Coordinator of the DIALANG Project www.dialang.org

2. Overview Online? Diagnosis? Feedback? DIALANG diagnosis and feedback Future possibilities? Research needed

3. Online? Stand-alone programs Interactive, via the Internet, e.g. DIALANG Virtual Learning Environments (VLEs)

4. Online? Stand-alone programs: fixed menu of tasks and feedback –Correct/ incorrect; Well done!; Try again! –Total score or level Interactive, via the Internet –Adjusts test/ exercise depending on student proficiency level (self-assessment and Vocabulary Size Placement Test) and preferred language of instructions. –Feedback categorised according to construct

5. Online? VLEs –Downloadable documents and tasks –Synchronous and asynchronous discussions –Tutor comments and feedback to forums –Private tutor comments to reflective journals –Emails from tutor to individuals and groups –Feedback in reaction to individual posts or queries –Tailored to individual? Can we learn from VLEs?

6. Diagnosis? Strengths and weaknesses, plus remediation NOT Placement: into levels NOT Achievement: what you have learned in general NOT Aptitude: phonetic coding, grammatical sensitivity, inductive language learning ability Very rarely truly diagnostic Poorly understood; never problematised; not theorised; under-researched; poor-quality tests

7. Diagnosis Based on variables known to predict development or weakness Known to be capable of remediation In-depth One-to-one exploration Feedback crucial part of diagnosis

8. Feedback Any communication or procedure to inform a learner of the accuracy of a response But also other information such as learning guidance, motivational messages, critical comparisons, and learning focus Any message or display that a computer presents to the learner after a response Aims to improve performance & learning (directly) But also to motivate, make learning more interesting, encourage different learning, raise awareness, … (Huhta, 2006)

9. Automatically Generated Feedback? Item Right/ Wrong Percent or proportion correct Some verbal description Allow multiple attempts (Try again) Immediate vs. delayed feedback Encouragement Total score/ Band level

10. Diagnosis in DIALANG Macro-level = CEF Level by Skill/Language Component (test-based and self-assessed) = Vocabulary Size Micro-level = Subskill level performance = Item level performance

11. Diagnosis and feedback in DIALANG

14. Reading sub-skills Inferencing (including lexical inferencing) Identifying main idea / Distinguishing from supporting detail Reading intensively for specific detail

15. Listening sub-skills Identifying main idea / distinguishing from suppporting detail Inferencing (including lexical inferencing) Listening intensively for specific detail

16. Writing sub-skills Knowledge of accuracy (grammar /vocabulary /spelling) Knowledge of register / appropriacy Knowledge of textual organisation (cohesion/coherence, text grammar / paragraph organisation)

17. Vocabulary sub-skills Meaning –denotation, connotation, appropriateness Semantic Relations –synonymy, antonymy, hyponymy, polysemy Word Formation –compounding, affixation Combination –collocation, idiomaticity

18. Grammar Morphology - Adjectives and Adverbs – comparison Morphology - Adjectives and Adverbs – inflection Morphology - Nouns - definite/indefinite articles Morphology - Nouns - proper/common Morphology - Numerals – inflection Morphology - Others Morphology - Pronouns – context Morphology - Verbs - active/passive voice Morphology - Verbs – inflection, tense, mood, person Syntax - Organisation/realisation of parts of speech – word order statements, questions Syntax – Punctuation Syntax - Simple Sentences vs complex sentences - coordination

19. Immediate or delayed item feedback

20. VSPT feedback

23. Explanatory feedback

25. Advisory feedback

26. How to make progress

27. Ironies DIALANG gets up to 5,000 hits per day in peak season Used for placement, NOT diagnosis Freely available No EU funding to sustain or develop Post-2004 EU-enlargement governments have not been persuaded to add their languages to the interface or as a target language

28. Problems with automatic feedback Feedback not tailored to individual performance/ profile of item responses, or background/ age/ mother tongue/ cognitive style….. Not clear that item categorisation is true and indicative of development. Nothing in CEFR on language-specific lexis or structure. No relation to theory or empirical research into second language development. Don't know how the diagnosis and feedback is received or acted upon. Yet usefulness is key to validation. Take it or leave it: no follow-up from the system.

29. Student and tutor reactions to feedback (Huhta, 2006) Overall test results the most popular part of feedback –The fact that the meaning of A1 etc was defined was appreciated Almost 60% preferred immediate review, 30% post-test review, 10% no preference –Why like: want to know immediately, remember better if see the result right away, want to learn (during test) –Why dont like: disturbs test-taking, can be depressing

30. Student and tutor reactions to feedback According to users, self-assessment is important part of (a system like) DIALANG as popular as item review (joint second) Explanatory feedback the least used / useful Extended level descriptions about CEFR less used/ useful than item review Advisory Feedback quite popular but not among all users

31. Student and tutor reactions to feedback Users clearly like to have feedback about their self- assessment –younger learners may need to be motivated first Good to have choice of two types of item review: immediate and post-test Advice appears useful for beginners in particular But challenges: –I dont bother to read the advice because I know Ill do things my way anyway –... you dont remember them any more when you study Value and meaningfulness of diagnosis? Need longitudinal studies (Long-term) effectiveness of feedback?

32. Can diagnosis and feedback be enhanced in a DIALANG-type environment? i.e. be automatically generated, yet with tailor-made help, hints, clues, feedback and explanations?

33. Ideas for improved items and feedback The following slides refer to several experimental DIALANG items available at http://www.lancs.ac.uk/fss/projects/linguistics/experimental/start.htm Readers are advised to visit that site, and take each item type before reading each of the next five slides

34. Item 2: Following instructions on a plan or map Feedback could include a map on which the route to the correct room(s) is displayed. If the learner had clicked on a room that was in a totally wrong part of the floor, the feedback could remind him / her of the point in the instructions which the learner had apparently missed (e.g. "You were asked to turn right after the lift, not to the left.").

35. Item 3: Video input Learners could be offered a chance of viewing again either the whole clip, or the section which gives the answer to the question he / she answered incorrectly. If the learner still cannot answer the question, a written transcript could be made available, possibly with key points highlighted. In the next step, access to a dictionary could be provided for the learner, and finally, the correct answer(s) could be disclosed.

36. Item 4: Drag and drop boxes, with clues Feedback can also include the number of attempts which the learner needed to arrive at the correct solution and the number of times the learner listened to the audio clip. The learner could also be shown visually how part of the construction is to be completed or where there are missing pieces in the construction.

37. Item 14: Drag and drop matching: Prompts another try Learners could be allowed access to a dictionary or to hints (e.g. questions, definitions) after a certain number of unsuccessful attempts. The number of attempts could also be reported to the learners.

38. Item 13: Self-assessment in speaking Learners could be asked to record their answer on tape or on the computer itself and then listen to their answer before proceeding to the samples. Hearing one's own response, rather than imagining what one might say, may make the comparison easier and result in more accurate self- assessment.

39. Need for research into diagnosis What to diagnose? What components of the skills are essential to further development? How does language develop across the CEFR levels?

40. Applied linguistic research Psycholinguistics of FL learning Grammatical and lexical errors, vocabulary size and depth Proficiency in other languages, language awareness Influence of mother tongue, age, learning history, aptitude, motivation, cognitive style, personality?

41. Learner corpora Specially designed, not ad hoc Learners CEFR level pre-assessed Performance on common, calibrated tasks A variety of mother tongues, ages, etc A variety of target languages SLA-type research into features predictive of CEFR level Development of diagnostic tools using such indicators

42. Source of insights Teachers Diagnostic Competence (Edelenbos and Kubanek-German, 2004) – "the ability to interpret foreign language growth in individual children" What do teachers do in class when giving feedback? What strengths and weaknesses do they concentrate on and what evidence has diagnostic potential?

43. Advantages of understanding teachers Since teachers are usually the ones who work most closely with learners, it makes sense to look at how they go about assessing their learners' strengths and weaknesses, and to explore what we can learn from them about diagnosis. ( Alderson, 2005b ) Procedures for dealing with childrens first language reading problems were evolved by observing teachers at work, challenging, discussing and consulting in an effort to link teacher and pupil behaviours with theory about the reading process. (Clay, 1979) Our understanding can be enhanced by studying how teachers diagnose and feedback to learners both in classrooms and in VLEs: an ideal data-gathering instrument.

44. Advantages of online diagnosis Provision of individualised assessment, immediate feedback, advice and even follow-up. Learner could input details of relevant personal characteristics like first language, age, learning strategies, motivation, learning history, and so on, and be presented with a diagnostic assessment tailored to such backgrounds.

45. Source of insights We need a diagnostic framework that can interface with the CEFR, or a future revised version, that can help us to explore how learners develop from one CEFR level to the next and how we can best diagnose problems in such development.

46. Advantages of online diagnosis However: Before that day comes, we will need to have developed a much better understanding of foreign language development. If we can then incorporate such understandings into assessment and make them useful to learners through the provision of meaningful and useful feedback and follow-up, then diagnosis will truly have become the interface between learning and assessment." (Alderson, 2005a:268).

47. What do I teach? What do I concentrate on and why? How do I give feedback and what sort? What do I diagnose? What can I diagnose? Can I give more meaningful feedback? Can this be incorporated into online diagnosis and feedback? Can learners choose their own feedback? Can they explore what is available?

48. What do I teach? Which language aspects do I notice/ comment on, at which level? Which language aspects are most amenable/ resistant to feedback? What sort of feedback do my students a) demand? b) pay attention to? c) find ultimately helpful?

49. Diagnosis Based on variables known to predict development or weakness Known to be capable of remediation In-depth One-to-one exploration Feedback crucial part of diagnosis

50. Embedding Bennett foresees three phases of innovation in e- learning: 1.Doing in e-form what we can already do in other media. 2.Enhancing e-learning through experimentation and technological advances 3.Seamless embedding of assessment within learning activities (Bennett, 1998) Then, diagnostic assessment procedures would be indistinguishable from learning procedures, and we would indeed have developed tests with learning validity. (Alderson, 2005a)

51. The future of online diagnosis and feedback ????

52. References Alderson, J. C. (2005a) Diagnosing foreign language proficiency: The interface between learning and assessment. London: Continuum. Alderson, J. C. (2005b) The challenge of (diagnostic) testing: do we know what we are measuring? Plenary presentation at LTRC, Ottawa, July Bennett, R.E. (1998) Reinventing assessment: speculations on the future of large-scale educational testing. Princeton, New Jersey: Educational Testing Service. Clay, M. M. (1979) The early detection of reading difficulties: A diagnostic survey with recovery procedures. Auckland, N.Z.: Heinemann. Edelenbos, P. and Kubanek-German, A. (2004) Teacher assessment: the concept of 'diagnostic competence'. Language Testing, 21(3), 259-283. Huhta, A. (2006) Users reactions to innovative computerised feedback – the case of DIALANG. Paper presented at EALTA Conference, Krakow, May.