1. 1 TESL 4340 -- Evaluating CALL Packages:Curriculum/Pedagogic al/Linguistics Dr. Henry Tao GUO 郭涛 Email: henryguo@uic.edu.hk henryguo@u Office: B 418

2. 2 Outline Part one: Language program evaluation (Lynch 1996: 1-9) Part two: Evaluating CALL( Chapelle 2001:44-94)

3. 3 Language program evaluation Definition Critical issues Context-adaptive model (CAM)

4. 4 Definition: evaluation Evaluation here is defined as the systematic attempt to gather information in order to make judgments or decisions. As such, evaluative information can be both qualitative and quantitative in form and can be gathered through different methods such as observation or the administration of pencil- and-paper tests.

5. 5 CAM Audience and goals Context inventory Preliminary thematic framework Data collection design/system Data collection and analysis Evaluation report (Lynch 1990a)

6. 6

7. 7 Audience and goals Audience Who is requesting the evaluation? Who will be affected by the evaluation? Stakeholders (or clients) & Peripheral audience Goals Why is the evaluation being conducted? What information is being requested and why? The goals of evaluation may be determined by audience

8. 8 The role of evaluator is determined by the particular audience and their goals  Someone providing consultation  An expert standing in judgment  A collaborator in program development  A decision-making facilitator Hiring external evaluators for greater objectivity or internal evaluators for improving the curriculum?

9. 9 Context inventory 1. Availability of a comparison group 2. Availability of reliable and valid measures of language skills 3. Availability of various types of evaluation expertise 4.Timing of the evaluation 5.The selection process for admitting students into the program

10. 10 Critical issues Why CALL programs should concern themselves with program evaluation? Internal quest for program improvement Externally imposed requirement in order to justify program funding

11. 11 6. Characteristics of the program students 7. Characteristics of the program staff 8. Size and intensity of the program 9. Instructional materials and resources 10. Perspective and purpose of the program 11. Social and political climate

12. 12 Preliminary thematic framework Where should the evaluator being? What aspects of the program should evaluator investigate in detail?

13. 13 A preliminary thematic framework provides a conceptualization of the program in terms of the salient issues and themes that have emerged from the determination of audience and goals and elaboration of the context inventory. Articulating this framework provides the evaluator with a focus that will guide the collection and analysis of evaluation data.

14. 14 An example of PTF developed for EST reading program 1. Effects of focusing instruction on reading only 2. Effects of focusing instruction on reading skills and strategies 3. Effects of using authentic reading texts 4. Feasibility of using Spanish versus English for instruction

15. 15 5. Availability of classrooms 6. Feasibility of using a ‘modified adjunct model’ approach 7. Feasibility and effects of conducting classroom-centered research 8. Levels of student proficiency in English upon entering the program (Adapted from Lynch 1990a)

16. 16 Data collection design/system What type of data need to be gathered- quantitative, qualitative or both? What will be the best method for gathering the data? The context inventory can be used to determine the feasibility of certain types of data collection design.

17. 17

18. 18 Data collection and analysis Appropriate conduct of the data-gathering procedures and the interpretation of the results. In the case of quantitative designs, have the assumptions of the design and statistical models been met? In the case of qualitative design, have the procedures for data gathering been portrayed accurately, and have alternative interpretations of the data been pursued?

19. 19 Evaluation report The evaluator must be sensitive to the audience and goals of the evaluation The social and political climate dimension of the context inventory needs to be considered carefully at this stage

20. 20 The critical issue is how to communicate the findings of the evaluation honestly and successfully. Prepare multiple reports or express the findings in different ways, depending on the intended audiences.

21. 21 Evaluating CALL The problem of instructed SLA Principles for CALL evaluation: three needs Judgmental evaluation of CALL Empirical evaluation of CALL

22. 22 The problem of instructed SLA Cognitive conditions for SLA Social-affective conditions for SLA Other factors

23. 23 Cognitive conditions for SLA  Choose a range of target structures  Choose tasks which meet the utility condition  Select and sequence tasks to achieve balanced goal development  Maximize the chances f focus on form through attentional manipulation  Use cycles of accountability

24. 24 Social-affective conditions for SLA  Willingness to communicate (WTC) The desire to communicate with a particular person Communicative self confidence at that particular moment Interpersonal motivation Intergroup motivation

25. 25 Self confidence Intergroup attitudes(e.g., integrativeness) Social situation(i.e.,features of context affecting communication) Communicative competence Intergroup climate personality

26. 26 Other factors  Individual differences in cognitive characteristics of learners due to, for example, age or cognitive style.  Learning situation such as the effects of task choice on teachers and learners and others who may be involved with the task  Practical factors such as the available resources

27. 27 Principles for CALL evaluation: three needs First, evaluation criteria should incorporate findings and theory-based speculation about ideal conditions for SLA such as those outlined above. Second, criteria should be accompanied by guidance as to how they should be used; in other words, a theory of evaluation needs to be articulated. Third, both criteria and theory need to apply not only to software, but also to the task that the teacher plans and that the learner carries out.

28. 28 Evaluation as an argument

29. 29 Judgmental and empirical analyses

30. 30 Criteria from theory and research on SLA

31. 31 Criteria applied based on task purpose These criteria for CALL appropriateness need to be applied in view of the purpose of a CALL task. Skehan's discussion of performance goals for task, including learners‘ fluency,accuracy, and complexity, might be augmented by comprehension goals and goals concerning pragmatic competence (Kapser,1997).

32. 32 Moreover, tasks may have different purposes at various stages of instruction (Doughty & Williams,1998). Whatever the goal of the CALL task, however, evaluation of the task requires that it have a stated purpose.

33. 33 The centrality of language learning Even though the importance of each of the six criteria may vary depending on the purpose of the tasks, language learning potential should be considered the most critical for CALL activities

34. 34 Judgemental evaluation of CALL

35. 35

36. 36 Cases(Chapelle 58-67) Computer-assisted classroom discussion Microworld Text analysis Storyboard Concordancing

37. 37 Summary

38. 38 Empirical evaluation of CALL Evidence for CALL qualities

39. 39 ‘Students are often doing soimthing very different from what [ language teachers ] assume they are doing’ ( Hosenfeld,1976:123). In other words, it is necessary to identify the observable data that provide evidence of CALL qualities.

40. 40 Evidence for CALL qualities

41. 41

42. 42 Language learning potential Focus on form Modified interaction Modified output Research methods

43. 43 Learner fit Level of linguistic difficulty Individual difference Research methods

44. 44 Meaning focus Effects of meaning-based instruction Assessing engagement with meaning

45. 45 Authenticity Comparing CALL with non-CALL activities Research methods

46. 46 Positive impact

47. 47

48. 48 Practicality Given the role of resources for the success of CALL, some formal mechanism needs to be in place to monitor adequacy, and an argument about CALL appropriateness should include a statement about sufficiency of resources.

49. 49 Conclusion Evidence concerning the six ideal qualities of a particular CALL task needs to be combined to form an evaluative argument about the appropriateness of a CALL task for particular learners at a given point in time.

50. 50 Integrated research is needed examine the types of CALL activites to seek evidence about CALL appropiratness in particular settings.

51. 51 From such research, implications can be drawn for specific contexts, and eventually research on particular types of activities might be generalised in a way that can inform general principles of CALL.