Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluating Task-Based Language Programs Colloquium – TBLT 2009 NOTE: This PowerPoint presentation has been modified by removal of all high-resolution graphics,

Similar presentations


Presentation on theme: "Evaluating Task-Based Language Programs Colloquium – TBLT 2009 NOTE: This PowerPoint presentation has been modified by removal of all high-resolution graphics,"— Presentation transcript:

1 Evaluating Task-Based Language Programs Colloquium – TBLT 2009 NOTE: This PowerPoint presentation has been modified by removal of all high-resolution graphics, to reduce the storage and downloading requirements (sorry, no pretty photos!).

2 The colloquium Why bother with TBLT program evaluation? Three presentations + clarification questions (2:00-3:30): Re-framing the evaluation of task-based language education Evaluating a TBLT Spanish immersion program Evaluation of TBLT in Flanders Open audience-panel discussion (3:30-3:50)

3 John M. Norris University of Hawaii at Mānoa TBLT 2009 Re-framing the evaluation of task-based language education Please cite as: Norris, J. M. (2009, September). Reframing the evaluation of task-based language education. Paper presented at the refereed colloquium Evaluating task-based language programs, at the 3rd International Conference on Task-Based Language Teaching, Lancaster, UK (September 14, 2009).

4 TBLL v. TBLT: Disconnects between inquiry and practice

5 What is task-based language learning (TBLL)? 1. Societal need for change in language education… Value Outcomes Methods 2. Emerging notions of L2 acquisition… Processes Impediments Indicators

6 What is task-based language learning (TBLL)? Proposals Practices Findings Hypotheses Observations 1. Societal need for change in language education… Value Outcomes Methods 2. Emerging notions of L2 acquisition… Processes Impediments Indicators Discussions Opportunity for a researched language pedagogy

7 What is task-based language learning (TBLL)? Pedagogic principles, such as… Promote learning by doing, experiential learning Use task as the unit of analysis for instruction & assessment Provide rich L2 input Elaborate (rather than simplify) L2 input Respect learner-internal syllabuses Promote collaborative-cooperative interaction Enable inductive/chunk learning Provide focus on form, negative feedback (e.g., Doughty & Long, 2003; Ellis, 2003)

8 Task-Based Language Teaching Cognitive Psychology Philosophy of Education Curriculum theory SLA Planning and policy Learners Assessment Curriculum Instruction Teacher development Materials Rationales and PrinciplesL2 Education Programs Sociocultural theory Needs Applied to Inform??? What is task-based language teaching (TBLT)?

9 What is the role of task-based inquiry? Task-Based Language Teaching Task-Based Language Learning Generate theory Discover robust, if small, truths Test hypotheses Improve teaching practice Inform curriculum, course design Understand what works, where, when, & why HOLISTIC GENERALIZABLE SITUATED DISCRETE

10 What is the role of task-based inquiry? Task-Based Language Teaching Task-Based Language Learning Generate theory Discover robust, if small, truths Test hypotheses Improve teaching practice Inform curriculum, course design Understand what works, where, when, & why HOLISTIC GENERALIZABLE SITUATED DISCRETE

11 What is the role of task-based inquiry? Task-Based Language Teaching Task-Based Language Learning Generate theory Discover robust, if small, truths Test hypotheses Improve teaching practice Inform curriculum, course design Understand what works, where, when, & why HOLISTIC GENERALIZABLE SITUATED DISCRETE

12 Challenges for task-based inquiry 1. The scope of task-based research does not match the scope of our claims about (for or against) TBLT. 2. The focus of theoretical task-based research does not relate to the situated realities of task-based teaching. Needs Curriculum Materials Instruction Teachers Learners Assessment TBLT Education Programs Goals, outcomes Scope, sequence Resources Practices History, training Individual differences Intended uses, users

13 Framing TBLT inquiry through program evaluation

14 Research emphasizes theoretical, conclusion-oriented inquiry Evaluation operationalizes decision- oriented inquiry Inquiry through evaluation Cronbach & Suppes (1969)

15 Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology. Norris (2006) MLJ Perspectives Inquiry frame and focus Inquiry question prioritization Inquiry impetus Inquiry through evaluation

16 The evaluator will be wise not to declare allegiance to either a quantitative-scientific- summative methodology or a qualitative- naturalistic-descriptive methodology. (p. 7) Cronbach et al. (1980) Paradigms Epistemology 1 Methodology 1 Epistemology 2 Methodology 2 Inquiry through evaluation

17 The evaluator will be wise not to declare allegiance to either a quantitative-scientific- summative methodology or a qualitative- naturalistic-descriptive methodology. (p. 7) Cronbach et al. (1980) Paradigms Epistemology 1 Methodology 1 Epistemology 2 Methodology 2 Inquiry through evaluation

18 The evaluator will be wise not to declare allegiance to either a quantitative-scientific- summative methodology or a qualitative- naturalistic-descriptive methodology. (p. 7) Cronbach et al. (1980) Pragmatism Who? Method 1 Why? Method 2 What?When? Method 5 Method 4 Method 3 Inquiry through evaluation

19 1. Participation – stakeholders, representatives, primary intended users 2. Prioritization – challenges, questions in immediate need of answers 3. Instrumentation – what data will answer the questions? 4. Collection – how can we get data in available time/resources? 5. Interpretation – what do findings mean in context? 6. Utilization – what decisions & actions are taken? Language educators are ultimately responsible for what happens in language education. Participation by language educators is essential throughout evaluation if contextual relevance is sought. A focus on specific intended uses for evaluation findings is essential from the outset, if evaluation is to make any difference. Inquiry through evaluation

20 Patton (1997) Utilization-focused evaluation Context: Intended uses Context: Intended users Questions + Methods Understand Improve Educate Demonstrate worth Hold accountable Empower (Test theory) Teachers Administrators Curriculum writers Learners Parents/public Funders (Researchers) Values clarification Implementation Process-product Outcomes Evaluating TBLT Programs in situ Learner needs, institutional resources, program goals and outcomes, curriculum, materials, instruction, assessment, teachers, teacher development, learners, etc. Context: Program features

21 Learning from evaluation examples: the brief history of TBLT

22 Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Prabhus Bangalore Project (See Prabhu, 1987) Context English L2 Education in Bangalore, India; Seeking Improvement via Innovation L2 learning by processing meaning; Unconscious grammar construction by learners Project/task- based work; 4 experimental schools; Implemented TheoryProgram

23 Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985) To assess, through appropriate tests, whether there is any demonstrable difference in terms of attainment in English between classes of children who have been taught on the CT project and their peers who have received normal instruction in the respective schools. Beretta & Davies (1985) Initial inquiry, final year of the project:

24 Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985) Purpose Test theory Demonstrate method effectiveness Methods Quasi-experimentation Class/method comparison Outcome achievement assessments Findings Structures test: Control > CTP Contextual grammar: Control = CTP Dictation: Control = CTP List/Read comp: CTP > Control Task-based test: CTP > Control Claims??? Task-based learners achieved as much or more than traditional on all but the least functional outcomes Task-based instruction is successful Warranted claims??? What do we really know???

25 Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Phase 2 (See Beretta, 1986, 1990, 1992) Purpose Understand program implementation Illuminate relation with apparent outcomes Methods Retrospective interview protocols Teacher level of concern questionnaires Document analysis Findings Lack of comparability (intact classes, no baseline data) More qualified teachers in CTP classes Implementation of CTP highly variable (over time, between classes, with structures) More confident teachers = better results

26 Learning from TBLT evaluation Evaluating the Communicational Teaching Project – Lessons Learned (See Beretta, 1992) Theory testing, methods comparisons, what works claims are rarely feasible in real educational programs Teachers (beliefs, training, commitment, time) play a key role in implementing programs: what they actually do must be understood Even poorly executed evaluations (e.g., post-hoc) can shed light on how programs function and help explain why learning does or does not occur Apparent differences in learning achievement, behaviors, etc. can only be explained by observation of multiple factors as they are experienced in real program contexts

27 Learning from TBLT evaluation Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999) Context French FL Education, UK university, Salford; Restructuring advanced FL teaching Input, text, task; Learning through form- function mapping in tasks Task-based syllabus; Multiple levels at university; Implemented TheoryProgram

28 Learning from TBLT evaluation Curriculum design, evaluation, application and enhancement is a slow process, and subject to a number of extraneous influences which make it impossible to measure with totally scientific precision…use of diaries and questionnaires on the first occasion enabled a number of lessons to be learned and these helped considerably in creating a second application where the testimony of the student population through a detailed questionnaire shows the success of the operation. Towell & Tomlinson (1999) Multiple iterations of development, implementation, evaluation, revision: Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999)

29 Learning from TBLT evaluation Program: Initial TBLT long group projects Methods: Learner diaries Learner surveys Assessments/exams Findings: Projects too long (6 wks) Training in group work Staged task objectives Gains in text/task learning Developing accuracy? STAGE1STAGE1 Program: Revised TBLT staged, short projects Methods: Learner surveys Focus groups Assessments/exams Findings: Increased satisfaction Higher learning of skills Improved oral translation Written translation? Developing accuracy? STAGE2STAGE2

30 Learning from TBLT evaluation Student/learner perspective on teaching with tasks sheds important light on the realities of implementation (how + how well) Learners can change how they learnacculturating to TBLTespecially when tasks, instructions, assessments are intentionally designed and staged to do so Triangulated learner feedback (diaries, self-assessments, questionnaires, exams) can lead to effective improvements in curriculum and task design, and in turn to higher evaluations Building evaluation activities into curricular delivery from the outset (e.g., student diaries), enables longitudinal insights about change, development, response to instruction Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999)

31 Learning from TBLT evaluation Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007) Context English FL education, Thai university; Improving EAP instruction Integrated-skills, communication; Life-long learning; Learner needs + interest orientation Task-based syllabus; English department; Implemented 12 months TheoryProgram

32 Learning from TBLT evaluation …relatively few empirical studies have documented how teachers and learners react to entirely task-based courses, as opposed to the use of individual tasks…The purposes of this case study were (a) to identify teacher and learner reactions to the course and (b) to describe how their concerns, if any, were addressed. McDonough & Chaikitmongkol (2007) Inquiry for developing and improving TBLT experiences: Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007)

33 Learning from TBLT evaluation Methods: Learner task evaluations (repeated) Learning notebooks Class observations Student course evaluations Teacher/student interviews Observer field notes Findings: Increased learner independence, language skills, learning strategies Decreased grammar obsession Non-specific real-world relevance Need time to adjust (T&L) More support, guidance from teachers Too much to cover, disparate materials Uses: Intro unit on language learning Teachers guide to instruction + workshop Enhanced task guidelines, built-in feedback opportunities Reduced number of tasks Consolidated materials Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007)

34 Learning from TBLT evaluation Cycles of evaluation planned into TBLT innovation, and carried throughout, can lead to increased likelihood of effectiveness TBLT based on learner needs can work well in EAP contexts, especially when evaluation is used to support on- going effectiveness of delivery from the outset Teachers and learners both require support in implementing TBLT, especially during early phases of introducing task-based instruction Systematic evaluation (a) from multiple stakeholder perspectives and (b) focused on multiple program elements (materials, preparedness, outcomes) enables balanced change Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007)

35 Dutch SL Education in Flanders; Nationwide; K-16; Ensuring Educational Access, Equity Large-scale Task-Based LT Innovation; Improving Functional DSL Abilities School-based Teacher- Training Programs; Enabling Change, Learning from TBLT evaluation ContextTheoryProgram Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006)

36 Van den Branden (2006): …the teacher tries to act as a true interactional partner, negotiating meaning and content with the students, eliciting and encouraging their output, focusing on form when appropriate and offering them a rich, relevant and communicative input (p. 217). Evaluation PROBLEMSEvaluation USES Teacher cognition Teacher action Teaching context What do they theorize about TBLT? How do they learn about TBLT? Are they willing to change with syllabus? How do they adopt/adapt TBLT in practice? What are the social constraints on T-Dev? How can T-Dev be optimized? Understand teachers Illuminate context Improve T-dev program Encourage teacher agency Ensure teaching success Enable TBLT learning Demonstrate outcomes Learning from TBLT evaluation

37 Program: Theoretical inservice training Methods: Teacher survey Training observation Findings: Transmission model Short term (3 hrs.) Try that with my students…Post- coursal depression! STAGE1STAGE1 Program: TB training + syllabus support Methods: Teacher logs, interviews, classroom observations Findings: +awareness of TBLT +student enthusiasm ?teacher adoption -teacher control -task complexity - groupwork STAGE2STAGE2 Program: Training + coaching + agency Methods: Coaching obs, classroom obs, coach/ teacher interviews Findings: +conscious decisions +TBLT adaptation +self-evaluation ?teacher control - transfer - groupwork STAGE3STAGE3 Learning from TBLT evaluation

38 Program: Sustained implementation of TBLT T-Dev with coaching, support Methods: Pre-post student learning outcomes, teacher surveys, classroom observations Findings: +incorporation of TBLT correlated with higher Dutch L2 proficiency outcomes +3-year gains in DSL higher in TBLT intensive adopting schools ?mixed incorporation of TBLT across schools, teachers STAGE4STAGE4 Learning from TBLT evaluation

39 Long-term evaluation of TBLT sheds light on how ideas are implemented, how participants change, and what support is needed Teachers can learn to engage with TBLT, but change takes time, requires individualized support, and must be valued Persistent follow-through on evaluation findings (use) underlies effective innovation Multi-directional evaluation (political, social, school, individual) increases our capacity to explain why task-based ideas work or do not Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006)

40 What have we learned? Reframing evaluation in TBLT From summative to intentional From assessment-driven to multi-methodological From external to participatory From method-testing to program-illuminating From one-shot to longitudinal, cyclical Learning from TBLT evaluation From theoretical conclusions to educational decisions

41 Research, evaluation, and the future of task-based education

42 TBLL research Sociocultural, cognitive, and other theories provide useful starting points for thinking about language teaching and learning, and offer principles for building educational programs Task-based language learning research helps in that it raises our awareness about particular factors that we should pay attention to in the instructed L2 learning process Task-based language learning research cannot tell us much about how or why language education programs work; findings from TBLL research should not be interpreted as direct implications for TBLT education

43 TBLT evaluation Intentional Evaluative Inquiry Answers questions & informs decisions of local interest Sheds light on how TBLT ideas work in practice Provides truths situated in rich contexts of programs Relates outcomes to TBLT delivery and other factors Focuses on scope that is meaningful to teachers, learners Tests and informs innovation on the ground, in situ Empowers participants to learn, and learn to change Forces an honest accounting of TBLT

44 TBLT evaluation Challenges For TBLT Evaluation Resources: It takes time and money to do evaluation well and to sustain it within L2 educational programs. Training: Effective evaluation calls upon skills that may not be easily available among personnel at hand. Dissemination: There are few venues for publishing evaluation reports, thereby limiting learning. Actual uses: There are many possible uses/needs for evaluation that we are not sufficiently attuned to, yet.

45 Cheers! (Mahalo!) References Beretta, A. (1986). Program-fair language teaching evaluation. TESOL Quarterly, 20, Beretta, A. (1990). Implementation of the Bangalore Project. Applied Linguistics, 11(4), Beretta, A. (1992). What can be learned from the Bangalore evaluation? In J. C. Alderson and A. Beretta (eds.), Evaluating second language education (pp ). Cambridge: Cambridge University Press. Beretta, A., & Davies, A. (1985). Evaluation of the Bangalore Project. ELT Journal, 39(2), Cronbach, L. J., & Associates. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass. Cronbach, L., & Suppes, (1969). Research for tomorrow's schools: Disciplined inquiry for education. New York: Macmillan, Doughty, C., & Long, M. H. (2003). Optimal psycholinguistic environments for distance foreign language learning. Language Learning & Technology, 7, Ellis, R. (2003). Task-based language learning and teaching. New York: Oxford University Press. McDonough, K., & Chaikitmongkol, W. (2007). Teachers and learners reactions to a task-based EFL course in Thailand. TESOL Quarterly, 41(1), Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), Norris, J. M. (2009). Task-based teaching and testing. In M. Long and C. Doughty (Eds.), Handbook of language teaching (pp ). Cambridge: Blackwell. Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage. Prabhu, N. S. (1987). Second language pedagogy. Oxford University Press, Oxford. Towell, R., & Tomlinson, P. (1999). Language curriculum development research at university level. Language Teaching Research, 3(1), Van den Branden, K. (2006). Training teachers: Task-based as well? In K. Van den Branden (ed.), Task-based language teaching in practice (pp ). Cambridge: Cambridge University Press.


Download ppt "Evaluating Task-Based Language Programs Colloquium – TBLT 2009 NOTE: This PowerPoint presentation has been modified by removal of all high-resolution graphics,"

Similar presentations


Ads by Google