1. Program Evaluation Week 6

2. Tonight Warm up Chapter 7 Chapter 8 Creating an evaluation HW

3. Warm up Take a moment to share with your partner: – What you learned – What you have questions about

4. Chapter 7 Who requested the evaluation? Why was it conducted? What data was requested and why?

5. Program objectives What were they? Look at objectives 1 and 2 – Create a research question to evaluate each objective

6. Operationalization How would you operationalize English language achievement? Motivation/interest? How could you measure these things?

7. How did they operationalize these constructs? Look p. 125

8. The authors talked about the tests created for the project. Why would the content of the achievement tests matter?

9. How might test content/operationalization impact “Truth” from a quantitative perspective?

10. p. 127 “raised awareness about the limited perspective that data from the experimental study would provide.” What limitations are they talking about? Why is this important?

11. Ethnography What is it? Why could it help? What assumptions is it based upon? Is reliability and validity still an issue?

12. Findings Improved English ability – Did context influence results? How/why? Increased attitudes P. 130– Disruption. What can be done with data about this? P. 133– the evaluation activities now include… Why these?

13. Your Q’s?

14. Chapter 8 Primary Modern Language Project Aims? Context? 2 ways it differed from others we read, p. 137

15. p. 142 Aims How could aims be assessed? Quant or Qual? – How would you do it?

16. Look at the aims of the evaluation and the instruments used. How are the aims reflected in the instruments? – In other words, what data help shed light on the aims?

17. P. 143 How does the test match its goals?

18. Administration of the questionnaire What problems did they encounter? Why does this matter?

19. The timing problem from different perspectives – Ex. P. 147

20. Stake & Ethnography Stake p. 137 and Ethnography p. 123 What commonalities are shared?

21. Closing points P. 148– tradition of focusing on outcomes.. Why do the authors state the need for classroom observations? What should be observed?

22. Your Q’s?

23. Creating an evaluation I am language center director and I want to know about the effectiveness of IT instruction in the development of oral language strategies. Operationalization: -what do I need to operationalize?

24. foci I am language center director and I want to know about the effectiveness of IT instruction in the development of oral language strategies. Quant question? Qual question (s)?

25. Context considerations?? What should we think about in regards to context and design?

26. What instruments will we use? – RQ 1- – RQ2-

27. What should we consider about the instruments?

28. OCSI What does it measure? Do we need to adapt? Which ones?

29. Adapted OCSI Show Additional data source – why?

30. Reliability of surveys – Show OCSI Cronbach alpha

31. Anything to consider regarding the groups?

32. Look at strat use of groups at pretest What if different? What if only one group?

33. HW Lynch (1996)– this was emailed to you Kiely & Rea-Dickens (2005)- Ch 14 Teacher led evaluations. Journal entries Have your research questions in mind. We will share these.