1. Nora Gannon, Tysza Gandha & Katherine Ryan 2009 CREATE Conference October 9, 2009

2. Accountability and Evaluation Using Questionnaires in Evaluation Using Mixed Methods in Questionnaire Development Our Mixed Methods Design and Implementation Summary

3. No Child Left Behind (NCLB) Credible Evidence Improving Evaluation Methods

4. Key Questions What are the intended and unintended consequences of the accountability assessments utilized in this state? What are the perspectives of teachers and principals on how these consequences influence: Instructional practices? Local assessment practices? Use of test data? School policies and practices? The Teaching profession?

5. Illuminate large-scale patterns in schools reform as a result of accountability systems in place Provide broad-based views of stakeholders perspectives (e.g., teachers and principals) Time and cost efficient Reference: Desimone & le Floch, 2004

6. Advantages to Using Questionnaires Challenge for Researchers Administer to many people to learn perspectives of diverse stakeholders How do we know if a respondents understand questionnaire items as we intended? Large scale survey study can examine a wide range of contexts (e.g., school settings) How can we strengthen our inferences across contexts?

7. Reduce response bias Reduce response burden Improve item clarity Reference: Czaja & Blair, 2005

8. Purposeful Systematic Comprehensive

10. P HASE 1: QUALITATIVE Methods Item Development Cognitive Labs Expert Review

11. Literature Review 5 dimensions: Instructional practices, Local assessment practices, Use of test data, School policies and practices, Teaching profession Dimension Task Lists Item development Item Bank Reverse engineering Content Review P HASE 1: QUALITATIVE

13. Literature Review 5 dimensions: Instructional practices, Local assessment practices, Use of test data, School policies and practices, Teaching profession Dimension Task Lists Item development Item Bank Reverse engineering Content Review P HASE 1: QUALITATIVE

14. What is the purpose of a cognitive lab? Gain insight into participant understandings. How do you sample for cognitive labs? Choose participants that represent your population. Implementation: the think-aloud process Be careful not to influence responses! P HASE 1: QUALITATIVE

15. Single vs. Dual Rating Scales P HASE 1: QUALITATIVE

16. Single vs. Dual Rating Scales P HASE 1: QUALITATIVE

17. Global Questions P HASE 1: QUALITATIVE

19. Language What does test preparation materials mean? P HASE 1: QUALITATIVE

20. Language Strategies for Addressing Language Issues Examples Alternate words Multiple questions P HASE 1: QUALITATIVE

21. Why use Expert Reviewers? P HASE 1: QUALITATIVE When to engage expert reviewers? Early Can solicit broad, big picture feedback Have time to make major changes with feedback Later Can solicit highly specific (item-level) feedback Useful for final review of instruments in almost- final draft Reference: Ramirez, 2002

22. Methods Pilot Questionnaire Cognitive Lab Focus Group P HASE 2: Q UANTITATIVE AND qualitative

23. Teacher and Principal Questionnaires Approximately 200 items; 60% shared between teacher and principal versions Sample -383 principals- 2415 teachers (1940 online, 475 mail) Proportionate Probability Sample (N=400 schools) Stratified by geographic region and school type (i.e. elementary schools and middle schools) Sampling Criteria (AYP Subgroups) P HASE 2: Q UANTITATIVE AND qualitative

24. Teachers: 37% (902) Online: 44% Paper-and-pencil: 9% Principals: 44% (175) P HASE 2: Q UANTITATIVE AND qualitative Population proportions Response rate proportions Elementary60%57% Middle School40%43%

26. Refine content of Questionnaire based on Low response endorsement Low response rates Low variance on scaled items P HASE 2: Q UANTITATIVE AND qualitative

27. IMPLEMENTATION: Cognitive lab in small group setting (4-6 people) PURPOSE: Group narrative SAMPLE: Depends on what you want to know StrengthsLimitations Group setting allows for conversations that elicit normal narrative Cost-efficient way to learn about potentially different interpretations Have to use smaller number of items than individual cognitive lab Group relations need to be considered (e.g., include teachers and principal?) P HASE 2: Q UANTITATIVE AND qualitative

28. To what extent is the change in local assessment due to the state accountability assessment? Mr. Smith: I think assessment is just a component of good teaching. Ms. Brown: I think you are naïve…its all about preparing for the state test. Ms. Rogers: Well, it depends on how you look at it, I guess… if you are cynical everything is because of accountability assessment! Cognitive Focus Group P HASE 2: Q UANTITATIVE AND qualitative

29. GoalMethod Refine scope of questionnaire Comprehensive literature review Expert reviewers Pilot administration Improving the interpretability of the questionnaire Cognitive labs Cognitive focus groups Strengthen capacity to warrant inferences within and across contexts Cognitive labs Purposeful sampling for cognitive labs Deliberate sample frame in pilot

30. Thank you!