1. Including Students in State Personnel Development Grant Evaluations SPDG Evaluator’s Webinar Sharon deFur, Ed.D. College of William and Mary Evaluator for Virginia’s SPDG February 26, 2013

2. Context happens… sharon.defur@wm.edu February 20132

3. Virginia’s SPDG 2005-2012 School wide initiative focused on improving adolescent literacy Began with 4 schools: 2 middle and 2 high schools School-based literacy teams Intensive all Faculty and Administrator Professional Development on intervention(s) including follow-up, coaching, implementation monitoring by administrators and coaches, data collections School-wide student assessment to identify students in need of interventions beyond classroom routines Teacher evaluation expectations for evidence of use of interventions Opportunities for career advancement with additional professional development for teachers sharon.defur@wm.edu February 20133

4. 4 Q 1. In your evaluation work, do you directly survey or interview students?

5. So think for a few seconds about why a program might not survey or interview students as part of the evaluation – what might be barriers?… sharon.defur@wm.edu February 20135

6. Q 2 What would be the top barrier for your project(s)? 1.Students, particularly students with disabilities, are a protected group, and research approval must be attained. 2.Schools will not grant access to students. 3.Parents will not grant permission for their child to participate. 4.Student time is limited to engage in evaluations. 5.Student voice may not carry weight in program changes. 6.Adults don’t credit students’ ability to effectively provide evaluation information. 7.Policy makers look for quantitative assessment measures. 8.Evaluation resources are limited. sharon.defur@wm.edu February 20136

7. My Arguments for Including Students in Program Evaluation… Student achievement is often an intended outcome. Students are often a subject of interventions. Students are primary clients of teachers and schools. Students theoretically experience the intervention or the teacher change every day. Students usually see more than one teacher and have an automatic comparison base. Students have a school history. Other? sharon.defur@wm.edu February 20137

8. Given these common evaluation questions, how might students provide value- added data? Are interventions being implemented? Are interventions reaching the intended audiences? Are interventions being implemented with fidelity? Are teachers changing (for the better!) as a result of the intervention? Are the intended audiences satisfied with the intervention? Are interventions making a positive difference for students? Will the effects of the intervention be sustained? Are there any positive or negative un-intended consequences of the intervention? What, if anything, needs to change? sharon.defur@wm.edu February 20138

9. Student Voice Timeline in Virginia’s SPDG Evaluation Student input was not on our original evaluation plan. First access to students was via a Virginia Department of Education request for student input on effective teaching as part of the development of standards for special education teachers. sharon.defur@wm.edu February 20139

10. Our Success in Gaining Access to Students Depended on… Virginia Department of Education support School and SPDG project support Assuming an evaluation perspective, not pure research perspective Voluntary participation by schools and students Service to schools and project Clear permission guidelines and short time obligation Human Subjects Protocol Approval (university and school division, as appropriate) sharon.defur@wm.edu February 201310

11. sharon.defur@wm.edu February 201311

12. Student Focus Group – SPDG Implementation Year I METHODOLOGY Four schools, 2 middle and 2 high 74 students ~ ½ with IEPs 8 focus group interviews Student selection and permission gained via school personnel METHODOLOGY Four schools, 2 middle and 2 high 74 students ~ ½ with IEPs 8 focus group interviews Student selection and permission gained via school personnel GUIDING FOCUS QUESTIONS Think about your school. Imagine you are talking to a new student and he or she asks about your school – what would be the “best” part of school for you and why? Talk about one class where you think you do best academically. If I were a student in your school and I needed help with a subject, how would I get this help? Imagine that you can create the perfect class. In this class you get good grades, learn what you need to know, and do well on the SOL tests. Describe how this class would look and what would happen there. Imagine that you are principal of your school and you have been asked to make changes so that you and your peers would be successful academically. What changes, if any, would you recommend? sharon.defur@wm.edu February 201312

13. Surprises… Themes –School as community –The power of belonging –Teachers make a difference –Active and Engaging Instruction –Co-teaching can be good –Class size matters –All students, regardless of ability can and want to contribute their voice S. deFur, & L. Korinek. (2010). Listening to student voices. The Clearinghouse, 15-19. Themes –School as community –The power of belonging –Teachers make a difference –Active and Engaging Instruction –Co-teaching can be good –Class size matters –All students, regardless of ability can and want to contribute their voice S. deFur, & L. Korinek. (2010). Listening to student voices. The Clearinghouse, 15-19. AND without prompting –Students used the vocabulary of the intervention indicating the intervention was being implemented –Students expressed opinions as to whether certain parts of the interventions were helpful or not –Students identified teachers who were using the interventions effectively –Students identified teachers who were using the intervention with fidelity (my words, not theirs) AND without prompting –Students used the vocabulary of the intervention indicating the intervention was being implemented –Students expressed opinions as to whether certain parts of the interventions were helpful or not –Students identified teachers who were using the interventions effectively –Students identified teachers who were using the intervention with fidelity (my words, not theirs) sharon.defur@wm.edu February 201313

14. Student Voice Became Part of Virginia’s SPDG Evaluation Schools became interested in student feedback –Schools began asking for student feedback and integrating that into their actions Schools wanted the evaluation team to return as part of the summative evaluation after 3 years Schools became interested in student feedback –Schools began asking for student feedback and integrating that into their actions Schools wanted the evaluation team to return as part of the summative evaluation after 3 years sharon.defur@wm.edu February 201314

15. Focus Group Interview II – Year 3 of Implementation Evaluation Focus –85 students –4 schools, two middle 8 th grades, two high 11 th & 12 th grades, two high 9 th grades –10 groups, 4 targeting students with IEPs, 4 students w/o IEPs, and 2 groups mixed –Intentional design to gather information directly related to intervention tools Similar themes to first cohort of interviews 90 -100% of participants indicated exposure to two of the most common intervention tools with no pronounced differences between the groups Similar themes to first cohort of interviews 90 -100% of participants indicated exposure to two of the most common intervention tools with no pronounced differences between the groups sharon.defur@wm.edu February 201315

16. Illustrative evaluation student quotes... What evaluation questions might these address? 1.“I’ve been here since 6 th grade and my grades have gotten better, math is better, science too. Since I’ve been here, everything has been helping; here there are more choices. Other schools don’t have as much.” 2.“in history, we talk and then fill it in... this helps me remember better.” 3.“If I have to write papers, I use the ***** to organize so I know exactly what I’m writing – I show what I want to include.” 4.“it helped me pass the SOL.” 5.“some teachers use it so much it gets on your nerves, then you don’t want to use it.” 6.“It might be useful to help 9th and 10th graders to get organized, but it’s not good for grades 11 and 12.” 7.“This stuff is in the wrong age group, should be used in a lower age group – perfect for first grade, when you don’t know stuff yet.” 8.“Teachers were stressing about it, we skipped chapters in order to use *****.” sharon.defur@wm.edu February 201316

17. In Spring 2012, there were students graduating who had been in schools using the intervention(s) since grade 6! sharon.defur@wm.edu February 201317

18. Focus Group Interview III: Intervention Year 7 Spring 2012 - 13 Seniors, 1 Junior, all exposed to intervention since grade 6. 7 exposed to whole class intervention routines and 7 also exposed to more intensive strategies Evaluation focus on specific instructional components of the intervention Coded by question and then theme using NVIVO 8. Spring 2012 - 13 Seniors, 1 Junior, all exposed to intervention since grade 6. 7 exposed to whole class intervention routines and 7 also exposed to more intensive strategies Evaluation focus on specific instructional components of the intervention Coded by question and then theme using NVIVO 8. sharon.defur@wm.edu February 201318

19. CODED THEMES sharon.defur@wm.edu February 201319 Most common coding reported the positive impact of the interventions Teacher effectiveness in the use of intervention matters Intervention utility perceived as more effective at earlier grades Student understanding of the use of the intervention tools Intensive instruction makes a difference for students with disabilities Most common coding reported the positive impact of the interventions Teacher effectiveness in the use of intervention matters Intervention utility perceived as more effective at earlier grades Student understanding of the use of the intervention tools Intensive instruction makes a difference for students with disabilities

20. Illustrative Quotes from Seniors: Evaluation Implications? 1.“Like, it’s become second nature when taking notes or for homework or when we have to outline a chapter…” 2.“I think that’s one of the more important things as long as the teacher is using it the right way and they know how to use it then I think it works better. And that’s why I think …because I’ve had lots of teachers that have known how to use it.” 3.“I think sometimes teachers just force it because that’s what they’re [emphasized] supposed to do so it like it’s really just kind of a waste of our time when it doesn’t work with what we’re learning about.” 4.“It was helpful in middle school because it just helped prepare us for, or like, prepared us for what we had here.” 5.“I also think that if some teachers can’t, all teachers can teach but some teachers, like, some kids don’t understand the teacher’s teaching. So, by them giving us the ***** I think that’s better understanding for us, which also they know that we don’t get it and we keep telling them then they’ll pull out their *****. Like we’ll do something like keep going over and over until we actually get it.” 6.“I don’t know where I’d be at right now if I didn’t have those classes because I couldn’t read at all.” 1.“Like, it’s become second nature when taking notes or for homework or when we have to outline a chapter…” 2.“I think that’s one of the more important things as long as the teacher is using it the right way and they know how to use it then I think it works better. And that’s why I think …because I’ve had lots of teachers that have known how to use it.” 3.“I think sometimes teachers just force it because that’s what they’re [emphasized] supposed to do so it like it’s really just kind of a waste of our time when it doesn’t work with what we’re learning about.” 4.“It was helpful in middle school because it just helped prepare us for, or like, prepared us for what we had here.” 5.“I also think that if some teachers can’t, all teachers can teach but some teachers, like, some kids don’t understand the teacher’s teaching. So, by them giving us the ***** I think that’s better understanding for us, which also they know that we don’t get it and we keep telling them then they’ll pull out their *****. Like we’ll do something like keep going over and over until we actually get it.” 6.“I don’t know where I’d be at right now if I didn’t have those classes because I couldn’t read at all.” sharon.defur@wm.edu February 201320

21. Questions? Comments? Ideas? sharon.defur@wm.edu February 201321

22. Disclaimer Funding for this project was partially provided by a grant from the Virginia Department of Education and the USDOE Virginia‘s State Personnel Development Grant # H232070029. This project was found to comply with appropriate ethical standards and was exempted from the need for formal review by the College of William and Mary. The opinions expressed herein do not reflect those of the Virginia Department of Education or the USDOE. sharon.defur@wm.edu February 201322

23. Protocol for Exemption (Koppelman, E. Online Ethics Center Explanation of Federal Regulations for Human Subject Research. Retrieved January 24, 2013 from http://www.onlineethics.org/cms/13121.aspx. ) Research conducted in established or commonly accepted educational settings, involving normal educational practices, such as –Research on special or regular instructional strategies –Research on the effectiveness of, or the comparison among instructional techniques, curricula, or classroom management methods Research involving the use of educational tests, survey procedures, interview procedures, or observation of public behavior, unless… Research involving the collection or study of existing data, documents, records, etc… Research and demonstration projects which are conducted by or subject to the approval of Department or Agency Heads, which are designed to study, evaluate, or otherwise examine… sharon.defur@wm.edu February 201323