1. Uncovering the Sources of Student Learning Challenges: Analyzing and Interpreting Data December 4, 2014 Roanoke, VA

2. Welcome and Introduction Aimee Evan Virginia Middle School Research Alliance Lead REL Appalachia

3. Regional Educational Laboratory (REL) Program U.S. Department of Education, Institute of Education Sciences (IES). RELs provide regional support for: – Applied research and evaluation. – Technical support and information sharing to build capacity to use data for improved education outcomes. 3

4. REL Appalachia’s Mission Meet the applied research and technical support needs of Kentucky, Tennessee, Virginia, and West Virginia. Bring evidence-based information to policymakers and practitioners: – Provide support for a more evidence-reliant education system. – Inform policy and practice for states, divisions, schools, and other stakeholders. – Focus on high-priority, discrete issues and build a body of knowledge over time. 4

5. What Does REL Appalachia Do? Assess regional research needs by monitoring emerging education issues and challenges. Maintain and refine research alliances through ongoing dialogue between educators in each region and researchers. Provide analytic technical support to increase use of data and analysis to understand policies and programs, make decisions, and support effective practice. Conduct research and evaluation studies of rigor and method appropriate to the questions the studies attempt to answer. Distribute results of REL research across the region. Coordinate and partner with other RELs and federal, state, and local education research and technical assistance organizations. 5

6. Research Alliances What is a research alliance? – A partnership between education stakeholders and REL Appalachia. What is the purpose of a research alliance? – As partners, REL Appalachia and alliance members develop and carry out a research and analytic technical assistance agenda on priority topics. Who are the education stakeholders in an alliance? – May include representatives from one or more schools, divisions, state education agencies, and other organizations (e.g., colleges and universities). 6

7. How Do We Do the Work: REL Appalachia Staff Justin Baer, Director, REL Appalachia Lydotta Taylor, Alliance Lead, REL Appalachia Kellie Kim, Analytic Technical Support Lead, REL Appalachia Becky Smerdon, Early Warning Systems Content Lead Aimee Evan, VMSRA Lead Angela Estacion, VMSRA Project Lead 7

8. What Is the Virginia Middle School Research Alliance? Collaborative working group of practitioners and researchers. – Superintendents, assistant superintendents, directors of curriculum & instruction, directors of assessment & testing, principals, teachers from the following divisions: Campbell County Greene County Harrisonburg County Louisa County Nelson County Norton City Pulaski County Rockbridge County Russell County Salem City Smyth County Staunton City – Executive directors of SURN (School-University Research Network) and VSUP (Virginia School-University Partnership). – Virginia Department of Education (VDOE). – REL Appalachia researchers in rural education, early warning systems, data use. 8

9. Goals of the Virginia Middle School Research Alliance Assist middle school practitioners in using data to inform instructional decisionmaking and improve student outcomes by: – Identifying struggling students who need additional support. – Selecting, implementing, and monitoring interventions to support students. Focus on expanding the state’s early warning system (EWS) to middle schools. 9

10. Virginia Middle School Research Alliance Projects Catalog EWS data, assessments, and interventions currently being collected/used in schools. Document how data are currently being used in schools, and what supports and barriers are in place to help or hinder use. Determine the most powerful data to use to identify students in need of further assistance. Provide workshops on using data efficiently and effectively to: – Identify struggling students and determine how to target resources to meet their needs. – Monitor each student’s progress. 10

11. Virginia Middle School Research Alliance Future Work Workshops* will continue to build the capacity of educators to interpret meaning from data in order to best meet students’ needs by: – Improving classroom instructional strategies (Spring 2015). – Improving division and schoolwide strategies (Summer 2015). – Workshop on determining the most powerful data in your own division to identify struggling students (Winter 2015). *These activities are currently under review by IES. 11

12. Workshop Overview and Goals Aimee Evan 12

13. Introductions Name Role Division 13

14. Goals for Participants Understand a process to use data to identify sources and causes of students’ learning challenges. Recognize what data sources can be used to accurately identify sources of students’ learning challenges and to monitor their progress toward improvement. Learn what building leadership can do to facilitate and improve teachers’ use of data. Understand your own and your school’s status in developing data-use capacity. 14

15. Logistics Binder Navigation – Agenda – Tabs – Survey Housekeeping: – Bathrooms – Breaks – Lunch 15

16. Introduction to the Early Warning System Process and Foundational Elements Aimee Evan

17. What Is an Early Warning System (EWS)? Provides a systematic way to identify (“flag”) students early who are at risk of failure. Grounded in research. Relies on readily available (and familiar) data: – ABCs: Attendance, Behavior, and Course grades/assessment results. Provides information that is actionable by educators in schools and divisions. Requires educators to diagnose further student needs, and to use professional judgment to support at-risk students. Targets resources to support at-risk students while they are still in school, before they go too far down the road of academic failure and drop out. Examines patterns and identifies school climate issues. Sources: Allensworth & Easton, 2005, 2007; Balfanz, 2009; Balfanz & Herzog, 2005 17

18. The EWS Continuous Improvement Cycle Step 1: Establish roles and responsibilities Step 2: Use the EWS Middle Grades Tool Step 3: Review the EWS data Step 4: Interpret the EWS data Step 5: Assign and provide interventions Step 6: Monitor students and interventions Step 7: Evaluate and refine the EWIMS process Source: Therriault et al., 2013 18

19. The EWS Continuous Improvement Cycle Infrastructure and Support Are Just as Important as the Process. 19

20. EWS – A Systems Perspective Step 1: Establish roles and responsibilities Step 2: Use the EWS Middle Grades Tool Step 3: Review the EWS data Step 4: Interpret the EWS data Step 5: Assign and provide interventions Step 6: Monitor students and interventions Step 7: Evaluate and refine the EWIMS process A. Data and Computer Data Systems B. Educator Knowledge & Skills for Data Use C. School Organization for Data Use 20

21. References Allensworth, E. M., & Easton, J. Q. (2005, June). The on-track indicator as a predictor of high school graduation. Chicago: University of Chicago, Consortium on Chicago School Research. Retrieved from http://ccsr.uchicago.edu/sites/default/files/publications/p78.pdf http://ccsr.uchicago.edu/sites/default/files/publications/p78.pdf Allensworth, E. M., & Easton, J. Q. (2007, July). What matters for staying on-track and graduating in Chicago public high schools: A close look at course grades, failures, and attendance in the freshman year. Chicago: University of Chicago, Consortium on Chicago School Research. Retrieved from http://ccsr.uchicago.edu/sites/default/files/publications/07%20What%20Matters%20Final.pdf http://ccsr.uchicago.edu/sites/default/files/publications/07%20What%20Matters%20Final.pdf Balfanz, R. (2009). Putting middle grades students on the graduation path: A policy and practice brief. Baltimore: Johns Hopkins University, Everyone Graduates Center. Retrieved from http://new.every1graduates.org/putting-middle-grades-students-on-the-graduation-path-a-policy-and- practice-brief/ ‎ http://new.every1graduates.org/putting-middle-grades-students-on-the-graduation-path-a-policy-and- practice-brief/ Balfanz, R., & Herzog, L. (2005). Keeping middle grades students on-track to graduation: Initial analysis and implications. Presentation at the second Regional Middle Grades Symposium, Philadelphia, PA. Therriault, S. B., O’Cummings, M., Heppen, J., Yerhot, L., Scala, J., & Perry, M. (2013). Middle grades early warning intervention monitoring system implementation guide. Washington, DC: National High School Center at American Institutes for Research. Retrieved from http://betterhighschools.org/EWS_middle.asphttp://betterhighschools.org/EWS_middle.asp 21

22. Local Example of Foundational Elements: How Northside Middle School Uses Data Lori Wimbush, Principal April Griffin, English Teacher Christina Hall, Math Teacher Linda Shiflett, Special Education Teacher Laurie Spickard, Data Specialist

23. “Schools to Watch”: Northside Middle School 23

24. Year of Change Northside Middle School Reading Mathematics All Students Disadvantaged SWD Black Hispanic White LEP All Students Disadvantaged SWD Black Hispanic White LEP 2009-10 8980707577936979656066658169 2010-11 9490889490947990847885839271 24

25. New Math SOL Results Northside Middle School Mathematics All Students Disadvantage d SWD Black Hispanic White LEP 2010-11 Results90847885839271 2011-12 Results88806668838179 2011-12 State Results68544052617559 2012-13 Results85796173778773 2012-13 State Results71574155647759 2013-14 Results857861767986 2013-14 State Results74614360678062 25

26. New Reading SOL Results Northside Middle School Reading All Students Disadvantage d SWD Black Hispanic White LEP 2010-11 Results94908894909479 2011-12 Results93877583939472 2011-12 State Results89816680849380 2012-13 Results79684262618344 2012-13 State Results75594359658254 2013-14 Results79684262618344 2013-14 State Results75594359658254 26

27. 2013–14 Annual Measurable Objectives (AMOs) Proficiency Gap Dashboard for Federal Accountability Reading Mathe- matics AMO Target AMO Result Met AMO Target AMO Target AMO Result Met AMO Target All Students6975YES6685YES Gap Group 1: Students with Disabilities, English Language Leaners, Economically Disadvantaged Students (unduplicated) 5960YES5776YES Gap Group 2: Black Students5766YES5676YES Gap Group 3: Hispanic Students60503YR6079YES Key: YES = Met objectives based on the current year result TS = Too small; objective not evaluated due to too few students NO = Did not meet objective - = No data for group N/A = Not applicable 3YR = Met objective based on the 3 year average result R10 = Met objective by reducing failure rate by at least 10 percent < = A group below state definition for personally identifiable results * = Data not yet available 27

28. Changes as a School Promote atmosphere where ALL students can improve! 90 minutes Math and English each day. All teachers remediate during class or before/after school. They turn in weekly SOL evaluation sheets. 28

29. Weekly SOL Evaluation 29

30. More Changes as a School Use Student-Based Performance By Question/By Teacher to drive instruction. Continue to use regular common assessments analysis to drive instruction. 30

31. Student Performance By Question/Teacher 31

32. Analysis by Question 32

33. Analysis by Student 33

34. Changes as a School, continued Monthly and bi-weekly data meetings by subject area: – Common planning. – Benchmark analyses. 34

35. Benchmark Analysis 35

36. Benchmark Analysis 36

37. Benchmark Analysis 37

38. Math Common planning and Math Team meetings: – Analyze common assessments and share successful teaching strategies. – Communicate across grade levels to help use common language when teaching concepts. – Continuous cumulative reviews. – Error analysis after assessments. – Various math programs. – Math Tutors help support teachers to fill in gaps in learning. 38

39. English Students reading 20 minutes per day: – Teachers model reading. – Conference with students about various topics. Students choose what to read and work on needed skill: – Allows students to make choices all year on their reading interests as well as assignments. – Students make choices and they take ownership of their learning. Ultimate goal is for students to become lifelong readers. All teachers in our building can count on students having a book to read during any down time. 39

40. Special Education Use data to determine self-contained Math and English classes. Re-evaluate student learning plans. Use data to drive IEPs. Look at the individual student and his/her specific needs. Focus on improvement for ALL students. 40

41. Social Studies Remediation occurs the month prior to SOLs, during lunch. Creation of computer lab with laptops to allow for increased computer use. Continuation of benchmark analysis meetings. 41

42. Connect with Us! www.relappalachia.org @REL_Appalachia Aimee Evan aevan@QRA-LLC.net 703-655-3695 42