1. Building Capacity for Effective Implementation and Student Impact Across Education Cammy Lehr and Eric Kloos September 11, 2014 Minnesota Department of Education education.state.mn.us 1

2. A Minnesota Journey…. education.state.mn.us 2

3. This session will explore how the MDE has been applying the Implementation Frameworks of the SISEP Center to agency priority initiatives, school and district level teams, and scaling-up PBIS. Session will include discussion of –Lessons learned - especially in relation to district teams –Rationale for the importance of using performance assessment to guide work –Examples and learnings that can be generalized across states and initiatives. Overview education.state.mn.us 3

4. National Center on State Implementation of Scaling-up of Evidence-based Practices (SISEP) June, 2008, Minnesota was selected to work actively with a new national technical assistance center, SISEP Renewed the partnership with SISEP in January of 2013 SISEP and MDE’s Shared Goal: Building state capacity for scaling up evidence-based practices Broader impact emerged over time stimulated by the MDE Implementation Team Engaged in Systems Change! Not a Project, A Way of Work

5. Original MDE Implementation Team Formed in the Spring of 2009

6. Minnesota Department of Education MDE Implementation Team (MIT) “Implementation Specialists” Members represent: Federal Programs, English Learners Office of Early Learning Special Education, Research, Practice and Implementation Regional Centers of Excellence Expansion Office of the Deputy Commissioner School Support, Professional Development Academic Standards and Instructional Effectiveness School Support, Statewide System of Support New additions coming soon… Postsecondary, College and Career Success Safety Health and Nutrition

7. Educational Best Practices Initiatives System Stability Mandates, System Supports, Foundational Polices & Regulations MDE & SISEP SEA Responsibilities and Leverage Points Implementation Best Practices and Infrastructure Effective Sustainable Accessible Scalable Continuously Improved Integrated Common Framework at MDE Systems Change

8. Capacity Development in Minnesota Technical Assistance from SISEP is different Driven by common mission and vision Proactive and responsive Focused on capacity building at multiple levels Gentle pressure, relentlessly applied Honest feedback is important – positive and developmental, mutual Staff from SISEP provide ongoing technical assistance  Monthly MN on-site visits from SISEP staff  Implementation Institutes  Active Implementation Hub and SISEP Website  Resource materials (briefs, binder, book)  Community of Practice  Feedback on products and adaptations

9. State Management Team Regional Implementation Team “District” Teams N ≈100 Schools Regional Implementation Team “District” Teams N ≈100 Schools Regional Implementation Team “District” Teams N ≈100 Schools Regional Implementation Team “District” Teams N ≈100 Schools SISEP Support 2.0 State Transformation Specialists State Design Team SISEP GENERIC SCALING-UP MODEL For Implementation Capacity Development & Systems Change

10. State Management Team MDE Executive Team Function: Messaging, address policy, support MDE Internal Implementation Support Team (IIST) Function: Apply and support implementation frameworks to MDE work Regional Statewide System of Supports 3 Regions n = 74 schools Regional PBIS 3 Regions n = 430+ schools District Leadership and Implementation Teams 2 Districts n = 46 schools Regional Early Childhood Special Education 8 Regions 12 PD Facilitators State Implementation Specialists MDE Internal Implementation Team (MIT) Function: Build District & MDE Capacity to Use Implementation Frameworks External Organizations (e.g., U of MN, MASA, Service Coops.)

11. 11 School Capacity Building Regional/District/Site Capacity Building State Capacity Building Leadership and Coordination by 1.0 State Implementation Specialist Internal Implementation Support Team (IIST) Director Level /n=9 MDE Implementation Team (MIT) Supervisor & Education Specialists /n=9 Improved Student Outcomes Federal Capacity Building State Implementation & Scaling-up of Evidence-based Practices (SISEP) Center Executive Team Providing Support Driving Improvement Aligning Policy and Practice Doing the Right Work at the Right Level Assessing How We Are Doing Engaging in Continuous Improvement Positive Behavior Interventions and Supports 3 Regions /Over 430 Schools (to date) District Leadership & Implementation Teams (Intensive Work) 2 Districts/46 Schools (currently) Statewide System of Supports 3 Regions/74 Underperforming Schools Early Childhood Special Ed. Centers of Excellence 8 Regions/12 Professional Development Facilitators Teacher & Principal Development and Evaluation Pilot Models 35 Districts (to date)

12. SEA Work Participation Within the Agency Implementation Specialist in many divisions on the MDE Implementation Team (MIT) Linked teams within SEA in place Support from Executive Team and Directors Building MDE Capacity – For example…Implementation Forum held in March, 2014 (80 participants) – Shared 4 Examples Measuring Effective Implementation – of Mathematics Instruction Managing Implementation at a District Level Embedding Implementation Frameworks in the Q Comp Program Statewide Systems of Support (SSOS) – Building Implementation Infrastructure, Sustainability and Capacity at the State and School Level

13. SEA Work What's working well… Functional and sustained MDE Implementation Team (MIT) Increasing recognition of the importance of using implementation frameworks Fits with “way of work” in the department with increased emphasis on building capacity of district and school staff Increased emphasis on collaboration across divisions, activities aligned with broad outcomes, communication and informed decision making based on reflection before moving forward Best practices in implementation should yield improved outcomes – MN is seeing increases in student achievement; decreases in the achievement gap; increased graduation rates Directors are asking for representation from their division on the MIT Continue to embed implementation requirements in proposals; position descriptions; and division work (e.g., safety, health and nutrition)

14. SEA Work Challenges and goals… Need more and better ways of measuring implementation of key initiatives along the way – e.g., SSOS advocates track their activities Need more and better ways of measuring adult performance and fidelity of implementation. Need increased understanding of practice profiles and how to use them well How to institutionalize despite constant change at all levels in the SEA More time for staff and money specifically designated to supporting implementation work Transition of State Implementation Specialist and need for more than one individual in this role

15. Key is Building Capacity Is this familiar to you? Reflective Question # 1: Name the 5 implementation frameworks. Foundational Concepts of Implementation Core to All Initiatives at Levels education.state.mn.us 15

16. Interventions Active Implementation Frameworks Stages Drivers Cycles Teams

17. State-Level Capacity

18. State Capacity

20. Regional Work – Key Initiatives Incorporating Implementation Science Positive Behavioral Interventions and Supports –3 Regional training projects with over 483 schools to date (about 24% of schools) –First PBIS training for schools offered in 2005 –Two year training cycle, 9 training days walking schools through the stages –Implementation frameworks are embedded within PBIS training, coaching and evaluation –Demonstrated outcomes in schools and districts implementing with fidelity –Ongoing use of continuous improvement (PDSA cycles) to revise training, coaching and evaluation

21. Establishing a priority, setting aside funding and sending staff to professional development are all examples of activities at what stage of implementation? a)Exploration b)Installation c)Initial Implementation d)Full Implementation Reflective Question #2: Getting Started Working With Districts with Evidence of Commitment education.state.mn.us 21

22. District Work – Scaling Up PBIS District 1 - 10,000 + students; 17 schools (pre-K-Grd 12) –8 elementary, three middle level (6-8), two high schools (9-12), three alternative learning schools, an early childhood center and a community education center –The district has had schools and programs in PBIS cohort training since the fall of 2005. By last fall, all schools and programs have participated in the training, and the focus has turned to effective, sustained implementation. District 2 - 20,000 + students; 27 schools (pre-K-Grd 12) –17 elementary (PreK-6), four junior highs (7-9), three senior highs (10-12), an area learning center, two early childhood centers, two special program sites, and an adult education/enrollment center. –This district has been engaged in SW-PBIS since 2010 –Eight out of 29 schools have been through the PBIS cohort training, with three more entering in the fall of 2014.

23. District Work What Does It Look Like? –Team meetings (e.g., DLIT role and function, strong attendance and active participation) –Monthly meetings with MDE implementation team members and MDE staff with expertise in PBIS - collaboration –Use of data (e.g., data dashboard; stage based assessment; PBIS implementation data) –Administration and use of District Capacity Assessment to action plan and stimulate discussion –Working for generalization of implementation concepts to other EBPs in the district

24. Performance assessment refers to: a)Measuring how well students are doing (student outcomes) as a result of an evidence-based practice. b)Measuring how well staff report liking the results and usefulness of an evidence-based practice. c)Measuring the extent to which adults can implement the evidence-based practice as intended. d)Measuring the support of parents and stakeholders for selected evidence-based practices. Reflective Question #3: Using Performance Assessment Data! education.state.mn.us 24

25. Cohort 8 SET Results Fall 12-Spring 14

26. Tools and Agreements “Give and Get Chart” –Establishes expectations and shared accountability including developing a functional District Leadership and Implementation Team District Data Dashboard – views schools simultaneously on measures of performance assessment and student outcomes to guide next steps District Calendar for Implementation District Capacity Assessment (DCA) to determine progress in building infrastructure and capacity to scale up EBP District Work education.state.mn.us 26

27. St. Cloud and MDE Agreement District 1 and MDE Agreement to Actively Manage District-wide PBIS Implementation Priority District GetMDE & Partners GetDistrict GiveMDE & Partners GiveShared Outcomes Improve District- wide PBIS Implementation Integration of implementation frameworks and tools into District management of PBIS implementation district-wide (see http://implementation.fpg.unc.edu/ module-1 ) http://implementation.fpg.unc.edu/ module-1 Working model of district-wide PBIS continuous regeneration and sustained implementation using implementation frameworks and tools Staff time at District monthly at PBIS meetings to learn about implementation frameworks and tools associated with sustaining PBIS Staff time preparing for, participating in, and meeting needs emerging from St. Cloud district PBIS meetings The District, MDE and Partners understand what it takes to provide ongoing active management of PBIS implementation and innovation across a district Data-based decision making Access to customized PBIS district-wide implementation data dashboards from MDE and Wilder Research, including regular and ongoing effort, fidelity, and outcome data reports Experience learning which data are most helpful to guide districts in actively managing and sustaining PBIS implementation Agreement to share and review the districts PBIS implementation data on a regular basis at monthly meeting District-wide PBIS implementation data in an actionable format (within context of statewide district implementation data patterns) A district-wide data-based decision making process is developed to implement and sustain ongoing PBIS efforts in the district. Professional Development Knowledge about PBIS implementation through participation in a Community of Practice with other selected districts actively managing PBIS implementation MDE & Partner staffs learn from Community of Practice discussions and data review cycles about how best to meet the PBIS professional development needs of participating districts and schools Staff dedicated time to participate in the Community of Practice and other professional development opportunities provided by MDE & Partners Data-informed district-wide professional development (provided by MDE staff, George Sugai, Karen Blase, and others) and reserved space at MDE Implementation Forums and content-specific PBIS trainings or capacity building sessions (such as SWIS facilitator training, TIPS training, PBIS trainer training etc.) Increased competence of the district, MDE and Partner staff to implement, manage and evaluate district level PBIS Strategic PBIS meeting calendar and agenda planning Assistance with planning PBIS district meeting calendar and customize agenda items based on available data and associated implementation frameworks Feedback on the efficiency and effectiveness of pre-planning district PBIS meeting calendar and agendas Share responsibility for district PBIS meeting calendar and agenda planning with MDE staff Staff time to assist with pre- planning and meeting needs emerging from district PBIS meeting calendar and agendas. The district, MDE and partner staff develop a model for meeting calendar and agenda planning to improve the efficiency and effectiveness of PBIS implementation and scale-up Transfer Implementation Frameworks and tools to other St. Cloud initiatives Understanding of Implementation Frameworks, tools, and processes (such as the District Capacity Assessment) and how they can be used to implement other district-wide initiatives Experience in working with a district team to learn about Implementation Frameworks and tools to transfer those frameworks and tools to other district-level initiatives Staff time to explore how the Implementation Frameworks and tools used to actively manage PBIS implementation can generalize to other district initiatives Systems coaching to explicitly highlight when and how to apply Implementation Frameworks and tools to other district initiatives The District PBIS Team, MDE and Partners understand how to successfully use and transfer Implementation Frameworks and tools to other initiatives

28. District Capacity Assessment (DCA)

29. Example of a District Data Dashboard: Effort, Fidelity and Outcome Data education.state.mn.us 29

30. District Calendar for Implementation

31. Student Outcomes: Reductions in Suspensions

32. District Work: What’s Next? Continuing to use DCA data to focus district action planning Collection of student outcomes to review impact at schools who are implementing PBIS with fidelity Building toward consistent district-wide implementation and sustaining once we get there. Working to generalize implementation frameworks, processes, and tools to other areas of district-level implementation. Expansion to new districts….

33. Also…SPDG Partnerships! Cohort Districts: MTSS in Minneapolis and St. Paul  High rate of staff turnover has a strong impact on implementation.  Coaching helps strengthen and sustain implementation.  Reductions in Suspensions and ODRs experienced, but not necessarily changing disproportionality in the short term.  District networking provides a new source of support, but not necessarily a strong influence to change specific evidence- based practices.  Whole-school approaches assist in improving outcomes, since the majority of academic and behavioral instruction and interactions are happening in the context of general education.  Mapping of practices shows annual changes in the menu of academic and behavioral practices.

34. On the Horizon education.state.mn.us 34

35. Some signs… Shifting from dependence on SISEP staff for presentations and information to MDE staff for expertise Implementation information designed and delivered to be responsive to requests Development of the Implementation Specialist Practice Profile to establish defined role and function More formal induction, training, and mentorship and recognition of new MDE Implementation Team members and their role Ability to sustain across staff and team member change Closing – Will This Work Continue? Capacity to Regenerate, Grow, and Improve Implementation Infrastructure education.state.mn.us 35

36. Anywhere we measure implementation, it varies. Any school who measures implementation over time will experience variation. Anyone who measures of the implementation of multiple schools within a district will see variation. We measure implementation over time because ________________? Closing Thoughts education.state.mn.us 36

37. The Active Implementation Hub: http://implementation.fpg.unc.edu/about-the-ai-hub The Minnesota Department of Education: http://education.state.mn.us Cammy.Lehr@state.mn.us Eric.Kloos@state.mn.us Resources education.state.mn.us 37