Presentation on theme: "Investigating Program Sustainability Andrew Powers, Research Associate Amy L. Powers, Principal Program Evaluation and Educational Research (PEER) Associates,"— Presentation transcript:
Investigating Program Sustainability Andrew Powers, Research Associate Amy L. Powers, Principal Program Evaluation and Educational Research (PEER) Associates, Inc.
2 Rationale for this research… …informing stakeholders that an initiative has been designed well, or been successful is no longer enoughbecause quite often, after funding ends or staff leave, such programs can collapse. Therefore, the evaluator is being asked to take on a new role where the question is How is this program going to be sustainable in the future? (Harvey & Hurworth, 2006)
3 Components of this research Burlington School District interviews Burlington School District interviews all elementary principals (n=6) all elementary principals (n=6) superintendent (n=1), and superintendent (n=1), and selected teachers (n=4) selected teachers (n=4) Broad literature review on program sustainability (7 articles selected) Broad literature review on program sustainability (7 articles selected) Adopt-A-Watersheds Stages of Implementation tool Adopt-A-Watersheds Stages of Implementation tool
4 Adopt-A-Watershed (Earthwater) Developed a model for successful, sustainable implementation of PBE Professional Development of leadership teams is a major component Complete evaluation system to accompany model Stages of Implementation Describes 9 components of implementation model Describes 9 components of implementation model Details elements of each component and three stages of implementation for each element Details elements of each component and three stages of implementation for each element Kim Stokelys top two thoughts on program sustainability Leadership PD for teams, not individuals Leadership PD for teams, not individuals Work with individuals to integrate who we are and what we do soul and role Work with individuals to integrate who we are and what we do soul and role
5 Defining Program Sustainability As defined by Billig, Sherry, and Havelock (2005): If a program is sustainable… the innovation endures over time the innovation typically does not lose its identity the innovation typically does not lose its identity the innovation becomes supported as part of the culture of the institution
6 What does Program Sustainability look like in a school? Teachers continue to implement the program and think about how to improve it Implementation of the program continues without ongoing input from program staff Teachers experience changes to their thinking The program becomes part of the school culture The program influences hiring practice The program becomes part of the story coming out of that school (e.g. district annual reports may cite the program) Lasting relationships with community partners are established
7 Literature Review Various fields of school based initiatives To build a theoretical framework to put local research in context Non-exhaustive The Seven Articles: Whole-school approaches to sustainability: An international review of sustainable school programs Exploring program sustainability: Identifying factors in two educational initiatives in Victoria The Sustainability of inclusive school reform Challenge 98: Sustaining the work of a regional technology integration initiative Sustainability: Examining the survival of schools comprehensive school reform efforts Sustaining 21st century community learning centers Building capacity and sustainable prevention innovations: a sustainability planning model
8 Sustainability Factors Articles identified different factors that contribute to program sustainability Factors have been compiled into a matrix to show which factors are most frequently mentioned in literature
9 The Matrix
10 Sustainability Factors Reviewed articles identified different factors that contribute to program sustainability Factors have been compiled into a matrix to show which factors are most frequently mentioned in literature This is a non-rigorous analytical method! Creates a framework for discussion
11 The Sustainability Factors Matrix
12 Most Prominent Factors emerged as strongest themes in literature and interviews
13 Strong Leadership and Program Champions Strong leadership stimulates the development of a shared vision stimulates the development of a shared vision motivates action and allegiance to the purpose of the project motivates action and allegiance to the purpose of the project engenders a sense of community within the project engenders a sense of community within the project (Billig, Sherry, & Havelock 2005) Formal and informal leaders within adopting systems, as well as champions who proactively promote an innovation from inside or outside of a system, are critical to creating an environment that supports and facilitates sustaining innovations. Sindelar et al. Some things come and go, some things come and linger, and some things come and stay because someone here (staff, parents, administrators, etc.) is passionate about it. Its not that you need two days of training vs. one day, etc., its about who you have thats willing to put in the long-term commitment. -Principal
14 Adopt-a-Watershed (Earthwater) on Leadership Leadership teams are more sustainable than individual leaders Team comprised of educators, administrators, community partners Teams work regularly and collaboratively on tasks such as developing an understanding of stakeholder needs developing an understanding of stakeholder needs generating awareness of the program generating awareness of the program fostering community involvement fostering community involvement sharing results sharing results networking networking facilitating partnerships facilitating partnerships contributing to evaluation and reporting contributing to evaluation and reporting Teams do professional development together Champions flare out. A successful leadership team will continue to function while losing or changing members. A successful leadership team will continue to function while losing or changing members.
15 District/Administrative/Political support The literature frequently listed this as a key factor for program sustainability. Interviews showed that on the ground, this translated to TIME and MONEY. You need a high level of commitment from up above that is willing to put forward the time and money. It takes a lot to do it right, but it can be done. -Teacher Its money and time, but time is money. If you want to give teachers time, you need a sub[stitute], and for that you need money…. When youre trying to increase teacher capacity, you also need money. Thats the bottom line. -Principal
16 Partnerships with external agencies, organizations, businesses, etc. Strategic partnerships are critical elements to program design and programs need to develop strategies for expanding capacity and relationships of the partners involved. – Henderson & Tilbury 2004 Other noted elements of strong partnerships collaboration in program development collaboration in program development mutually beneficial mutually beneficial shared common vision shared common vision perform interdependent tasks perform interdependent tasks
17 Sustained Professional Development Comments from Burlington Principals This is my 30th year in schools and I cant tell you how many efforts Ive watched that were ineffective to create change. Its not until the last 7 or 8 years where the people who are our leaders have taken a big chunk of money towards professional development for teachers. They've brought in nationally known authors, or educators, nationally recognized experts, etc. Constant follow up during the year [is needed] to support the program with half day or full day workshop programs….Every year this stuff has to be offered if you want it to be sustainable, making sure that new teachers are brought in as well. So during the year, even if they cant keep getting national gurus, they get the local experts or the teachers that have gone in for extra training and then they can be the in-house experts.
18 Ongoing program resource support Program support needs to be varied and responsive to the local context needs. The products need to be aligned with the program goals and objectives as well as the professional development components. Support should be multi-layered and not just confined to resource kits and lesson materials. Evidence suggests that dedicated staff assigned to schools can oversee, facilitate and motivate staff to work towards deeper levels of change. Henderson & Tilbury 2004 Program Support offered by SSP curriculum planning assistance curriculum planning assistance meeting facilitation meeting facilitation community partner training community partner training assisting in the classroom assisting in the classroom providing physical teaching resources providing physical teaching resources organizing community events. organizing community events. At school, we dont say were working on capital letters today and then never revisit it. Real change happens with continued revisiting and support. – Principal
19 Monitoring and Evaluation More evaluation? Do we have to? Formative evaluation to improve program implementation Summative evaluation to measure and document outcomes Establishment of credibility At least 100 more benefits! Sometimes people get passionate about something thats not tried and true. Theres a big push and then a couple of months later, they just kind of fade and fail. - Principal Oops! Forgot to evaluate!
20 Strong and receptive institutions, coherent school culture Schools with shared vision and cultures of communication and shared decision making, and schools that involve teachers in the design of an innovation, are more likely to sustain innovations. Sindelar et al Senge (1999) conducted research on organizations that sustained practices over the long term and found that once the innovation produced visible results, the dynamics associated with the diffusion process played a determining role in sustainability. The organizations readiness and capacity for change were critical elements at this stage. These factors were found to be related to the permeability of organizational boundaries, communication, and the extent to which the organizational culture encouraged learning. Key to success was the ability of the innovators to capture lessons learned, best practices, and networks of support. Johnson et al. 2004
21 Other prominent factors Ability to be accommodated within existing organizational structures Curriculum, standards, staff meetings If it fits well with the curriculum, even if the passionate people move on, it will still stick, -Principal Consistency with teachers beliefs or teaching style …successful adoption of innovative practices occurred when it was consistent with teachers' beliefs or teaching style. (Sindelar et al. 2006) …when that money runs out, it will not be here anymore. It didnt fit the model of what [the teachers] had learned and bought into. -Principal on a 3 million dollar Reading First grant. Rewards and incentives New teaching skills, networking opportunities, positive feedback This may be crass, but when I want teachers to work in the summer, I pay em. It doesnt have to be much, but its an acknowledgement of their time. -Principal
22 Conclusions Continue retrospective evaluation at sites to further understand sustainability factors specific to the local context Develop forward looking program sustainability plans Identify most important sustainability factors at a site Identify most important sustainability factors at a site Tools for ongoing evaluation of these factors Tools for ongoing evaluation of these factors Actions to promote program sustainability Actions to promote program sustainability
23 Authors, Sources, and Abstracts p. 1 of 3 Article (1) Title: Whole-School Approaches to Sustainability: An International Review of Sustainable School Programs Authors: Henderson, K and Tilbury, D. Source: Report prepared by the Australian Research Institute in Education for Sustainability (ARIES) for The Department of the Environment and Heritage, Australian Government Abstract: Education for sustainability is an emerging concept encompassing a new vision of education that seeks to empower people of all ages to assume responsibility for creating a sustainable future (UNESCO, 2002). With the development of a number of national whole school initiatives, including in Australia, this report reviews, documents, and identifies lessons from some of these programs to inform future Sustainable Schools initiatives. Article (2) Title: Exploring program sustainability: identifying factors in two educational initiatives in Victoria. Authors: Harvey, G., & Hurworth R. Source: Evaluation Journal of Australasia, Vol. 6 (1), p , Abstract: This paper examines two recent successful school based health initiatives in Victoria, particularly in relation to factors that seem to foster program sustainability. These programs, dealing with drug education and healthy eating, are described before presenting two different methods (individual and group) used to determine elements that allow for the continuation of such projects. The findings on sustainability from each program are discussed using the broad areas of factors associated with the programs themselves; those associated with the context in which the programs were implemented; and finally, those factors external to the programs and their implementation contexts. These results indicate a strong congruence with factors identified in the literature but also highlight the influence of the use of change theory in strengthening sustainability approaches in program development as well as the need to focus on funding options in forward planning. The possible roles for evaluators in assisting program development and supporting the integration of factors supporting sustained use are also discussed.
24 Authors, Sources, and Abstracts p. 2 of 3 Article (3) Title: The sustainability of inclusive school reform Authors: Sindelar, P., Shearer, D., Yendol-Hoppey, D., and Liebert, W. Source: Exceptional Children, Vol. 72 (3), p , Abstract: For over a decade, University of Florida researchers worked with middle schools in a large urban and suburban south Florida district, as they developed and then worked to sustain inclusive reform. One middle school, Socrates, was notably successful, having built its inclusion model on a foundation of previous reform and a school culture characterized by shared decision making, collaboration, and teaming. For 4 years, we studied Socrates and the sustainability of its program. Inclusion was not sustained; our analysis of teacher and administrator interviews revealed three primary factors that help explain why: leadership change, teacher turnover, and state and district assessment policy change. Reduced support for the program, a by-product of the primary factors, also contributed to the lack of sustainability. Article (4) Title: Challenge 98: Sustaining the Work of a Regional Technology Integration Initiative Authors: Billig, S., Sherry, L., & Havelock, B. Source: British Journal of Educational Technology, Vol. 36, p , Abstract: In this article, we offer a research-based theoretical framework for sustainability, describing the proven qualities of a project and the innovations that support its sustained existence over time. We then describe how a US Department of Education Technology Innovation Challenge grantee, working to promote technology integration in a socio-economically disadvantaged region of the state of Texas, succeeded in creating a sustainable set of activities around its work to support educators' uses of technology. We examine the factors that served to nurture and facilitate sustainability of the practices associated with technology integration to promote student achievement. We take the fact that it is not the project but rather the change in practice that is important.
25 Authors, Sources, and Abstracts p. 3 of 3 Article (5) Title: Sustainability: Examining the Survival of Schools Comprehensive School Reform Efforts Author: Taylor, J. Source: Prepared for the annual meeting of the American Educational Research Association, Montreal, Canada, April 11–15, 2005 Abstract: One of the greatest challenges, if not the greatest challenge, to comprehensive school reform (CSR) is sustaining reform over a time period long enough to produce substantial effects. By examining how comprehensive school reformers complete their life course, this paper highlights the importance of studying sustainability as well as the importance of being clear about what is being sustained. It is critical to distinguish between a sustained reform relationship and sustained implementation of a reform. Article (6) Title: Sustaining 21st Century Community Learning Centers: What Works for Programs and How Policymakers Can Help Authors: Szekely, A., and Padgette, H. Source: A Finance Project Strategy Brief, 2006 Abstract: For nearly a decade, schools and communities across the country have implemented comprehensive out-of-school time programming with grants from the U.S. Department of Educations 21st Century Community Learning Centers (21CCLC) program. The only federal funding source dedicated exclusively to out-of-school time programs, 21CCLC supports tutoring, enrichment, and other services for low income children and their families. From the programs inception, 21CCLC grants have been largely used as seed grants for new programs; they were not intended to provide programs with long term funding. As the first rounds of state-administered grants expire, many schools and community partners are struggling to ensure the long-term sustainability of their out-of-school time programs. Through interviews with former and current 21CCLC grantees and state 21CCLC administrators, The Finance Project has learned about the challenges to sustainability and the keys to success. It became clear through these conversations that various factors both at the program level and in the administration of grants can help or hinder success with sustainability. This publication lays out the findings of the study and describes how both grantees and policymakers can promote the sustainability of 21CCLC programs. Article (7) Title: Building capacity and sustainable prevention innovations: a sustainability planning model Authors: Johnson, K., Hays, C., Center, H., and Daley, C. Source: Evaluation and Program Planning 27, p , Abstract: This article presents an informed definition of sustainability and an associated planning model for sustaining innovations (pertinent to both infrastructure and interventions) within organizational, community, and state systems. The planning model stems from a systematic review of the literature and from concepts derived from a series of think tanks made up of key substance abuse prevention professionals. The model assumes a five-step process (i.e. assessment, development, implementation, evaluation, and reassessment/modification) and addresses factors known to inhibit efforts to sustain an innovation. One set of factors concerns the capacity of prevention systems to support sustainable innovations. The other pertains to the extent to which a particular innovation is sustainable. A sustainability action strategy is presented that includes goals with corresponding sets of objectives, actions, and results that determine the extent of readiness to sustain an innovation. Sustainability tools to assist in implementing the planning model are illustrated, and next steps for the model are discussed. This planning model provides a conceptual and practical understanding of sustainability that can lead to further investigation.
26 Investigating Program Sustainability Andrew Powers, Research Associate Amy L. Powers, Principal Program Evaluation and Educational Research (PEER) Associates, Inc.