Presentation is loading. Please wait.

Presentation is loading. Please wait.

Investigating Program Sustainability

Similar presentations


Presentation on theme: "Investigating Program Sustainability"— Presentation transcript:

1 Investigating Program Sustainability
January 17, 2005 Investigating Program Sustainability America’s greatest anxiety? Fear of public speaking. Jerry seinfeld said that at a funeral, most people would rather be in a casket than giving the eulogy. On that note, welcome to an informal presentation on an informal report about program sustainability. Let’s begin Andrew Powers, Research Associate Amy L. Powers, Principal Program Evaluation and Educational Research (PEER) Associates, Inc. NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

2 Rationale for this research…
January 17, 2005 Rationale for this research… “…informing stakeholders that an initiative has been designed well, or been successful is no longer enough—because quite often, after funding ends or staff leave, such programs can collapse. Therefore, the evaluator is being asked to take on a new role where the question is ‘How is this program going to be sustainable in the future?’” (Harvey & Hurworth, 2006) Why did we do this? After a number of consecutive years of outcomes evaluation, SSP looking at other ways to use us as evaluators and researchers I don’t think you will find the results surprising, as your programs are based on Research, experience, best practice, etc. Successful progs tend to be Sustainable programs NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

3 Components of this research
January 17, 2005 Components of this research Burlington School District interviews all elementary principals (n=6) superintendent (n=1), and selected teachers (n=4) Broad literature review on program sustainability (7 articles selected) Adopt-A-Watershed’s Stages of Implementation tool Interviews conducted by a consultant for SSP I was lit review guy Last one is bonus came from EPA meeting NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

4 Adopt-A-Watershed (Earthwater)
January 17, 2005 Adopt-A-Watershed (Earthwater) Developed a model for successful, sustainable implementation of PBE Professional Development of leadership teams is a major component Complete evaluation system to accompany model Stages of Implementation Describes 9 components of implementation model Details elements of each component and three stages of implementation for each element Kim Stokely’s top two thoughts on program sustainability Leadership PD for teams, not individuals Work with individuals to integrate “who we are and what we do” “soul and role” Many of you met Kim Stokely at EPA. Founder now ed director of AAW/Earthwater. 19 yrs exp etc. Anyone familiar with her work. Have SOI here. NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

5 Defining Program Sustainability
January 17, 2005 Defining Program Sustainability As defined by Billig, Sherry, and Havelock (2005): If a program is sustainable… the innovation endures over time the innovation typically does not lose its identity  the innovation becomes supported as part of the culture of the institution NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

6 What does Program Sustainability look like in a school?
January 17, 2005 What does Program Sustainability look like in a school? Teachers continue to implement the program and think about how to improve it Implementation of the program continues without ongoing input from program staff Teachers experience changes to their thinking The program becomes part of the school culture The program influences hiring practice The program becomes part of the “story” coming out of that school (e.g. district annual reports may cite the program) Lasting relationships with community partners are established Examples from past evaluation reports Used in interview guide to illustrate NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

7 Literature Review Various fields of school based initiatives
January 17, 2005 Literature Review Various fields of school based initiatives To build a theoretical framework to put local research in context Non-exhaustive The Seven Articles: Whole-school approaches to sustainability: An international review of sustainable school programs Exploring program sustainability: Identifying factors in two educational initiatives in Victoria The Sustainability of inclusive school reform Challenge 98: Sustaining the work of a regional technology integration initiative Sustainability: Examining the survival of schools’ comprehensive school reform efforts Sustaining 21st century community learning centers Building capacity and sustainable prevention innovations: a sustainability planning model These are the articles to give you a sense. If anyone reads broadly in the field they may have come across Will cite by authors NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

8 Sustainability Factors
January 17, 2005 Sustainability Factors Articles identified different factors that contribute to program sustainability Factors have been compiled into a matrix to show which factors are most frequently mentioned in literature Many articles conducted lit reviews in sust research Some simply listed factors My lit review is meta lit review NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

9 The Matrix January 17, 2005 Oops, wrong matrix…
NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

10 Sustainability Factors
January 17, 2005 Sustainability Factors Reviewed articles identified different factors that contribute to program sustainability Factors have been compiled into a matrix to show which factors are most frequently mentioned in literature This is a non-rigorous analytical method! Creates a framework for discussion Not a scientfic method of proving which are the most important factors NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

11 The Sustainability Factors Matrix
January 17, 2005 Here’s the matrix of most of the factors that came up. Number refer to the articles, cited that way in the report. Informal report not full of billig, citing billig referring to huberman… Before you get too involved in this, let me simplify NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

12 January 17, 2005 Most Prominent Factors emerged as strongest themes in literature and interviews I have done some aggregating and consolidation of factors Again, not offered as a product of analysis, but to provide framework for discussion of sust NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

13 Strong Leadership and Program Champions
January 17, 2005 Strong Leadership and Program Champions Strong leadership stimulates the development of a shared vision motivates action and allegiance to the purpose of the project engenders a sense of community within the project (Billig, Sherry, & Havelock 2005) “Formal and informal leaders within adopting systems, as well as champions who proactively promote an innovation from inside or outside of a system, are critical to creating an environment that supports and facilitates sustaining innovations.” Sindelar et al. “Some things come and go, some things come and linger, and some things come and stay because someone here (staff, parents, administrators, etc.) is passionate about it. It’s not that you need two days of training vs. one day, etc., it’s about who you have that’s willing to put in the long-term commitment Principal Program champions are often the “early adopters” that Amy talked about Examples of Tewks, etc, for SSP. Teachers who when they talk it is “her mouth to Fred Bay’s ear” NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

14 Adopt-a-Watershed (Earthwater) on Leadership
January 17, 2005 Adopt-a-Watershed (Earthwater) on Leadership Leadership teams are more sustainable than individual leaders Team comprised of educators, administrators, community partners Teams work regularly and collaboratively on tasks such as developing an understanding of stakeholder needs generating awareness of the program fostering community involvement sharing results networking facilitating partnerships contributing to evaluation and reporting Teams do professional development together “Champions flare out.” A successful leadership team will continue to function while losing or changing members. As mentioned KS, very strong on leadership teams One of the big things they do is offer PD for teams. Have brochure here if curious. Not promoting Like CO-SEED’s seed team NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

15 District/Administrative/Political support
January 17, 2005 District/Administrative/Political support The literature frequently listed this as a key factor for program sustainability. Interviews showed that on the ground, this translated to TIME and MONEY. “You need a high level of commitment from up above that is willing to put forward the time and money. It takes a lot to do it right, but it can be done.” -Teacher “It’s money and time, but time is money. If you want to give teachers time, you need a sub[stitute], and for that you need money…. When you’re trying to increase teacher capacity, you also need money. That’s the bottom line.” -Principal NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

16 Partnerships with external agencies, organizations, businesses, etc.
January 17, 2005 Partnerships with external agencies, organizations, businesses, etc. “Strategic partnerships are critical elements to program design and programs need to develop strategies for expanding capacity and relationships of the partners involved.” – Henderson & Tilbury 2004 Other noted elements of strong partnerships collaboration in program development mutually beneficial shared common vision perform interdependent tasks An overarching concept not made explicit here is “working with the community” Interdependent tasks “each needs what the other knows” ex SSP report cards NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

17 Sustained Professional Development
January 17, 2005 Sustained Professional Development Comments from Burlington Principals “This is my 30th year in schools and I can’t tell you how many efforts I’ve watched that were ineffective to create change. It’s not until the last 7 or 8 years where the people who are our leaders have taken a big chunk of money towards professional development for teachers. They've brought in nationally known authors, or educators, nationally recognized experts, etc.” “Constant follow up during the year [is needed] to support the program with half day or full day workshop programs….Every year this stuff has to be offered if you want it to be sustainable, making sure that new teachers are brought in as well. So during the year, even if they can’t keep getting national gurus, they get the local experts or the teachers that have gone in for extra training and then they can be the in-house experts. Again, strong support in the lit and the Earthwater model for PD NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

18 Ongoing program resource support
January 17, 2005 Ongoing program resource support “Program support needs to be varied and responsive to the local context needs. The products need to be aligned with the program goals and objectives as well as the professional development components. Support should be multi-layered and not just confined to resource kits and lesson materials. Evidence suggests that dedicated staff assigned to schools can oversee, facilitate and motivate staff to work towards deeper levels of change.” Henderson & Tilbury 2004 Program Support offered by SSP curriculum planning assistance meeting facilitation community partner training assisting in the classroom providing physical teaching resources organizing community events. “At school, we don’t say we’re working on capital letters today and then never revisit it. Real change happens with continued revisiting and support.” – Principal Time frame a consideration What to input ongoing once major intervention is done? NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

19 Monitoring and Evaluation
January 17, 2005 Monitoring and Evaluation More evaluation? Do we have to? Formative evaluation to improve program implementation Summative evaluation to measure and document outcomes Establishment of credibility At least 100 more benefits! “Sometimes people get passionate about something that’s not tried and true. There’s a big push and then a couple of months later, they just kind of fade and fail.” - Principal Oops! Forgot to evaluate! NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

20 Strong and receptive institutions, coherent school culture
January 17, 2005 Strong and receptive institutions, coherent school culture “Schools with shared vision and cultures of communication and shared decision making, and schools that involve teachers in the design of an innovation, are more likely to sustain innovations.” Sindelar et al. 2006 “Senge (1999) conducted research on organizations that sustained practices over the long term and found that once the innovation produced visible results, the dynamics associated with the diffusion process played a determining role in sustainability. The organization’s readiness and capacity for change were critical elements at this stage. These factors were found to be related to the permeability of organizational boundaries, communication, and the extent to which the organizational culture encouraged learning. Key to success was the ability of the innovators to capture lessons learned, best practices, and networks of support.” Johnson et al. 2004 First thought is that this is a contextual factor Maybe in a site selection rubric SSP evaluation has seen that SSP has helped strengthen school culture Senge presenting the fifth discipline in Dubai using shadow puppets. That guy knows how to consult! NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

21 Other prominent factors
January 17, 2005 Other prominent factors Ability to be accommodated within existing organizational structures Curriculum, standards, staff meetings “If it fits well with the curriculum, even if the passionate people move on, it will still stick,” -Principal Consistency with teacher’s beliefs or teaching style “…successful adoption of innovative practices occurred when it was consistent with teachers' beliefs or teaching style.” (Sindelar et al. 2006) “…when that money runs out, it will not be here anymore. It didn’t fit the model of what [the teachers] had learned and bought into.” -Principal on a 3 million dollar Reading First grant. Rewards and incentives New teaching skills, networking opportunities, positive feedback “This may be crass, but when I want teachers to work in the summer, I pay ‘em. It doesn’t have to be much, but it’s an acknowledgement of their time.” -Principal NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

22 January 17, 2005 Conclusions Continue retrospective evaluation at sites to further understand sustainability factors specific to the local context Develop forward looking program sustainability plans Identify most important sustainability factors at a site Tools for ongoing evaluation of these factors Actions to promote program sustainability As suggested in early slide, a role of evaluation now is to inform program sust. NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms

23 Authors, Sources, and Abstracts p. 1 of 3
Article (1) Title: Whole-School Approaches to Sustainability: An International Review of Sustainable School Programs Authors: Henderson, K and Tilbury, D. Source: Report prepared by the Australian Research Institute in Education for Sustainability (ARIES) for The Department of the Environment and Heritage, Australian Government Abstract: Education for sustainability is an emerging concept encompassing “a new vision of education that seeks to empower people of all ages to assume responsibility for creating a sustainable future” (UNESCO, 2002). With the development of a number of national whole school initiatives, including in Australia, this report reviews, documents, and identifies lessons from some of these programs to inform future Sustainable Schools initiatives. Article (2) Title: Exploring program sustainability: identifying factors in two educational initiatives in Victoria. Authors: Harvey, G., & Hurworth R. Source: Evaluation Journal of Australasia, Vol. 6 (1), p , 2006. This paper examines two recent successful school based health initiatives in Victoria, particularly in relation to factors that seem to foster program sustainability. These programs, dealing with drug education and healthy eating, are described before presenting two different methods (individual and group) used to determine elements that allow for the continuation of such projects. The findings on sustainability from each program are discussed using the broad areas of factors associated with the programs themselves; those associated with the context in which the programs were implemented; and finally, those factors external to the programs and their implementation contexts. These results indicate a strong congruence with factors identified in the literature but also highlight the influence of the use of change theory in strengthening sustainability approaches in program development as well as the need to focus on funding options in forward planning. The possible roles for evaluators in assisting program development and supporting the integration of factors supporting sustained use are also discussed.

24 Authors, Sources, and Abstracts p. 2 of 3
Article (3) Title: The sustainability of inclusive school reform Authors: Sindelar, P., Shearer, D., Yendol-Hoppey, D., and Liebert, W. Source: Exceptional Children, Vol. 72 (3), p , 2006. Abstract: For over a decade, University of Florida researchers worked with middle schools in a large urban and suburban south Florida district, as they developed and then worked to sustain inclusive reform. One middle school, Socrates, was notably successful, having built its inclusion model on a foundation of previous reform and a school culture characterized by shared decision making, collaboration, and teaming. For 4 years, we studied Socrates and the sustainability of its program. Inclusion was not sustained; our analysis of teacher and administrator interviews revealed three primary factors that help explain why: leadership change, teacher turnover, and state and district assessment policy change. Reduced support for the program, a by-product of the primary factors, also contributed to the lack of sustainability. Article (4) Title: Challenge 98: Sustaining the Work of a Regional Technology Integration Initiative Authors: Billig, S., Sherry, L., & Havelock, B. Source: British Journal of Educational Technology, Vol. 36, p , 2005. In this article, we offer a research-based theoretical framework for sustainability, describing the proven qualities of a project and the innovations that support its sustained existence over time. We then describe how a US Department of Education Technology Innovation Challenge grantee, working to promote technology integration in a socio-economically disadvantaged region of the state of Texas, succeeded in creating a sustainable set of activities around its work to support educators' uses of technology. We examine the factors that served to nurture and facilitate sustainability of the practices associated with technology integration to promote student achievement. We take the fact that it is not the project but rather the change in practice that is important.

25 Authors, Sources, and Abstracts p. 3 of 3
Article (5) Title: Sustainability: Examining the Survival of Schools’ Comprehensive School Reform Efforts Author: Taylor, J. Source: Prepared for the annual meeting of the American Educational Research Association, Montreal, Canada, April 11–15, 2005 Abstract: One of the greatest challenges, if not the greatest challenge, to comprehensive school reform (CSR) is sustaining reform over a time period long enough to produce substantial effects. By examining how comprehensive school reformers complete their life course, this paper highlights the importance of studying sustainability as well as the importance of being clear about what is being sustained. It is critical to distinguish between a sustained reform relationship and sustained implementation of a reform. Article (6) Title: Sustaining 21st Century Community Learning Centers: What Works for Programs and How Policymakers Can Help Authors: Szekely, A., and Padgette, H. Source: A Finance Project Strategy Brief, 2006 For nearly a decade, schools and communities across the country have implemented comprehensive out-of-school time programming with grants from the U.S. Department of Education’s 21st Century Community Learning Centers (21CCLC) program. The only federal funding source dedicated exclusively to out-of-school time programs, 21CCLC supports tutoring, enrichment, and other services for low income children and their families. From the program’s inception, 21CCLC grants have been largely used as seed grants for new programs; they were not intended to provide programs with long term funding. As the first rounds of state-administered grants expire, many schools and community partners are struggling to ensure the long-term sustainability of their out-of-school time programs. Through interviews with former and current 21CCLC grantees and state 21CCLC administrators, The Finance Project has learned about the challenges to sustainability and the keys to success. It became clear through these conversations that various factors both at the program level and in the administration of grants can help or hinder success with sustainability. This publication lays out the findings of the study and describes how both grantees and policymakers can promote the sustainability of 21CCLC programs. Article (7) Title: Building capacity and sustainable prevention innovations: a sustainability planning model Authors: Johnson, K., Hays, C., Center, H., and Daley, C. Source: Evaluation and Program Planning 27, p , 2004. This article presents an informed definition of sustainability and an associated planning model for sustaining innovations (pertinent to both infrastructure and interventions) within organizational, community, and state systems. The planning model stems from a systematic review of the literature and from concepts derived from a series of ‘think tanks’ made up of key substance abuse prevention professionals. The model assumes a five-step process (i.e. assessment, development, implementation, evaluation, and reassessment/modification) and addresses factors known to inhibit efforts to sustain an innovation. One set of factors concerns the capacity of prevention systems to support sustainable innovations. The other pertains to the extent to which a particular innovation is sustainable. A sustainability action strategy is presented that includes goals with corresponding sets of objectives, actions, and results that determine the extent of readiness to sustain an innovation. Sustainability tools to assist in implementing the planning model are illustrated, and next steps for the model are discussed. This planning model provides a conceptual and practical understanding of sustainability that can lead to further investigation.

26 Investigating Program Sustainability
January 17, 2005 Investigating Program Sustainability Andrew Powers, Research Associate Amy L. Powers, Principal Program Evaluation and Educational Research (PEER) Associates, Inc. NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms


Download ppt "Investigating Program Sustainability"

Similar presentations


Ads by Google