Presentation on theme: "Investigating Program Sustainability"— Presentation transcript:
1 Investigating Program Sustainability January 17, 2005Investigating Program SustainabilityAmerica’s greatest anxiety? Fear of public speaking. Jerry seinfeld said that at a funeral, most people would rather be in a casket than giving the eulogy. On that note, welcome to an informal presentation on an informal report about program sustainability. Let’s beginAndrew Powers, Research AssociateAmy L. Powers, PrincipalProgram Evaluation and Educational Research(PEER) Associates, Inc.NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
2 Rationale for this research… January 17, 2005Rationale for this research…“…informing stakeholders that an initiative has been designed well, or been successful is no longer enough—because quite often, after funding ends or staff leave, such programs can collapse.Therefore, the evaluator is being asked to take on a new role where the question is‘How is this program going to be sustainable in the future?’”(Harvey & Hurworth, 2006)Why did we do this? After a number of consecutive years of outcomes evaluation, SSP looking at other ways to use us as evaluators and researchersI don’t think you will find the results surprising, as your programs are based onResearch, experience, best practice, etc.Successful progs tend to be Sustainable programsNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
3 Components of this research January 17, 2005Components of this researchBurlington School District interviewsall elementary principals (n=6)superintendent (n=1), andselected teachers (n=4)Broad literature review on program sustainability (7 articles selected)Adopt-A-Watershed’s Stages of Implementation toolInterviews conducted by a consultant for SSPI was lit review guyLast one is bonus came from EPA meetingNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
4 Adopt-A-Watershed (Earthwater) January 17, 2005Adopt-A-Watershed (Earthwater)Developed a model for successful, sustainable implementation of PBEProfessional Development of leadership teams is a major componentComplete evaluation system to accompany modelStages of ImplementationDescribes 9 components of implementation modelDetails elements of each component and three stages of implementation for each elementKim Stokely’s top two thoughts on program sustainabilityLeadership PD for teams, not individualsWork with individuals to integrate “who we are and what we do” “soul and role”Many of you met Kim Stokely at EPA. Founder now ed director of AAW/Earthwater.19 yrs exp etc.Anyone familiar with her work. Have SOI here.NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
5 Defining Program Sustainability January 17, 2005Defining Program SustainabilityAs defined by Billig, Sherry, and Havelock (2005):If a program is sustainable…the innovation endures over timethe innovation typically does not lose its identity the innovation becomes supported as part of the culture of the institutionNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
6 What does Program Sustainability look like in a school? January 17, 2005What does Program Sustainability look like in a school?Teachers continue to implement the program and think about how to improve itImplementation of the program continues without ongoing input from program staffTeachers experience changes to their thinkingThe program becomes part of the school cultureThe program influences hiring practiceThe program becomes part of the “story” coming out of that school (e.g. district annual reports may cite the program)Lasting relationships with community partners are establishedExamples from past evaluation reportsUsed in interview guide to illustrateNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
7 Literature Review Various fields of school based initiatives January 17, 2005Literature ReviewVarious fields of school based initiativesTo build a theoretical framework to put local research in contextNon-exhaustiveThe Seven Articles:Whole-school approaches to sustainability: An international review of sustainable school programsExploring program sustainability: Identifying factors in two educational initiatives in VictoriaThe Sustainability of inclusive school reformChallenge 98: Sustaining the work of a regional technology integration initiativeSustainability: Examining the survival of schools’ comprehensive school reform effortsSustaining 21st century community learning centersBuilding capacity and sustainable prevention innovations: a sustainability planning modelThese are the articles to give you a sense.If anyone reads broadly in the field they may have come acrossWill cite by authorsNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
8 Sustainability Factors January 17, 2005Sustainability FactorsArticles identified different factors that contribute to program sustainabilityFactors have been compiled into a matrix to show which factors are most frequently mentioned in literatureMany articles conducted lit reviews in sust researchSome simply listed factorsMy lit review is meta lit reviewNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
9 The Matrix January 17, 2005 Oops, wrong matrix… NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
10 Sustainability Factors January 17, 2005Sustainability FactorsReviewed articles identified different factors that contribute to program sustainabilityFactors have been compiled into a matrix to show which factors are most frequently mentioned in literatureThis is a non-rigorous analytical method!Creates a framework for discussionNot a scientfic method of proving which are the most important factorsNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
11 The Sustainability Factors Matrix January 17, 2005Here’s the matrix of most of the factors that came up.Number refer to the articles, cited that way in the report.Informal report not full of billig, citing billig referring to huberman…Before you get too involved in this, let me simplifyNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
12 January 17, 2005Most Prominent Factors emerged as strongest themes in literature and interviewsI have done some aggregating and consolidation of factorsAgain, not offered as a product of analysis, but to provide framework for discussion of sustNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
13 Strong Leadership and Program Champions January 17, 2005Strong Leadership and Program ChampionsStrong leadershipstimulates the development of a shared visionmotivates action and allegiance to the purpose of the projectengenders a sense of community within the project(Billig, Sherry, & Havelock 2005)“Formal and informal leaders within adopting systems, as well as champions who proactively promote an innovation from inside or outside of a system, are critical to creating an environment that supports and facilitates sustaining innovations.” Sindelar et al.“Some things come and go, some things come and linger, and some things come and stay because someone here (staff, parents, administrators, etc.) is passionate about it. It’s not that you need two days of training vs. one day, etc., it’s about who you have that’s willing to put in the long-term commitment PrincipalProgram champions are often the “early adopters” that Amy talked aboutExamples of Tewks, etc, for SSP.Teachers who when they talk it is “her mouth to Fred Bay’s ear”NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
14 Adopt-a-Watershed (Earthwater) on Leadership January 17, 2005Adopt-a-Watershed (Earthwater) on LeadershipLeadership teams are more sustainable than individual leadersTeam comprised of educators, administrators, community partnersTeams work regularly and collaboratively on tasks such asdeveloping an understanding of stakeholder needsgenerating awareness of the programfostering community involvementsharing resultsnetworkingfacilitating partnershipscontributing to evaluation and reportingTeams do professional development together“Champions flare out.”A successful leadership team will continue to function while losing or changing members.As mentioned KS, very strong on leadership teamsOne of the big things they do is offer PD for teams. Have brochure here if curious. Not promotingLike CO-SEED’s seed teamNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
15 District/Administrative/Political support January 17, 2005District/Administrative/Political supportThe literature frequently listed this as a key factor for program sustainability.Interviews showed that on the ground, this translated to TIME and MONEY.“You need a high level of commitment from up above that is willing to put forward the time and money. It takes a lot to do it right, but it can be done.”-Teacher“It’s money and time, but time is money. If you want to give teachers time, you need a sub[stitute], and for that you need money…. When you’re trying to increase teacher capacity, you also need money. That’s the bottom line.”-PrincipalNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
16 Partnerships with external agencies, organizations, businesses, etc. January 17, 2005Partnerships with external agencies, organizations, businesses, etc.“Strategic partnerships are critical elements to program design and programs need to develop strategies for expanding capacity and relationships of the partners involved.” – Henderson & Tilbury 2004Other noted elements of strong partnershipscollaboration in program developmentmutually beneficialshared common visionperform interdependent tasksAn overarching concept not made explicit here is “working with the community”Interdependent tasks “each needs what the other knows” ex SSP report cardsNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
17 Sustained Professional Development January 17, 2005Sustained Professional DevelopmentComments from Burlington Principals“This is my 30th year in schools and I can’t tell you how many efforts I’ve watched that were ineffective to create change. It’s not until the last 7 or 8 years where the people who are our leaders have taken a big chunk of money towards professional development for teachers. They've brought in nationally known authors, or educators, nationally recognized experts, etc.”“Constant follow up during the year [is needed] to support the program with half day or full day workshop programs….Every year this stuff has to be offered if you want it to be sustainable, making sure that new teachers are brought in as well. So during the year, even if they can’t keep getting national gurus, they get the local experts or the teachers that have gone in for extra training and then they can be the in-house experts.Again, strong support in the lit and the Earthwater model for PDNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
18 Ongoing program resource support January 17, 2005Ongoing program resource support“Program support needs to be varied and responsive to the local context needs. The products need to be aligned with the program goals and objectives as well as the professional development components. Support should be multi-layered and not just confined to resource kits and lesson materials. Evidence suggests that dedicated staff assigned to schools can oversee, facilitate and motivate staff to work towards deeper levels of change.” Henderson & Tilbury 2004Program Support offered by SSPcurriculum planning assistancemeeting facilitationcommunity partner trainingassisting in the classroomproviding physical teaching resourcesorganizing community events.“At school, we don’t say we’re working on capital letters today and then never revisit it. Real change happens with continued revisiting and support.” – PrincipalTime frame a considerationWhat to input ongoing once major intervention is done?NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
19 Monitoring and Evaluation January 17, 2005Monitoring and EvaluationMore evaluation? Do we have to?Formative evaluation to improve program implementationSummative evaluation to measure and document outcomesEstablishment of credibilityAt least 100 more benefits!“Sometimes people get passionate about something that’s not tried and true. There’s a big push and then a couple of months later, they just kind of fade and fail.” - PrincipalOops! Forgot to evaluate!NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
20 Strong and receptive institutions, coherent school culture January 17, 2005Strong and receptive institutions, coherent school culture“Schools with shared vision and cultures of communication and shared decision making, and schools that involve teachers in the design of an innovation, are more likely to sustain innovations.” Sindelar et al. 2006“Senge (1999) conducted research on organizations that sustained practices over the long term and found that once the innovation produced visible results, the dynamics associated with the diffusion process played a determining role in sustainability.The organization’s readiness and capacity for change were critical elements at this stage. These factors were found to be related to the permeability of organizational boundaries, communication, and the extent to which the organizational culture encouraged learning. Key to success was the ability of the innovators to capture lessons learned, best practices, and networks of support.”Johnson et al. 2004First thought is that this is a contextual factorMaybe in a site selection rubricSSP evaluation has seen that SSP has helped strengthen school cultureSenge presenting the fifth discipline in Dubai using shadow puppets.That guy knows how to consult!NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
21 Other prominent factors January 17, 2005Other prominent factorsAbility to be accommodated within existing organizational structuresCurriculum, standards, staff meetings“If it fits well with the curriculum, even if the passionate people move on, it will still stick,” -PrincipalConsistency with teacher’s beliefs or teaching style“…successful adoption of innovative practices occurred when it was consistent with teachers' beliefs or teaching style.” (Sindelar et al. 2006)“…when that money runs out, it will not be here anymore. It didn’t fit the model of what [the teachers] had learned and bought into.”-Principal on a 3 million dollar Reading First grant.Rewards and incentivesNew teaching skills, networking opportunities, positive feedback“This may be crass, but when I want teachers to work in the summer, I pay ‘em. It doesn’t have to be much, but it’s an acknowledgement of their time.” -PrincipalNPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
22 January 17, 2005ConclusionsContinue retrospective evaluation at sites to further understand sustainability factors specific to the local contextDevelop forward looking program sustainability plansIdentify most important sustainability factors at a siteTools for ongoing evaluation of these factorsActions to promote program sustainabilityAs suggested in early slide, a role of evaluation now is to inform program sust.NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
23 Authors, Sources, and Abstracts p. 1 of 3 Article (1)Title: Whole-School Approaches to Sustainability: An International Review of Sustainable School ProgramsAuthors: Henderson, K and Tilbury, D.Source: Report prepared by the Australian Research Institute in Education for Sustainability (ARIES) for The Department of the Environment and Heritage, Australian GovernmentAbstract:Education for sustainability is an emerging concept encompassing “a new vision of education that seeks to empower people of all ages to assume responsibility for creating a sustainable future” (UNESCO, 2002). With the development of a number of national whole school initiatives, including in Australia, this report reviews, documents, and identifies lessons from some of these programs to inform future Sustainable Schools initiatives.Article (2)Title: Exploring program sustainability: identifying factors in two educational initiatives in Victoria.Authors: Harvey, G., & Hurworth R.Source: Evaluation Journal of Australasia, Vol. 6 (1), p , 2006.This paper examines two recent successful school based health initiatives in Victoria, particularly in relation to factors that seem to foster program sustainability. These programs, dealing with drug education and healthy eating, are described before presenting two different methods (individual and group) used to determine elements that allow for the continuation of such projects. The findings on sustainability from each program are discussed using the broad areas of factors associated with the programs themselves; those associated with the context in which the programs were implemented; and finally, those factors external to the programs and their implementation contexts. These results indicate a strong congruence with factors identified in the literature but also highlight the influence of the use of change theory in strengthening sustainability approaches in program development as well as the need to focus on funding options in forward planning. The possible roles for evaluators in assisting program development and supporting the integration of factors supporting sustained use are also discussed.
24 Authors, Sources, and Abstracts p. 2 of 3 Article (3)Title: The sustainability of inclusive school reformAuthors: Sindelar, P., Shearer, D., Yendol-Hoppey, D., and Liebert, W.Source: Exceptional Children, Vol. 72 (3), p , 2006.Abstract:For over a decade, University of Florida researchers worked with middle schools in a large urban and suburban south Florida district, as they developed and then worked to sustain inclusive reform. One middle school, Socrates, was notably successful, having built its inclusion model on a foundation of previous reform and a school culture characterized by shared decision making, collaboration, and teaming. For 4 years, we studied Socrates and the sustainability of its program. Inclusion was not sustained; our analysis of teacher and administrator interviews revealed three primary factors that help explain why: leadership change, teacher turnover, and state and district assessment policy change. Reduced support for the program, a by-product of the primary factors, also contributed to the lack of sustainability.Article (4)Title: Challenge 98: Sustaining the Work of a Regional Technology Integration InitiativeAuthors: Billig, S., Sherry, L., & Havelock, B.Source: British Journal of Educational Technology, Vol. 36, p , 2005.In this article, we offer a research-based theoretical framework for sustainability, describing the proven qualities of a project and the innovations that support its sustained existence over time. We then describe how a US Department of Education Technology Innovation Challenge grantee, working to promote technology integration in a socio-economically disadvantaged region of the state of Texas, succeeded in creating a sustainable set of activities around its work to support educators' uses of technology. We examine the factors that served to nurture and facilitate sustainability of the practices associated with technology integration to promote student achievement. We take the fact that it is not the project but rather the change in practice that is important.
25 Authors, Sources, and Abstracts p. 3 of 3 Article (5)Title: Sustainability: Examining the Survival of Schools’ Comprehensive School Reform EffortsAuthor: Taylor, J.Source: Prepared for the annual meeting of the American Educational Research Association, Montreal, Canada, April 11–15, 2005Abstract:One of the greatest challenges, if not the greatest challenge, to comprehensive school reform (CSR) is sustaining reform over a time period long enough to produce substantial effects. By examining how comprehensive school reformers complete their life course, this paper highlights the importance of studying sustainability as well as the importance of being clear about what is being sustained. It is critical to distinguish between a sustained reform relationship and sustained implementation of a reform.Article (6)Title: Sustaining 21st Century Community Learning Centers: What Works for Programs andHow Policymakers Can HelpAuthors: Szekely, A., and Padgette, H.Source: A Finance Project Strategy Brief, 2006For nearly a decade, schools and communities across the country have implemented comprehensive out-of-school time programming with grants from the U.S. Department of Education’s 21st Century Community Learning Centers (21CCLC) program. The only federal funding source dedicated exclusively to out-of-school time programs, 21CCLC supports tutoring, enrichment, and other services for low income children and their families. From the program’s inception, 21CCLC grants have been largely used as seed grants for new programs; they were not intended to provide programs with long term funding. As the first rounds of state-administered grants expire, many schools and community partners are struggling to ensure the long-term sustainability of their out-of-school time programs. Through interviews with former and current 21CCLC grantees and state 21CCLC administrators, The Finance Project has learned about the challenges to sustainability and the keys to success. It became clear through these conversations that various factors both at the program level and in the administration of grants can help or hinder success with sustainability. This publication lays out the findings of the study and describes how both grantees and policymakers can promote the sustainability of 21CCLC programs.Article (7)Title: Building capacity and sustainable prevention innovations: a sustainability planning modelAuthors: Johnson, K., Hays, C., Center, H., and Daley, C.Source: Evaluation and Program Planning 27, p , 2004.This article presents an informed definition of sustainability and an associated planning model for sustaining innovations (pertinent to both infrastructure and interventions) within organizational, community, and state systems. The planning model stems from a systematic review of the literature and from concepts derived from a series of ‘think tanks’ made up of key substance abuse prevention professionals. The model assumes a five-step process (i.e. assessment, development, implementation, evaluation, and reassessment/modification) and addresses factors known to inhibit efforts to sustain an innovation. One set of factors concerns the capacity of prevention systems to support sustainable innovations. The other pertains to the extent to which a particular innovation is sustainable. A sustainability action strategy is presented that includes goals with corresponding sets of objectives, actions, and results that determine the extent of readiness to sustain an innovation. Sustainability tools to assist in implementing the planning model are illustrated, and next steps for the model are discussed. This planning model provides a conceptual and practical understanding of sustainability that can lead to further investigation.
26 Investigating Program Sustainability January 17, 2005Investigating Program SustainabilityAndrew Powers, Research AssociateAmy L. Powers, PrincipalProgram Evaluation and Educational Research(PEER) Associates, Inc.NPS Eval Work Group Evaluation Primer Amy Powers Conservation Study Institute Shelburne Farms
Your consent to our cookies if you continue to use this website.