Promoting a Culture of Evidence Through Program Evaluation Patti Bourexis, Ph.D. Principal Researcher The Study Group Inc. OSEP Project Directors’ Conference July 21, 2009

Session purpose Stimulate thinking and discussion about the important outcomes of evaluation for project directors. Illustrate frameworks and processes that project directors can use to support both project implementation and accountability. The Study Group Inc.

Session themes Project directors can use evidence to assure sound project planning, quality implementation and maximum results. Evaluation tools and processes can help project directors do the best job with project planning, implementation and accountability. Part D programs have their own priorities for evidence, which project directors need to support. The Study Group Inc.

First theme Project directors can use evidence to assure sound project planning, quality implementation and maximum results. The Study Group Inc.

Culture of evidence Shared value and belief about the importance of evidence in project planning, implementation, management and accountability. The Study Group Inc.

Kinds of evidence Research Pilot-tests Models theoretical experiential experimental Needs assessment Expert opinion Experience and expertise External review The Study Group Inc.

Kinds of evidence Prior evaluations Statutes/regulations/policies Standards Databases Products/artifacts Documentation Assessments individual programmatic organizational The Study Group Inc.

Characteristics of a culture of evidence Evidence is used throughout the project’s life cycle. Numerous design and implementation changes occur during the projects’ life cycle. Project outcomes are clearly stated in operational terms that are understood by all staff. Project directors make reliance on evidence a priority. Processes for gathering and applying evidence are explicit and integrated into project planning, implementation and management tasks. The Study Group Inc.

Characteristics of a culture of evidence Staff are expected to use evidence to enhance the quality of their work. Staff have a common language and understanding about what evidence is, how to gather it, and how to apply it to improve project performance. There are opportunities for staff to expand their abilities to gather and apply evidence. The Study Group Inc.

Two characteristics of culture of evidence Evidence is used throughout the project’s life cycle. Numerous design and implementation changes occur during the project’s life cycle. The Study Group Inc.

Project life cycle Design Launch Development Stability ImplementationSustainability The Study Group Inc.

Project planning Why evidence is important. Ensures the credibility of the project design: that it reflects our knowledge about successful projects of this type. Demonstrates relevance of the project to the target audience and situation. Builds confidence that the project will succeed. The Study Group Inc.

Project planning The kinds of evidence that are important. Research Models theoretical experiential experimental Databases Needs assessment Prior evaluations Standards The Study Group Inc.

ED mission & principles of high-quality professional development Focuses on teachers as central to student learning, yet includes all other members of the school community. Focuses on individual, collegial and organizational improvement. Is planned collaboratively by those who will participate in and facilitate that development. Requires substantial time and other resources. The Study Group Inc.

Project implementation Why evidence is important. Show that project activities are implemented as planned. Modify activities when performance falls short of expectation or to take advantage of opportunities. Demonstrate that the outcomes are feasible and reasonable. Engender confidence that the project is well- managed and on track to realize its outcomes. The Study Group Inc.

Project implementation The kinds of evidence that are important. Pilot-tests Documentation Assessments individual programmatic organizational Products/artifacts External review Experience and expertise Databases The Study Group Inc.

Stability and sustainability Why evidence is important. Confirm that the project’s implementation is stable. Show that the project outcomes are reliably achieved and attributable to the project. Demonstrate the importance of the project to the target audience and situation. Plan sustainability. The Study Group Inc.

Stability and sustainability The kinds of evidence that are important. Assessments individual programmatic organizational Products/artifacts Documentation Databases Statutes/regulations/policies Expert review Expert opinion The Study Group Inc.

Third characteristic of a culture of evidence Project outcomes are clearly stated in operational terms that are understood by all staff. The Study Group Inc.

Capacity-building “Capacity building involves developing the collective ability---dispositions, skills, knowledge, motivation and resources---to act together to bring about positive change…” (Fullen, 2005). “Organizational capacity is its potential to perform---its ability to successfully apply its skills and resources to accomplish its goals and satisfy its stakeholders’ expectations” (Horton, et. al, 2003). M. Fullen. (2005) Leadership and Sustainability: System Thinkers in Action. Corwin Press: Thousand Oaks, CA. D. Horton, et.al. (2003). Evaluating Capacity Development. International Service for National Agricultural Research: The Hague, The Netherlands. The Study Group Inc.

Indicators of increased capacity Increases in staff disposition, knowledge and skill. Increases in use of evidence-based interventions. Closer ties to networks for exchange of resources and expertise. Improvements in policies, processes and management practices. Increases in resources devoted to the particular issue or concern (e.g., money, staff FTE). Changes in agency priorities. The Study Group Inc.

Summary Characteristics of a culture of evidence Evidence is used throughout the project’s life cycle. Numerous design and implementation changes occur during the projects’ life cycle. Project outcomes are clearly stated in operational terms that are understood by all staff. Project directors make reliance on evidence a priority. Processes for gathering and applying evidence are explicit and integrated into project planning, implementation and management tasks. The Study Group Inc.

Summary Characteristics of a culture of evidence Staff are expected to use evidence to enhance the quality of their work. Staff have a common language and understanding about what evidence is, how to gather it, and how to apply it to improve project performance. There are opportunities for staff to expand their abilities to gather and apply evidence. The Study Group Inc.

Contact information Patti Bourexis, Ph.D. Principal Researcher The Study Group Inc.