1. Erin M. Burr, Ph.D. Oak Ridge Institute for Science and Education Jennifer Ann Morrow, Ph.D. Gary Skolits, Ed.D. The University of Tennessee, Knoxville Measuring Evaluation Use and Influence Among Project Directors of State Gaining Early Awareness and Readiness for Undergraduate Programs Grants

2. Overview Purpose to develop an instrument to measure evaluation use, evaluation influence, and impacting factors Participants current state project directors of Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) GEAR UP nationwide Dept. of Education college access grant program developed in order to prepare low-income students to meet the requirements for college enrollment and to succeed at the postsecondary level 6-year grants serving schools in multiple counties

3. Evaluation Use Instrumental Use “results are used in making decisions about program structure and function” ( Clavijo, Fleming, Hoermann, Toal, & Johnson, 2005) Conceptual Use “something that is newly learned about a program, its participants, its operations, or outcomes through an evaluation” ( Henry & Mark, 2003) Symbolic Use “involves drawing on evaluation evidence in attempts to convince others to support a political position, or to defend such a position from attack” ( Leviton & Hughes, 1981) “use of evaluation findings to retrospectively support a decision made prior to the evaluation finding” ( Henry & Rog, 1998) Process Use “…refers to and is indicated by individual changes in thinking and behavior, and program or organizational changes in procedures and culture, which occur among those involved in evaluation as a result of the learning that occurs during the evaluation process” ( Patton, 1997)

4. Kirkhart’s Integrated Theory of Influence Kirkhart, K. E. (2000). “Reconceptualizing Evaluation Use: An Integrated Theory of Influence.” In V. Caracelli and H. Preskill (eds.), The Expanding Scope of Evaluation Use. New Directions for Evaluation, no. 88. San Francisco: Jossey-Bass. “The term influence (the capacity or power of persons or things to produce effects on others by intangible or indirect means) is broader than use, creating a framework with which to examine effects that are multidirectional, incremental, unintentional, and instrumental.” (p. 7)

5. Henry and Mark’s Three Level Model of Evaluation Influence Levels of InfluenceIndividual Attitude Change Salience Elaboration Priming Skill Acquisition Behavioral Interpersonal Justification Persuasion Change Agent Social Norms Minority Opinion Influence Collective Agenda Setting Policy Oriented Learning Policy Change Diffusion Henry, G. T., & Mark, M. M. (2003). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24, 293-314.

6. Factors That Impact Evaluation Use ImplementationDecision & Policy Setting Quality Credibility Relevance Communication Findings Timeliness Information needs Decision characteristics Politics Funding Competing information Personal characteristics Commitment to evaluation Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research in evaluation utilization. Review of Educational Research, 56, 331-364.

7. Participants, Method, and Results Participants 17 current state GEAR UP project directors 44% response rate Survey completion 10 online 4 paper-and-pencil 3 email PDs used their program’s evaluations in all four types of use, reported influence at all three levels, and identified multiple factors that impacted their use of the evaluations.

8. Levels of Influence & Change MechanismsTypes of Evaluation Use Individual InstrumentalConceptualSymbolicProcess Attitude Change51 Salience11 Elaboration11 Priming11 Skill Acquisition1 Behavioral21 Interpersonal Justification1 Persuasion1 Change Agent111 Social Norms1 Minority Opinion Influence1 Collective Agenda Setting1 Policy Oriented Learning2 Policy Change1 Diffusion

9. Survey Instrument Measure# of ItemsCronbach’s alpha Evaluation UseTotal = 31* Instrumental7.84 Conceptual8.86 Symbolic8.88 Process8.92 Evaluation InfluenceTotal = 27 Individual16.93 Interpersonal7.82 Collective4.78 Impacting FactorsTotal = 13 Implementation6.84 Decision and policy setting7.89 Note. N = 17. *Four items were “other” types of use in each category, so they could not be labeled in terms of influence. Scale: 0 = No extent, 1 = Some extent, 2 = A moderate extent, 3 = A great extent, 4 = A very great extent.

10. Limitations and Implications for Use Limitations Participation Low number of respondents - limited statistical analysis No PDs participated in focus groups - limited interpretation of findings Self-report instrument – social desirability Generalizability – limited to project directors of federal grants Findings from this study can be used in GEAR UP to promote communication between grantees and evaluators about evaluation use and influence, raise awareness about the consequences of evaluations, and guide the design of grantee capacity-building workshops and training sessions on evaluation use.

11. Implications for Other Evaluators Use measure with project directors (PD) of other programs to Raise awareness about the types of use among PDs Track use and influence over time (e.g., multiple-year grants) Reflective practice Compare survey results to intended uses specified by the client (PD) at the beginning of an evaluation Assess the impact of their work (if responses are validated) and identify areas for improvement in the future To use as a conversation starter with program directors and staff about why they did or did not choose to use their evaluation results.

12. Directions for Future Research Additional research is needed to validate this instrument. More participants Different federal grant programs Addition of focus groups, interviews, or observations New items could be added to assess the collective change mechanism, diffusion, or a separate measure could be developed to assess diffusion. Incorporate Kirkhart’s Integrated Theory of Influence into measurement. Items could be reworded or expanded up by addressing the dimensions of time, source, and intention as they apply to each type of use.