1. The Development and Validation of the Evaluation Involvement Scale for Use in Multi-site Evaluations Stacie A. ToalUniversity of Minnesota Why Validate a Scale for Involvement? 1)Improve research on participatory evaluation by making it easier for researchers to replicate studies in various settings. 2)Identify factors critical to involvement so that evaluators can incorporate activities into multi-site evaluations to help participants feel involved. 3)Justify the investment of time and resources in participatory approaches if there is more research available to demonstrate a positive relationship between involvement and evaluation use. Theoretical Framework: Messick’s Unitary Concept of Validity Validity is “an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.” (Messick, 1989, 1995a, 1995b) 1. Discussions that focused the evaluation 2. Identifying evaluation planning team members 3. Developing the evaluation plan 4. Developing data collection instruments 5. Developing data collection processes 6. Collecting data 7. Reviewing collected data for accuracy and/or completeness 8. Analyzing data 9. Interpreting collected data 10. Writing evaluation reports 11. Presenting evaluation findings (e.g., to staff, to stakeholders, to an external audience) Funded by the National Science Foundation, this three-year research project is studying the use and influence of evaluations of four NSF-funded programs by examining the relationship between the extent of involvement of evaluation stakeholders and the long-term impact of the evaluations on project staff, on the science, technology, engineering, and mathematics (STEM) community, and on the evaluation community. The Beyond Evaluation Use Research Team Theoretical Soundness 1. Amount of published research 2. Data from think-alouds 3. Interview data Internal Consistency 1. Item variance 2. Scale reliability (alpha=.94) Inclusive of Relevant Activities & Processes 1. Expert opinion 2. Scale items mentioned in interviews 3. Types of involvement (mentioned in literature or interviews) covered by scale Measures Actual Involvement 1. Survey response distributions 2. Project interviews Statistical Factors Match Rational Structures 1. Exploratory Factor Analysis 2. Confirmatory Factor Analysis Differentiates between Groups that are Rationally or Theoretically Different 1. Significant difference between evaluators and non-evaluators in implementation, but not in planning. Correlates as Expected with Other Variables 1. Significant, positive correlation with evaluation use. Measures Involvement in Other Settings 1. Equally strong reliabilities for each program, but low level of involvement in one program. Consequential Validity 1. Discussion of possible biases related to multicultural validity and evaluation use. Messick’s Aspect of Validity Evidence Presented Strength of Evidence StrongWeak The Beyond Evaluation Use Research Project Key Validity References Brualdi, A. (1999). Traditional and modern concepts of validity (Report No. ED435714). Washington D.C.: ERIC Clearinghouse on Assessment and Evaluation. Kane, M. T. (1992). Quantitative methods in psychology: An argument-based approach to validity. Psychological Bulletin, 112(3), 527-535. Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319-342. Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5- 11. Messick, S. (1995a). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5-8. Messick, S. (1995b). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. Implications of this Research Factor 1: Involvement in Planning The Evaluation Involvement Scale Response Options: 1=No 2=Yes, a little 3=Yes, some 4=Yes, extensively N/A=This activity did not take place Question Stem: To what extent were you involved in… Factor 2: Involvement in Implementation Primary Data Sources PI & Evaluator Online Survey Aug 2006 – Jan 2007 Topic: Involvement and use in four NSF multi- site evaluations. Sample Size: 372/810 Response Rate: 46% *Non-respondent study indicated no significant differences in levels of involvement between respondents and non- respondents. Interviews Mar 2007 – Apr 2007 Topic: Follow-up to survey with two questions related to involvement. Sample Size: 12 respondents who have various levels of involvement and use. Literature Review Fall 2005 – June 2007 Topic: Participatory evaluation research studies. Scope: 27 empirical and theoretical articles directly related to participatory evaluation. Validated evaluation research tool  11-item survey on evaluation involvement in multi-site evaluations  2 factors: planning and implementation Improved understanding of what involvement looks like in multi-site evaluations  Data collection  Attendance at meetings Facilitates future research stakeholder involvement  Positively correlated to evaluation use  Needs to be validated in different evaluation contexts B. Volkov, S. Toal, F. Lawrenz, L. Greenseid, K. Johnson, & J. King