1. Using Citation Analysis to Study Evaluation Influence: Strengths and Limitations of the Methodology Lija Greenseid, Ph.D. American Evaluation Association Conference Orlando, Florida November 12, 2009 This material is based on work supported by the National Science Foundation under Grant No. REC 0438545. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

2. Lija Greenseid, PhD AEA, November 2009 Background – Evaluation Use and Influence Use of evaluations by decision makers a key concern of evaluation practitioners, theorists, and sponsors Interest in the broad impact of evaluations (“evaluation influence”) (Kirkhart; Mark & Henry; Alkin & Taut) (Alkin & Taut, 2003)

3. Lija Greenseid, PhD AEA, November 2009 Context Four large-scale, multi-site NSF funded program evaluations or evaluation technical assistance programs Evaluations funded for 4-10 years Evaluation budgets: $1 to $6.5 million Evaluated between 20 and 350 projects Produced between 24 and 98 evaluation products Dissemination part of two contracts

4. Lija Greenseid, PhD AEA, November 2009 How can evaluation influence be measured? This study examined the usefulness of citation analysis as a method for assessing evaluation influence Assess the impact of knowledge generated through evaluations Examine use and influence of evaluation products (e.g., reports, publications, instruments, presentations) through examining the extent and patterns in citations of the products

5. Lija Greenseid, PhD AEA, November 2009 Delimitations Evaluation products only one possible source of evaluation influence Citation analysis proposed as one possible way of measuring the influence of evaluation products

6. Lija Greenseid, PhD AEA, November 2009 Method Collected citation data using: ISI Web of Science Google Scholar (scholar.google.com) Google search engine (www.google.com) Analyzed distributions of citation counts, levels, content, and network maps

7. Lija Greenseid, PhD AEA, November 2009 Evaluation Citation Levels Table 1. Program evaluation products and citation counts Program evaluation Number of Products Number of Citations LSC98248 ATE9864 CETP2442 MSP-RETA2510 CETP & ATE112 Total246376 Points of reference: Single-site evaluation example: WASDI R2 – 1 citation Large-scale international study: TIMSS – approximately 750-1,500 citations

8. Lija Greenseid, PhD AEA, November 2009 Coverage of literature Examined coverage by citation databases of key journals in STEM education and evaluation: Web of Science Good coverage of academic, research-oriented journals Poor coverage of practitioner-oriented sources Google Scholar Covered all key journals in both STEM and evaluation fields Captured references within evaluation reports Google Captured more non-formal sources of citations (e.g., electronic resource lists, Web sites, discussion groups)

9. Lija Greenseid, PhD AEA, November 2009 Methodology - Coverage Individually In Combination

10. Lija Greenseid, PhD AEA, November 2009 Methodology - Replicability Search Engine Researcher 1: Time 1/ Time 2 Researcher 2: Time 1/ Time 2 Researcher 1 & 2: Time 2 Across All Three Trials Web of Science 96.6% 100.0%96.6% Google Scholar 72.9%64.4%78.0%57.6% Google 67.8%78%76.3%61.0% Table 2. Percent agreement of citation data across replication trials

11. Lija Greenseid, PhD AEA, November 2009 Key Findings – Product Types Table 3. Citation levels for types of evaluation products Evaluation Product Type Number of Products Number of Citations Mean Citations/ Product Type Instrument/Tool201457.25 Evaluation Report1031761.72 Publication58550.95 Presentation6500 Total2463761.53 (F=13.241, df=3, p<.001, R 2 =14.1% )

12. Lija Greenseid, PhD AEA, November 2009 Key Findings – Content of Citations Content Analysis (n=35 citations from 30 randomly selected citing works): N% Instrument/Method1645.7 Empirical Findings617.1 Factual Statement38.6 Validity38.6 Resource38.6 Concept Marker25.7 Further Information12.9 Not citation12.9 Classic00.0 Total35100.0 N% Modified instrument743.8 Instrument was developed for use in the evaluation 425.0 Used instrument in another context318.8 Did not use212.5 Total16100 Table 4: Content of Citations Table 4a. Instrument/Method Citations

13. Lija Greenseid, PhD AEA, November 2009 Key Findings – Citation Patterns Figure 2. Citation network for the program evaluations

14. Lija Greenseid, PhD AEA, November 2009 Citation Patterns – ATE Eval. Figure 3. ATE products and citations Color Key ATE Product Citing Work

15. Lija Greenseid, PhD AEA, November 2009 Citation Patterns- LSC Eval. Figure 4. LSC evaluation products and citations Color Key LSC Product Citing Work

16. Lija Greenseid, PhD AEA, November 2009 Theoretical Implications Newly proposed evaluation influence framework (adapted from Kirkhart, 2000 and Alkin and Taut, 2003)

17. Lija Greenseid, PhD AEA, November 2009 Conclusions Citation analysis is useful to a limited extent Helpful for comparing patterns of influence Best suited for comparative study of large-scale evaluations Measures only one limited type of evaluation influence – arising from products Citation counts underestimate true levels Highly labor-intensive process for collecting and managing data

18. Lija Greenseid, PhD AEA, November 2009 Contact Information Lija Greenseid, Ph.D. Senior Evaluator Professional Data Analysts, Inc. www.PDAstats.com Minneapolis, MN lija@pdastats.com