1. www.edvantia.org Organizational Consultant, Critical Friend, and Evaluator: The Value and Challenge of Flexible Roles Kristine L. Chadwick, Ph.D. Evaluator for The Spatial Intelligence and Learning Center (SILC) The Science of Learning Center on Visual Language and Visual Learning (VL 2 )

2. www.edvantia.org SILC and VL 2 Evaluations NSF encouraged centers to change evaluators Have served as SILC’s evaluator since its third year of operation (2008-2009) Served as VL 2 ’s evaluator during its fourth year of operation (2009-2010) (and will continue in Years 6-10) 2

3. www.edvantia.org What is SILC? (http://spatiallearning.org) “The Spatial Intelligence and Learning Center (SILC) brings together scientists and educators from Temple University, Northwestern University, the University of Chicago, the University of Pennsylvania, and the Chicago Public Schools (CPS) to pursue the overarching goals of –understanding spatial learning –using this knowledge to develop programs and technologies that will transform educational practice, helping learners to develop the skills required to compete in a global economy SILC participants include researchers from cognitive science, psychology, computer science, education, and neuroscience, as well as practicing geoscientists and engineers who are particularly interested in spatial thinking in their fields, and teachers in the CPS.” 3

4. www.edvantia.org What is VL 2 ? (http://vl2.gallaudet.edu) “The purpose of VL 2 is to gain a greater understanding of the biological, cognitive, linguistic, sociocultural, and pedagogical conditions that influence the acquisition of language and knowledge through the visual modality. Gallaudet University houses VL 2, and brings together deaf and hearing researchers and educators from a variety of disciplines and institutions to study how language and literacy develop in deaf individuals.” 4

5. www.edvantia.org SILC and VL 2 Evaluation Priorities Satisfy NSF evaluation “requirements” Present well at Year 4 high-stakes site visit Consult on center management issues Gather and report data on aspects of center PIs knew less about (e.g., outreach for SILC, training at affiliate IHEs for VL 2 ) 5

6. www.edvantia.org Articulating Center Theory Theory of the center, not its research per se –Value added of center-mode research –Challenging for the center scientists Immediate and intermediate outcome categories –Centerness –Dissemination –Interdisciplinarity –Outreach –Training –Tools and resources 6

7. www.edvantia.org Program Theory / Logic Model Long-term Outcomes (> 10 years) –Social Capital (reputation, networks) –Intellectual Capital (research advancement) –Physical Capital (tools and resources) –Human Capital (diverse interdisciplinary researchers) Impacts based on NSF goals –advancing science –applying science to specific technological, educational, and workforce challenges –enabling science to capitalize on new opportunities and discoveries and respond to new challenges 7

8. www.edvantia.org Evaluation Approach Operationalizing, measuring (or documenting) resources, activities, outputs, immediate outcomes in logic model Reflect their growth as centers Consulting on center processes Outsider viewpoint, critical friend 8

9. www.edvantia.org NSF Requirements Center-level evaluation requirements not very clear –Show value added of center-mode functioning –Present outcomes of center at site visits NSF need: To demonstrate to the National Science Board the worth or value of the Science of Learning Centers 9

10. www.edvantia.org Limited Resource Tensions External Evaluator Show merit or worth of the center Demonstrate how center- mode functioning better than individual RO1 grants Focus on outcomes Consultant & Critical Friend Fix some process gaps Work through the logic model (are activities happening and resulting in expected outputs?) Attend to developmental stage of center Focus on process and some immediate outcomes 10

11. www.edvantia.org Not enough resources to attend to all needs Caption for this vacation photo: “Up a Tree” 11

12. www.edvantia.org Conflicting Role Example SILC Year 3 Evaluation –Process focused –Organizational consulting –Conversation and informal feedback and suggestions to Pis –SILC leadership satisfied –Site visit team wants outcome data 12

13. www.edvantia.org Example SILC & VL 2 Year 4 Evaluation –Outcomes oriented –Bibliometric analyses Limited corpus of publications Not useful to center at this early stage –Less feedback mid-year –PIs less satisfied –Fewer questions/critiques from site visit team 13

14. www.edvantia.org Balancing Tensions in a Most Unusual Way Years 6-7 will bring back focus on process (new 5-year funding cycle) Update logic model and key measures Focus also on intermediate outcomes (now appropriate in center’s lifecycle) “SLC Evaluation Network” economies of scale may result in better –center-specific evaluation –SLC program-wide evaluation 14

15. www.edvantia.org Thank You! Kristine Chadwick, Ph.D. Executive Director Evaluation and Applied Research Initiatives 304.347.0429 kristine.chadwick@edvantia.org 15