1. The Logic Model as Compass: Guiding Program Staff Through the Consequences of Evaluation Ellen Roscoe Iverson, Carleton College, eiverson@carleton.edu eiverson@carleton.edu John A McLaughlin, Managing for Results, macgroupx@aol.com macgroupx@aol.com Cathryn Manduca, Carleton College, cmanduca@carleton.edu cmanduca@carleton.edu This project is supported by the National Science Foundation (NSF) Division of Undergraduate Education under Grants No. 0127310,0127141,0127257, 0127018, and 0618482. Opinions, findings and conclusions or recommendation expressed herein are those of the authors and do not necessarily reflect the views of the NSF. On the Cutting Edge is sponsored by the National Association of Geoscience Teachers (NAGT) and is part of the Digital Library for Earth System Education (DLESE).

2. Overview  On the Cutting Edge program  Goals of evaluation  Logic models  Evaluation methods  Results

3. But first…have you used: Computer-based interventions?Computer-based interventions? Pedagogically-based professional development for faculty?Pedagogically-based professional development for faculty? Logic models as part of iterative design?Logic models as part of iterative design?

4. On the Cutting Edge Delivered at national level to geoscience facultyDelivered at national level to geoscience faculty Combines residential workshops and websites for faculty professional developmentCombines residential workshops and websites for faculty professional development Supported by grant from National Science FoundationSupported by grant from National Science Foundation Began workshops in 2002Began workshops in 2002 Funded for 3 more yearsFunded for 3 more years

5. Workshops (3 to 5 days) Emerging themes (2/year) Teaching X (1/year) Course Design (online and face-to-face) Career Preparation and Management (2/year) Website Instructional materials Datasets and tools Pedagogical resources Tutorials Course resources Assessment tools Bibliographies Visualizations On the Cutting Edge

6. http://serc.carleton.edu/NAGTWorkshops

11. Developing Evaluation Purpose According to Guskey (2000)*, evaluators of professional development make three mistakes: 1.Collect and report descriptive information – who was involved. 2.Focus on attitudes of participants – did they think their time was well spent – and not on actual changes in the participant knowledge or skill. 3.Keep evaluations brief and limit opportunities for application. *Guskey, T.R. (2000). Evaluating professional development. Published: Thousand Oaks, Calif.: Corwin Press.

12. Initial Evaluation Purpose  Continuously improve the workshops and website  Create information for others in our community about what works and does not work with respect to professional development for the members of our community. *

13. Goals – 5 basic questions 1.Was the program implemented as planned? 2.What was the quality of the implementation? 3.What was the effect of the program on the participants? 4.What was the impact of the program? 5.What caused the observed effects and impacts?

14. Logic Models

15. Workshop Logic Model

19. Website Logic Model

22. Methodologies Workshops –Road checks –End of workshop surveys –Observations and interviews –Online surveys –Baseline survey –Telephone interviews Website –Web statistics reports –Pop-up survey –Awareness poll –External Heuristic Review of website –Focus groups –Telephone interviews Pilot – Imbedded assessment – Online discussion artifacts

23. Results Evaluation question InstrumentHighlights 1.Was the program implemented as planned? Baseline survey and follow up email survey Workshop application Web log statistics 1/3 of geoscience faculty awareness 27 workshops and 1100 participants. 1500 cataloged resources, and over 1000 web pages 267,000 unique visits to website for 2005 2.What was the quality of the implementation? End of workshop surveys Website heuristic Mean workshop ratings of 9.0 out of 10. Usability of site strong.

24. Effects

25. Results Evaluation question InstrumentHighlights 3.What was the effect of the program on participants? Imbedded assessment End of workshop surveys New knowledge and skills acquired at workshops with plans to put it into practice End of workshop surveys 54 telephone interviews Prevailing change in attitude “what do I teach”  “what are they learning”. 70% of interviews End of workshop surveys Online surveys Ability to discuss their new knowledge with others ranked high

26. Impacts

27. Results Evaluation question InstrumentHighlights 4.What was the impact of the program? Telephone interviews Online survey 80% attribute changes in teaching methods to workshop Sustained interest aided by website Telephone interviews Online survey Indicators 75% attribute increased sphere of influence to workshops Caveat: workshop networks weak

28. Results Evaluation question InstrumentHighlights 5.What caused the observed effects and impacts? End of workshop surveys Telephone interviews Being able to immediately apply ideas Networking with other participants shaped the context of their learning Telephone interviews Focus groups Online surveys Website resources helped sustain changes and enhance knowledge

29. Summary

30. Implications for future Snowball sampling to evaluate website-only participantsSnowball sampling to evaluate website-only participants Imbedded assessmentImbedded assessment Repeat baseline surveyRepeat baseline survey Formal leadership programFormal leadership program 100% participants contribute to website100% participants contribute to website

31. For more information: http://serc.carleton.edu/NAGTWorkshops/evaluation.html