1. Broader Impact Project Evaluation Chris Parsons Word Craft

2. Webinar Agenda Who are you? Define Evaluation Show Where Evaluation Fits with BI Plans How happy are you? Apply Evaluation to NSF Broader Impact Criteria Example: BI Evaluation Plan Sum Up How’d we do? Q&A

3. Who am I? Independent Educator & Evaluator (schooling in biology, education & business) Current Projects –COSEE NOW Evaluator –Evaluator on Ross Sea Glider Project (J. Kohut) –Evaluation Advisor/Resource for NOAA’s California B-WET Grant Program –External Evaluator for Monterey Bay Aquarium Education Programs

4. Front-end Evaluation

5. At the Start: Who are you? Front-end Evaluation

6. Who are you? #1 Did you participate in any of the previous BI webinars in this series? Yes XNo

7. Who are you? #2 Rate your experience with conducting evaluations of BI projects. A.No experience: a neophyte B.Little experience: involved in 1 evaluation C.Some experience: more than 1 evaluation D.A lot of experience: an expert

8. Who are you? #3 When does evaluation occur during a BI project? A.At the beginning B.At the end C.In the middle D.All of the above

9. What is Evaluation? Copyright joe-ks http://www.joe-ks.com/archives_dec2003/SantaJobEvaluation.htm

10. What is Evaluation? Evaluation is the systematic collection of information about activities, characteristics, and outcomes of projects in order to make judgments about the project, improve effectiveness, and/or inform decisions about future programming. NSF, adapted from M.Q. Patton Project evaluation helps determine a project’s merit (does it work?) & its worth (do we need it?). NOAA, Designing Ed Projects 2005

11. Why Should You Care? Government Performance & Results Act (GPRA) Helps connect you with your audience(s) and improve what you do Can help you make a case to continue, to get some or more funding Enables you to keep what’s good and toss what’s bad And, wouldn’t you like to know that what you’re doing makes a difference?

12. NSF Broader Impacts Criterion 2 How well does the activity advance discovery and understanding while promoting teaching, training and learning? How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks and partnerships? Will the results be disseminated broadly to enhance scientific and technological understanding? What may be the benefits of the proposed activity to society?

13. BI & Evaluation: The Fit Choose BI Plan BI Project (incl. Evaluation) Do BI Project Design/Develop BI Project

14. BI & Evaluation: The Fit Choose BI Plan BI Project (incl. Evaluation) Do BI Project Design/Develop BI Project

15. BI & Evaluation: The Fit Choose BI Plan BI Project (incl. Evaluation) Do BI Project Design/Develop BI Project Summative Eval

16. BI & Evaluation: The Fit Choose BI Plan BI Project (incl. Evaluation) Do BI Project Design/Develop BI Project Summative Eval Formative Eval

17. BI & Evaluation: The Fit Choose BI Plan BI Project (incl. Evaluation) Do BI Project Design/Develop BI Project Summative Eval Formative Eval

18. BI & Evaluation: The Fit Choose BI Plan BI Project (incl. Evaluation) Do BI Project Design/Develop BI Project Front-end Eval Summative Eval Formative Eval

19. Formative Evaluation http://www.funnydb.com/

20. In the Middle: How happy are you? Formative Evaluation http://www.funnydb.com/

21. How happy are you? So far, are your information needs being met? A.Yes, definitely B.Yes, mostly C.Not sure yet D.No, not at all

22. Formative check-in Where Next?

23. NSF Broader Impacts Criterion 2 How well does the activity advance discovery and understanding while promoting teaching, training and learning? How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks and partnerships? Will the results be disseminated broadly to enhance scientific and technological understanding? What may be the benefits of the proposed activity to society?

24. NSF Broader Impacts Criterion 2 How well does the activity ADVANCE discovery and understanding while promoting teaching, training and learning?

25. NSF Broader Impacts Criterion 2 How well does the proposed activity BROADEN the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)?

26. NSF Broader Impacts Criterion 2 To what extent will it ENHANCE the infrastructure for research and education, such as facilities, instrumentation, networks and partnerships?

27. NSF Broader Impacts Criterion 2 Will the results be disseminated broadly to ENHANCE scientific and technological understanding?

28. NSF Broader Impacts Criterion 2 What may be the benefits of the proposed activity to society? (e.g., BROAD & USEFUL application in service to society)

29. NSF Broader Impacts Criterion 2 How well does the activity ADVANCE discovery and understanding while promoting teaching, training and learning? How well does the proposed activity BROADEN the participation of underrepresented groups (e.g., gender, ethnicity, disability, geographic, etc.)? To what extent will it ENHANCE the infrastructure for research and education, such as facilities, instrumentation, networks and partnerships? Will the results be disseminated broadly to ENHANCE scientific and technological understanding? What may be the benefits of the proposed activity to society? (e.g., BROAD & USEFUL application in service to society)

30. BI Example

31. Josh’s Antarctic Glider Project BI Plan Online expedition journal (photographer & writer) Museum program Teacher professional development & support materials

32. Josh’s Antarctic Glider Project BI Evaluation Plan The evaluation will focus on effectiveness of the instructional materials, including a teacher ’ s guide, implementation strategies, handouts, student worksheets and any other materials needed to implement the unit and meet the intended learning outcomes. The evaluator will use surveys and interviews, starting with the summer workshop and continuing through the schoolyear, to collect data and feedback on the effectiveness of the teacher professional development training and materials. Accompanying the web-based photo essays will be an online survey (similar to that used in the WHOI Live from the Poles project) to gather viewers' feedback on the expedition, our reporting of it and viewers ’ understanding of polar science issues. At the Liberty Science Center presentation we will use surveys to measure viewers ’ satisfaction and learning.

33. Sum Up Choose BI Plan BI Project (incl. Evaluation) Do BI Project Design/Develop BI Project Front-end Eval Summative Eval Formative Eval

34. Summative Evaluation

35. At the End: How’d we do? Summative Evaluation

36. Q&A Let’s Talk....

37. Thank You! To COSEE NOW To All of You