1. E101 Section 8 October 31, 2012 Policy and Program Evaluation

2.  Situating Ourselves in the Course (1 minutes)  Logical Framework Problem Tree Analysis (10 mins)  Discussion of Weiss (40 mins)  Design an Evaluation (30 mins)  Housekeeping & Questions (5 minutes) Agenda

3. 1 Introduction to Comparative and International Education 2 The Process of Policy Analysis 3 Education Policy Options Course Overview

4. 1 Introduction to Comparative and International Education 2 The Process of Policy Analysis 3 Education Policy Options Course Overview –Week 7 – Assessing the Performance of an Education System –Week 8 – Policy and Program Evaluation –Week 9 – Curriculum, Standards, and Assessment

5. “Your problem definition should not include an implicit solution introduced by semantic carelessness. Projected solutions must be evaluated empirically and not legitimated merely by definition. Therefore, keep the problem definition stripped down to a mere description, and leave open where you will look for solutions.” (Bardach, 2009, pg. 7) Defining the Solution into the “Problem”

6. “New schools are being built too slowly.” vs. “There are too many schoolchildren relative to the currently available classroom space.” Example

7. 1.List all the problems that come to mind. Problems need to be carefully identified: they should be existing problems, not possible, imagined or future ones. The problem is an existing negative situation, it is not the absence of a solution. 2.Identify a core problem (this may involve considerable trial and error before settling on one). 3.Determine which problems are “Causes” and which are “Effects.” 4.Arrange in hierarchy both Causes and Effects, i.e., how do the causes relate to each other - which leads to the other, etc. LogFrame Problem Analysis Tree

9. Where we are  Part II: The Process of Policy Analysis -Last week we explored Assessing the performance of an education system -This week we take a look at Policy and Program Evaluation  We read Evaluation (1998) by Carol Weiss, (Ch. 1-5)

10. Goals  Define and discuss key terms  Review main points from the reading  Compare frameworks from Weiss and Reimers & McGinn

11. Key Terms  Evaluation  What is?  Why is it important in international education policy research and policy making?  Summative Evaluation, Formative Evaluation, Process-Outcome  Identify the focus and purpose of each type of evaluation  Provide an example

12. Formative-Summative Process – Outcome  Formative Evaluation:  Study results feed back into program development. Often used during beginning stages of a program.  Summative Evaluation:  Provides information about a program’s effectiveness. Conducted after a program is completed. Helps decision makers decide whether to continue or stop a program  Process Evaluation:  Examines what happens during the program, from the perspective of the participant  Outcome Evaluation:  Examines what happens to the participants after the program is finished Evaluation: “The systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit and implicit standards, as a means of contributing to the improvement of a policy or program.” (Weiss, p.4)

13. Main Points from Weiss  We will break into three groups for this portion of discussion  As a group…  Choose a spokesperson  Identify three to five main points from the chapters to which your group is assigned (10 minutes)  As a large group, discuss conclusions from small group discussions

14. Main Points from Weiss - Groups  Group 1: Chapters 1&2  Group 2: Chapters 2&3  Group 3: Chapters 4&5

15. Role of the Evaluator Weiss Reimers & McGinn  Empowerment Evaluation: community-conducted evaluation research  Collaborative Evaluation: researcher as co-investigator; no wall between practitioner and evaluator  Stakeholder Evaluation: engages different stakeholders, learns their concerns, assumptions, questions, data needs and intentions/reasons for conducting the evaluation Policy Dialogue as Organizational Learning:  the researcher involves the client and relevant stakeholders in the community, from planning the study and developing the instrument to analyzing the data and presenting the results of the study.  Dialogue among stakeholder groups and researcher(s)  Policy research as a wave How are the frameworks similar or different? Which approach would you use and why?

16. Design a Program Evaluation  Imagine that NFTE wants to conduct an alumni study in 2012-13.  Their goal with this study is to re-connect with our alumni in order to tell the story of the deep, long-term impact of their youth entrepreneurship programs.  They hope that this survey will help inform the field on the impact of youth entrepreneurship programs.

17.  NFTE has approximately 200,000 alumni in the U.S. They instituted formal systems for gathering alumni information only four years ago (2008).  Prior to that, they kept track of many of our program graduates on an ad hoc basis, largely via personal communications with founder, Steve Mariotti, and other NFTE staff members. NFTE Facts

18.  NFTE has basic data – name, address, NFTE program information, email in some cases – for only about 10,000 alumni, plus piecemeal data on another 3,000 or so.  They also have a set of alumni connected to NFTE through Facebook and LinkedIn, some of which are likely duplicative of the 13,000.  Their system contains lots of incomplete data, many duplicates to be cleaned, and much data that is likely obsolete, in the numerous cases where alumni have not updated their information since their initial registration. Alumni Database

19. Challenges of Tracking Alumni  It is very difficult to find alumni who completed NFTE prior to 2008 when NFTE set up the database.  Moreover our alumni population is relatively young and therefore naturally transient. They are unlikely to prioritize updating their contact information with NFTE each time they relocate.  Finally, NFTE has not devoted significant staffing resources to alumni tracking.  THINK OF WAYS IN WHICH NFTE COULD POTENTIALLY USE SOCIAL MEDIA TO RECONNECT WITH ALUMNI.

20. Design an Evaluation In teams design an evaluation that will answer the following questions:  Where are NFTE graduates now? What have they accomplished? How does their business ownership, employment, and education compare to that of their broad peer group? Keep in mind that not only does NFTE want to answer these questions, but they also want to inform the field’s general understanding of the impact of entrepreneurship education programs that target youths.

21. Design an Evaluation  Familiarize yourself with NFTE (vision, mission statement, where they operate).  www.nfte.com www.nfte.com  Design a survey tool that will help NFTE measure impact.  Please create a shared google doc for this exercise.  Invite veb682@mail.harvard.eduveb682@mail.harvard.edu  Think of existing datasets that NFTE could compare the results of their study to. They will use these comparisons to build a picture of our impact, into which we would integrate compelling individual stories.  Develop a timeline for NFTE. The data must be collected and analyzed before April 2013.