1. CONTINUING EVALUATION OF HIGHER EDUCATION PROGRAMS: CONCEPTUAL AND METHODOLOGICAL PERSPECTIVES AND CASE STUDY ANA MARIA CARNEIRO (NEPP – UNICAMP) CIBELE YAHN DE ANDRADE (NEPP – UNICAMP) ADRIANA BIN (FCA – UNICAMP) STELA M. B. S. TELLES (NEPP – UNICAMP) AEA Session 318 - Higher Education Evaluation Data to Support Planning and Resource Investment October 2013

2. Presentation goal  To discuss the concept of continuing evaluation  To exemplify it by the experience of evaluating University of Campinas’s (Unicamp) Interdisciplinary Higher Education Program (ProFIS)

3. Continuing evaluation  It is organized as a series of evaluations and decisions regarding the definition of policy agenda and following the whole cycle of execution and after its conclusion, to follow its second or third order impacts  It is implemented “within the program”  Evaluation conceived as a part of the definition of the policy  As a tool to subside management  To enhance planning activity

4. Monitoring Needs assessment Priority setting Evaluation of results Impact evaluation Program & management review Definition of agenda Identification and selection of alternatives Implementation Continuing evaluation

5.  Continuing evaluation uses a specific methodology to cover all stages of a given program  from implementation through to impact evaluation  so as to integrate monitoring and impact evaluation including long-term impacts in a single design  Formative evaluation (for learning) adding summative evaluation (for accountability)

6. Before implementation Support for definition of policy agenda (selection of policy and its design) During implementation Support for policy implementation (driving policy to achieve goals established at the begining) After implementation (short and medium term) Results and impact evaluation (for accountability and to support new policy design) After implementation (long term) Results and impact evaluation (to support new policy design) Time-bound policy

7. Before implementation Support for definition of policy agenda (selection of policy and its design) During implementation Support for policy implementation (driving policy to achieve goals established at the beginning and redefinition of goals ) After implementation (short and medium term) Results and impact evaluation (for accountability and to redefinition of goals) After implementation (long term) Results and impact evaluation (for accountability and to support new policy design, to redefine the goals ) policy with indefinite term

8. ProFIS (Interdisciplinary Higher Education Program)  Program is a national and international reference General Education New selection process Social inclusion  Courses of all academic areas  Introduction to research practice  Courses of all academic areas  Introduction to research practice  First access to university and therefore undergraduate Program  Support to carrier selection  First access to university and therefore undergraduate Program  Support to carrier selection  To increase the access of low-income and minority students  Inspired by Texas Top 10% system  To increase the access of low-income and minority students  Inspired by Texas Top 10% system

9. Evaluation objectives  To evaluate implementation and viability of ProFIS as a general education Program for Unicamp  To evaluate ProFIS impact on academic and professional trajectory of student  To evaluate the selection process of ProFIS for promoting social inclusion  2 main studies  Longitudinal and quase-experimental study  Evaluation of implementation of program

10. ProFIS continuing evaluation Monitoring Evaluation of results Impact evaluation Program & management review Definition of agenda Identification and selection of alternatives Implementation Evaluation of Implementaion Longitudinal and quase- experimental study

11. Coordinator of ProFIS Representative of undergraduate pro-rector Expert on general education Advisory committee Researchers of NEPP Undergraduate students Graduate students Execution team Undergraduate students Graduate students Researchers of Unicamp Colaborators Evaluation management

12. Evaluative matrix GoalHypothesesIndicators To evaluate implementation and viability of ProFIS as a general education Program for Unicamp H1. The general and interdisciplinary formation of ProFIS amplifies knowledge and general culture of students with an investigative attitude, critical vision and civic engagement H2... H3… H4… Main contribution of ProFIS Self-evaluation of development of basic habilities and contribution of ProFIS …

13. Longitudinal and quase-experimental study  Goal: to follow academic and professional trajectories  Cohorts: beneficiaries and members of control groups by year  Waves are annual since the moment they apply to ProFIS or undergraduate programs of Unicamp until integration in the world of work  Duration: ten years for each cohort

14. Longitudinal and quase-experimental study  Data source  Surveys (application, enrollment, annual monitoring)  Secondary data (academic records, Internet searches and social networks)  Analysis  Continuous (e.g. social inclusion on access and retention)  Deep studies (e.g. drop out process, process of carrier selection)

15. Evaluation about implementation of ProFIS  Goal: evaluate implementation and viability of ProFIS as  A general education Program  Its selective process  Duration  2011-2013 – pilot project  2014 - Expansion of the Program

16. Evaluation about implementation of ProFIS  Primary data  Interview with teachers  Monitoring survey of students  Interview with coordinator of ProFIS and other administration team  Secondary data  Courses offered  Costs  Infrastructure  Analysis:  Continuous information available to management of ProFIS  The viability report  Presentations within and outside of Unicamp

17. Critical points of Continuing evaluation

18. Advantages of continuing evaluation  Evaluation can integrate the decision process to improve efficiency, efficacy and effectivity  To follow the results and impacts since the beginning  To produce and use data as a subsidy of planning  More flexibility to define methodologies (normally mixed methods)  To compare the situation before and after the policy  To design quasi-experimental evaluations

19. Challenges of continuing evaluation  Methodological  To constitute a reliable methodology, that is inexpensive, that can follow change on program objectives  To limit what has to be evaluated – avoiding the risk of creating an extensive list of meaningless indicators  To keep the stability of the system of evaluation and a historical series of data even with the revision of themes and indicators as the program evolves

20. Challenges of continuing evaluation  Technological  To constitute a monitoring and evaluation system that is both comprehensive and effective  To integrate secondary data from several systems within Unicamp with primary data from our own surveys

21. Challenges of continuing evaluation  Organizational  To assure that evaluation will happen on a scheduled basis with appropriated resources  leading to the institutionalization of evaluation  Skilled internal staff and reliable external collaborators  To keep the participation of stakeholders, management staff, as well as beneficiaries and members of control groups  To coincide evaluation timing with management timing  To develop within management staff the ability to understand results of continuing evaluation to be able to use them effectively