1. 1 Program Performance and Evaluation: Policymaker Expectations 2009 International Education Programs Service Technical Assistance Workshop Eleanor Briscoe February 2, 2009

2. 2 Who Are Our Policymakers? Congress The White House Office of Management and Budget Department of Education

3. 3 Presentation Objectives Government Performance and Results Act of 1993 Higher Education Opportunity Act (Title VI data collection and evaluation activities) National Research Council’s 2007 Review of Title VI/Fulbright-Hays programs FY 2008 House Appropriations Committee Report (directive addressing management and program needs) Future of the PART Process Expectations of our Policymakers

4. 4 ED/Budget Service The Department of Education’s Budget Service has lead responsibility for: developing and implementing the Department’s annual budget, formulating budget and related legislative policies; presenting the budget and related policy proposals to the public and to Congress; establishing and maintaining a Department wide performance-based budget formulation, execution, and management system; and reviewing and analyzing Department program operations.

5. 5 To that end, we as budget analysts— Develop all Department-level budget documents. Work with the Department’s offices to assure that performance measures ensure the effectiveness of the Department programs. Maintain liaison with OMB, congressional committees; CBO; GAO; and other organizations on the formulation, presentation, and execution of the Department’s budget.

6. 6 GPRA Holds Federal agencies accountable for achieving program results Requires setting goals and public reporting Seeks to reduce waste and inefficiency Helps identify what works and what does not

7. 7 How Budget Service presents your data? The Budget Service reports what you do as part of the GPRA process to Congress and OMB— through technical assistance to congressional staffers on behalf of their representatives, at OMB and congressional hearings, and as part of the annual budget submission to OMB and Congress.

8. 8 Overview of Higher Education Opportunity Act (Title VI data collection and evaluation activities) Requires the Secretary to assist grantees to develop a student survey to determine postgraduate employment, education, and training. Requires that the survey be administered once every two years. Specifies that up to 1 percent of authorized funds may be used to support Title VI program evaluations.

9. 9 National Research Council’s 2007 Review of Title VI/Fulbright-Hays programs Study concluded— that the Department should commission independent outcome and impact evaluations of all programs every 4 to 5 years; a need for better and more reliable data; a lack of rigorous, reliable information available on Title VI program performance.

10. 10 FY 2008 House Appropriations Committee Report Directive (addressing management and program needs) Urged the Department to improve IRIS to ensure that— (1)data collected contain performance outputs and outcomes that are relevant to program monitoring and improvement; (2)data system provide greater standardization, allow comparison across years and across programs, and provide information to all grantees and to the public.

11. 11 Actions taken by the Department to improve management practices and performance— Developing and implementing a strategy to use efficiency measures to improve cost effectiveness. Analyzing data for annual and long-term performance measures including looking at effective ways to measure post graduation data. Developing and implementing a strategy for conducting independent, rigorous, periodic, meaningful evaluations of the outcomes and impact of the international education programs. Developing a measure to track language skill changes through the use of reliable assessment tools. Making grantee performance data available to the public in a transparent and meaningful manner.

12. 12 The PART Process Developed to assess the effectiveness of federal programs and to help inform management action, budget requests, and legislative proposals directed at achieving results. Examined various factors that contribute to the effectiveness of a program and required that conclusions be explained and substantiated with evidence. Assessed if and how program evaluation is used to inform program planning and to corroborate program results. Consisted of four critical areas of assessment—purpose and design, strategic planning, management, and results and accountability.

13. 13 Future of the PART Process Future of PART uncertain. President Obama created a new White House position—Chief Performance Officer—aimed at eliminating government waste and improving efficiency. The office will work with federal agencies to set performance standards and hold agency managers accountable for progress.

14. 14 What do our policymakers and the public expect? Current, useful information on what the Title VI programs undertake and what they accomplish. Data on postgraduate employment, training, and education. Evidence that the Title VI programs are actually producing experts in areas of vital interest to the United States.

15. 15 Outcomes vs. Outputs Outcomes – Events or conditions external to the program and of direct importance to the public, beneficiaries and/or customers. They relate to the program’s mission, purpose and strategic goals. Outputs – Internal program activities; products and services delivered to the public, beneficiaries.

16. 16 Outcomes vs. Outputs (cont’d) Examples of Title VI outputs: number of foreign language course offerings by Title VI institutions number of comprehensive instructional resources (assessments, publications, curricular materials, etc.) produced at Title VI-supported institutions number of students served/teachers trained Examples of Title VI outcomes: percentage of graduates who not only find employment, but are utilizing their language and area skills improved outreach and dissemination of information about foreign languages and area studies improved instructional materials decreased shortage of foreign language/area studies experts in public service and academia

17. 17 What is the primary purpose of Title VI?

18. 18 Title VI Program Purpose Higher Education Act (Title VI, Section 601(b) ( b) PURPOSES- The purposes of this part are-- (1)(A) to support centers, programs, and fellowships in institutions of higher education in the United States for producing increased numbers of trained personnel and research in foreign languages, area studies, and other international studies; (B) to develop a pool of international experts to meet national needs; (C) to develop and validate specialized materials and techniques for foreign language acquisition and fluency, emphasizing (but not limited to) the less commonly taught languages; (D) to promote access to research and training overseas; and (E) to advance the internationalization of a variety of disciplines throughout undergraduate and graduate education;

19. 19 Title VI Program Purpose (cont’d) Higher Education Act (Title VI, Section 601(b)— (cont’d) 2)to support cooperative efforts promoting access to and the dissemination of international and foreign language knowledge, teaching materials, and research, throughout education, government, business, civic, and nonprofit sectors in the United States, through the use of advanced technologies; and 3)to coordinate the programs of the Federal Government in the areas of foreign language, area studies, and other international studies, including professional international affairs education and research.

20. 20 Conclusion Satisfying our policymakers include: Managing your program to its maximum effectiveness Telling your story effectively Sustaining your project