1. Access to the World and Its Languages www.ed.gov/ope/iegps Evaluation Access to the World and Its Languages www.ed.gov/ope/iegps I N T E R N A T I O N A L E D U C A T I O N P R O G R A M S S E R V I C E CIBE, FLAS, IIPP, LRC, NRC Technical Assistance Workshop February 2-3, 2009

2. Access to the World and Its Languages www.ed.gov/ope/iegps Title VI 50 th Anniversary Update Madeline Albright is the Keynote speaker Please make separate reservations –Conference registration –Hotel registration –Rooms are going quickly Contacts Rob Glew and Tomas Hult, Michigan State University http://titlevi50th.msu.edu

3. Access to the World and Its Languages www.ed.gov/ope/iegps Evaluation Love

4. Access to the World and Its Languages www.ed.gov/ope/iegps Panelists Eleanor Briscoe, Senior Budget Analyst John Clement, Director, Strategic Planning Services Susan Lehmann, Evaluation Analyst, FIPSE Karla Ver Bryck Block, Senior Advisor, IEPS

5. Access to the World and Its Languages www.ed.gov/ope/iegps Why??? ED believes in and supports high quality program evaluations to help inform budgetary and program decisions ED Strategic Plan –Goal 3, Objective 1: Increase success in and completion of quality postsecondary education. Strategy 3. Prepare more graduates for employment in areas of vital interest to the United States, especially in critical-need languages, mathematics, and the sciences.

6. Access to the World and Its Languages www.ed.gov/ope/iegps Administration Update “Accountability and Transparency” GPRA continues PART may or may not continue but the need to measure program effectiveness remains

7. Access to the World and Its Languages www.ed.gov/ope/iegps Congress Clear interest in program evaluations Directed a $1.5 million review of Title VI and Fulbright-Hays programs by the National Research Council of the National Academies –International Education and Foreign Language – Keys to Securing America’s Future (March 2007)

8. Access to the World and Its Languages www.ed.gov/ope/iegps Keys to Securing America’s Future National Academies recommendation –The Department of Education should commission independent outcome and impact evaluations of all programs every 4 to 5 years

9. Access to the World and Its Languages www.ed.gov/ope/iegps IEPS Program Level Initiatives Evaluations are underway with AIR –Language Resource Centers –Fulbright-Hays Group Projects Abroad – Overseas Intensive Language Projects Analysis of the International Research and Studies Program is concluding National Resource Centers are next Future plans include Centers for International Business Education

10. Access to the World and Its Languages www.ed.gov/ope/iegps IRIS IEPS gathers information via IRIS that is used for program evaluations Breakouts for 4 programs today (CIBE, LRC) and tomorrow (NRC, FLAS) to discuss: –Current and future program performance and efficiency measures –Changes to IRIS for the next clearance –Evaluation efforts –Trivia, prizes, chocolate!!!

11. Access to the World and Its Languages www.ed.gov/ope/iegps Keys to Securing America’s Future National Academies Recommendation –The Department of Education should work with universities to create a system of continuous improvement for Title VI and Fulbright-Hays programs. The system would help develop performance indicators and other improvement tools and should include networks of similar centers and university officials with overall responsibilities in language, area, and international studies.

12. Access to the World and Its Languages www.ed.gov/ope/iegps Your Project Think… –Goal-oriented –Data driven ED does not have specific models, timelines, budgets, etc for your evaluation plans. Develop a plan that serves your needs.

13. Access to the World and Its Languages www.ed.gov/ope/iegps Performance Objectives & Measures A measurable indication used to determine how well goals are being met –How will progress be measured? –How much progress will constitute success? –How will it be known if an objective or part of an objective has been achieved? Are we measuring process or outcome?

14. Access to the World and Its Languages www.ed.gov/ope/iegps Objective & Measure Types A specific achievement/accomplishment Change in a level of achievement/accomplishment Achievement/accomplishment (or change) compared with other students or institutions

15. Access to the World and Its Languages www.ed.gov/ope/iegps Components of Objectives & Measures What will change How much change is expected Who will achieve the change When the change will take place

16. Access to the World and Its Languages www.ed.gov/ope/iegps Performance Objective Example “20% (how much) of local businesses with more than 200 employees (who) will open an office in a foreign country or enter into a joint venture with a foreign company (what) within two years (when).” Help local businesses to internationalize

17. Access to the World and Its Languages www.ed.gov/ope/iegps Performance Objective Example Teacher workshop – Developing Materials for LCTLs “50 (how much) LCTL teaching professionals (who) will complete LCTL workshop (what) annually (when).” “90% (how much) of LCTL workshop participants (who) respond with 4 or higher on survey (what) one year after completing program (when).”

18. Access to the World and Its Languages www.ed.gov/ope/iegps Performance Measure Example Revise public health curriculum to enhance international knowledge “The required MPH courses (who) will be revised (what) to cover the 10 important international health areas (how much) by next fall (when).”

19. Access to the World and Its Languages www.ed.gov/ope/iegps Performance Measure Example Improve enrollment in summer language institute “Undergraduate student enrollment in the summer language institute (who) will increase (what) by five percent (how much) next summer (when).”

20. Access to the World and Its Languages www.ed.gov/ope/iegps Improving Performance Measures Original Performance Measure: Provide overseas internships. What/Who: –Provide dynamic student internships with partners in a foreign country. When: –Provide dynamic student internships with partners in a foreign country each semester. How Much: –20 students will engage in dynamic student internships with partners in a foreign country each semester.

21. Access to the World and Its Languages www.ed.gov/ope/iegps Improving Performance Measures Original Performance Measure: Enhance Business Resource Center. What/Who: –Enhance Business Resource Center to advise businesses on international trade regulations. When/How Much: –Business Resource Center documents and staff will be updated on new international trade regulations within one week of passage.

22. Access to the World and Its Languages www.ed.gov/ope/iegps Remember!!! Activities are not performance measures!

23. Access to the World and Its Languages www.ed.gov/ope/iegps Resolving the “Activity- as-Measures” Dilemma “Create opportunities for local faculty to strengthen area skills.” (activity) –Create “visiting” faculty training programs and train 2 per year. (outcome) “Enhance study abroad.” (activity) –Increase number of students studying abroad by X% each year. (outcome)

24. Access to the World and Its Languages www.ed.gov/ope/iegps Summary: Good Objectives & Measures Each objective should have one or a few specific performance measures to demonstrate how progress will be measured toward meeting the objective More objectives and measures ≠ better objectives and measures! Ensure data is collectible Quantify and define success

25. Access to the World and Its Languages www.ed.gov/ope/iegps Summary: Strong Evaluation Partner with a competent evaluator Develop quantifiable objectives and measures Measure what is important Measure what can be changed Get frequent progress reports Keep key stakeholders informed of results Use results to adjust project regularly Keep analyzing…