1. Evaluating Your IEPS Grant Dr. Susan Lehmann Education Research Analyst, FIPSE Presented at the 2009 International Education Programs Service Technical Assistance Workshop held on 2/2/2009 in Washington, D.C.

2. Your job, and ours, is to turn vague, general program goals into specific benchmarks that you hope to achieve.

3. For each award you should ask: What knowledge do you expect fellows to acquire? What gains in skills do you expect fellows to make? What counts as a gain in professional skills and how will you measure gain?

4. Things to Evaluate in Title VI Programs

5. 1. Are you advancing the internationalization of academic disciplines?

6. You are probably keeping track of course materials that are being produced as a result of the grant.

7. Can you tell us the number of: new interdisciplinary courses that have been adopted, field-specific courses that have added new units on internationalization using your materials, collaborative faculty projects that have resulted from your award?

9. 58% of students studying abroad on FIPSE Programs learned about challenges and problems that face Americans in their field who choose to work abroad. The American Councils study found that:

10. 56% of students studying abroad on FIPSE programs learned about international laws and regulations governing work in their field. The American Councils study found that:

11. By having an alumni newsletter you could collect information that would allow you compile a list like this:

12. Recent Selected Alumni Publications Building Democracy in Contemporary Russia: Western Support for Grassroots Organizations. 2003. The North Korean Nuclear Program: Security, Strategy, and New Perspectives from Russia. 2000. America’s New Allies: Poland, Hungary, and the Czech Republic in NATO. 1999. Enlarging Europe: The Industrial Foundations of a New Political Reality. 1998. Guide to Scholars of the History and Culture of Central Asia. 1995.

13. 2. Can you document gains in foreign language skills?

14. Ways to Track and Discuss Language Gain Pre and post fellowship language testing using standardized tests. Pre and post language testing using a specialized list of terms or concepts that you have designed. Self-reported ratings of language speaking, reading, writing, and listening ability. Self-reported ability to use foreign language primary source materials in various contexts. Self-reported ability to create original materials in the foreign language – surveys, interviews, research presentations.

17. 3. Are you tracking postgraduate employment, education, and training?

23. 4. Can you document that you are producing experts in areas of vital interest to the U.S.?

24. By looking up alumni names in Lexus/Nexus or Thomas at www.loc.gov/thomas, you could make up a chart like this: www.loc.gov/thomas

26. Title VIII Alumni Testimony Before House Committees

27. By collecting information on the affiliation of people who use your centers, you could re- package the information in a chart like this:

28. Title VIII Award Recipients with U.S. Print Media Affiliations (Data based on a review of 3,174 awards made between 1985-2003) Forbes Magazine The Washington Post Princeton Research Forum Journal of International Wildlife Law & Policy East European Politics & Society Post-Communist Studies The Ukrainian Quarterly

29. By asking fellows whether they subsequently shared information that they learned as a result of your award, you might find:

30. Diffusion Effect: 1  36 We found that every person who went to the U.S. on exchange, subsequently trained more than 36 people (superiors, co-workers, professional colleagues, government officials, and young people) when they returned home. 2003 Community Connections Alumni Survey, (N=5,429).

31. Assessment requirements did not start under the last administration and they are not going to go away. As taxpayers you should demand value for your money. Please assist us in making the case for the value of your project and your programs in general.

32. Measuring language gain efficiently and inexpensively. Measuring language gain in a manner that allows for comparisons across languages. Tracking students once they graduate. Documenting the long-term career impact of your program on fellows. Documenting the long-term policy impact of your programs. Title VI Evaluation Challenges

33. Don’t leave us in the dark! Give us some suggestions about what you could reasonably assess. We could come up with an endless list of data that you need to submit. But we respect your judgment as educators. We would rather make assessment a collaborative process.

34. Set out to obtain data that move knowledge forward by documenting what works, what does not work, and why. At the end of the project, you may conclude that your original model didn’t get you the results you had hoped for. If so, tell us what you would change next time – the schedule, the activities, the partners, the type of students that your program targets, etc. YOU WILL NOT BE PUNISHED FOR PROVIDING AN HONEST ASSESSMENT. YOU WILL EDUCATE US ABOUT WHAT WORKS.

35. 1: Refining the Project 2: Identifying the Data to Collect 3: Setting the Timeline 4: Identifying Interested Parties to Share Results With A Good Evaluation Plan Has 4 Parts

36. Part 1: Refining the Project: Things to Consider 1-3 main themes, a.k.a “project goals”. 1-2 key questions per theme. These are your “project objectives”. What will be documented or measured so that you will be able to determine whether you met your objectives? Speed and extent of change.

37. Part 2: Data Collection Baseline measures. Data collection instruments. Who is the respondent, interview subject, etc. Who is changing? Comparison or control group, if possible.

38. Part 3: Timeline When will your evaluation instruments be drafted? When will you collect your data? When will you analyze your data? Will your evaluation results provide feedback during the project that will enable you to modify project activities? Which activities might you modify? When will your written evaluation findings be ready for an outside audience?

39. Part 4: Dissemination: Who should hear about your findings? On campus? In the local community? At similar institutions? Professional groups & associations? Professional colleagues? Local, state, and federal agencies & officials? Interested others?

40. Now that you have identified who should know about your project, review the project objectives and data collection plan one more time. Did you address issues of high interest to the people you intend to share results with? Are you collecting data that will be convincing to a skeptic? Will the data you collect help you rule out alternative explanations for any changes you notice?

41. Role of the Outside Evaluator Assists the Project Director in completing the initial evaluation plan. Assists the Project Director in designing the evaluation instruments. This includes making the PD aware of existing evaluation instruments and comparison data available in the public domain.

42. Together the Project Director and Outside Evaluator Work together on the analysis of assessment data that are collected. Decide how you divide up data entry and analysis tasks.

43. The Project Director’s Evaluation Responsibilities Include: Collecting baseline data. Collecting project evaluation data. Implementing any changes to the project as a result of the preliminary evaluation. Disseminating evaluation results.