1. EVALUATING US-CHINA CULTURAL EXCHANGE PROGRAMS: STRATEGIES TO REDUCE BIASES Wenjie Tang Oct 19, 2013 1

2. Background of US-China cultural exchange programs  “100K Strong Initiative” 12x Chinese study in the US than Americans study in China (The 100,000 Strong Foundation) Aims to “increase dramatically the number and diversify the composition of American students studying in China.”  Evaluation needs 2

3. What is unique about evaluating US- China cultural exchange programs?  Broad concepts to clarify for each program  “Culture”, “mutual understanding”, etc.  The identity of the evaluator may help and bias evaluations 3

4.  Which picture do you prefer? 4

5. How to reduce biases in evaluation? Evaluation design – logic model Evaluation methods – mixed methods Data collection Data interpretation Self-reflection 5

6. Evaluation design 6  RFP: “program outcomes, strengths, weaknesses, and opportunities”  Evaluation questions:  To what extent did participant engage in the program?  To what extent did the program reached its initial objectives? What are the objectives?  What are the strengths and weaknesses of the program design and execution? What are the opportunities for program improvement?

7. Evaluation design  Identify outcomes valued by stakeholders  Review program documents  Encourage program designer and implementer to speak to the largest extent possible Logic modeling 7

8. Choose Methods – Mixed Methods Logs Examine participation Surveys Evaluate participants’ satisfaction on each program components Collect data on outcomes identified in logic model Interviews Help interpret survey rating scores Explore opportunities for program improvement Document analysis Conduct content analysis on lesson plans Examine utilization of content knowledge learned during the program 8

9. Collect data  Address social desirability in data collection  Remind participants of the evaluator’s role  Keep reasonable distance between interviewee and interviewer 9

10. Interpret data  Use interview findings to explain survey data  Have team members with different backgrounds to interpret the same data  “Beijing was overwhelming”  Talk about limitations of evaluation methods 10

11. Self-reflection  My self-reflection questions: Did I lay too much emphasize on my own personal interest in writing evaluation questions and identifying program outcomes? Did I judge a participant’s response? Did I make a participant feel reluctant to speak about negative issues about China and the program? Did I hint participants to give answers that I want to (or participants want me to) hear? Did I only highlight findings that I like in data interpretation? 11

12. Recap  There is no value-free evaluation  Examine evaluator’s own values  Rely on rigorous design and self-reflection to reduce biases 12

13. Comments? Questions?  Thank you! Wenjie Tang Northern Arizona University wenjie.tang@nau.edu 13