1. Closing the Communication Gap Between Undergraduates and Mathematics Professors Daniel Villarreal November 24, 2009

2. Presentation Format Basics about the project Background Linguistics info Mini literature review Methodology Results/Discussion Homework!

3. Project Basics

4. Conducted in support of an Honors thesis in Linguistics for the 2009-2010 academic year (Committee: Charity Hudley, Taylor, Li) Supported by a Dintersmith Fellowship Social-science-grounded project The inspiration…

5. James Villarreal

6. Background Linguistics Info

7. Terminology Syntax Semantics Phonetics/Phonology Prosody Pronunciation – Includes phonetics, phonology, and prosody

8. Terminology Accent – The way a person pronounces the words in a language Dialect – A variety of the language spoken by some pre- defined group (regional, social, ethnic) Ideolect – The variety of the language particular to a single person

9. Languages ~ Species A good way to conceptualize linguistic variation is to consider biodiversity – Example: Dogs Just as there’s no one canonical “dog”, there’s no canonical “English” It is possible, however, to define a “Standard”

10. Linguistic Principles Universality of Accentedness Rule-based Accents – William Labov 1969: “The Logic of Nonstandard English” In other words, no accent is inherently inferior to any other But some accents may be harder for a speaker of another dialect to understand When that “different dialect” is Standard American, that accent is “incomprehensible”

11. Mini Literature Review

12. Backstory Beginning in the 1980s, the majority of American-born STEM (Science, Technology, Engineering, and Mathematics) graduates began to choose jobs in industry rather than academia (Mooney, 1989) By 1989, more than half of STEM degree recipients were foreign-born (ibid.)

13. The “Foreign TA Problem” In 1984, the linguist Kathleen Bailey identified what she called the “Foreign TA Problem”: “the communicative difficulties engendered by [the interaction between non-native speaking teaching assistants and their students]” (Bailey, 1984, p. 3) In a 1980 study of University of Minnesota undergraduates, almost half reported that having a NNS TA had hurt the quality of a course they had taken, whereas only 9% believed that an NNS TA had helped (cited in Bailey, 1984)

14. The Communication Gap A 1989 study examined the effects of instructor gender, student SAT score, class term, age, international TA, and textbook on undergraduates’ test scores in a macroeconomics survey course. – Of these, no variable was responsible for a greater drop in scores than was the presence of an international TA (Watts & Lynch, 1989) Anecdotal problems abound

15. The Communication Gap, Cont’d So it’s just a matter of the professor speaking in a way that the students can’t understand, right? WRONG Rubin, 1999 Prof. Li’s California story Thus, in my formulation, the communication gap consists not only of actual misunderstanding, but also of bias

16. English Language Learners The older a second language learner is at the time of learning a language, the more difficult it becomes to make one’s accent resemble a native accent (Gass & Selinker, 2001) English language learners can have great difficulties pinpointing the source of accent- related communication breakdowns (Derwing, 2003)

17. Professor Training Lots of universities, especially state universities, mandate that new hires (especially TAs) whose first language is not English pass TSE, TOEFL, etc. (Cassell, 2007; Plakans, 1997; Davies, Tyler, and Koran, 1989) – Several researchers have doubts as to the efficacy of these tests in evaluating classroom readiness (Tyler, 1992; Young, 1989; Halleck and Moder, 1995)

18. Student-Centered Research Donald Rubin: “Support for ITAs (and also continuing support for non-native English speaking faculty members) is key, and much progress has been made in many fine programs on that score. But also key is attention to undergraduates' listening abilities. Very few--if any--programs exist to support undergraduates as listeners of World Englishes.” (qtd. in Gravois, 2005)

19. Math is Different Byrd and Constantinides, 1992: “so many of our early assumptions about teaching (based on teaching styles preferred in ESL) do not hold for the teaching of mathematics.” (p. 166) Topic familiarity influences comprehension of non-native speakers to a greater degree than even accent familiarity (Gass and Varonis, 1984) Math anxiety

20. The Need for This Research In summary, there are several factors that make this research unique and necessary: – Focuses on interactions with professors, not TAs – Shifts some of the burden of communication to students – Focuses on the mathematics classroom – Attempts to address both components of the communication gap: bias and genuine misunderstanding

21. Methodology

22. Null hypothesis: Training program not effective in aiding students’ comprehension Alternative hypothesis: Training program IS effective Population of interest: William and Mary undergraduates Sampling method: Random…ish – In social science research, recruitment is often the hardest part!

23. Recruitment Hoped to get students to participate out of the goodness of their hearts – Help advance research that improves classroom interactions between professor and student – Help a fellow student out with research But in the end… Fortunately, the Charles Center agreed to reimburse Cheese Shop-related expenses

24. Testing Sessions Six testing sessions in Dulin Learning Center (in Swem) from October 3-6 Five parts: – Consent Form/Assignment – Lesson 1 + Assessment 1 – Inter-lesson Module – Lesson 2 + Assessment 2 – Linguistic Profile Questionnaire

25. Testing Sessions Six testing sessions in Dulin Learning Center (in Swem) from October 3-6 Five parts: – Consent Form/Assignment – Lesson 1 + Assessment 1 – Inter-lesson Module – Lesson 2 + Assessment 2 – Linguistic Profile Questionnaire Three different inter-lesson modules: the accent training program, a Control group, and another variable group Thus, the variable of interest was the difference in scores: A 2 – A 1

26. Experimental Design Check But what if Assessment 2 were significantly harder (or easier) than Assessment 1? We need to randomize the order of presentation to control for difficulty effects I ended up creating three lesson videos for the project, so there were ?? different orders for presenting two videos What do I mean by “creating lesson videos”? 6

27. Lesson Videos In my original design, I would record professors teaching mini-lessons (audio only) – Confidentiality – Bias effects (Rubin 1999) I would then make Flash video animations of these lessons, simulating professors writing on a chalkboard But recruiting professors turned out to be a lot harder than I thought

28. Webcasts On the advice of a friend, I looked around to see if I could find any webcasts or open courseware to fit my needs – At least moderate accent – Not too high-level, not too low-level – Not too long – Derivative works permitted This turned out to eliminate just about everything on the Web!

29. Lesson Videos Using audio from the Discrete Structures course, I chose three lessons: – Permutations and Combinations – The Proof That the Square Root of Two is Irrational – Relations and the Cartesian Product I then created three “videos” on PowerPoint, with text appearing on the screen as Prof. Kamala spoke it (chalkboard simulation)

30. Inter-Lesson Modules Three groups, as defined by the inter-lesson module they viewed: Control, Bias, and Training – Training module instructed students on accent ideology, then specific features of Prof. Kamala’s accent – Control module consisted of inert materials – Bias module contained an article, blog post, and table of professor ratings meant to simulate campus conversation by presenting professors in a less-than-favorable light

31. Testing Groups Each participant, then, had two different treatment factors: their inter-lesson module and the videos they viewed (and in what order) – 3 inter-lesson modules – 6 permutations of 2 videos These were independently randomized This meant that there were 6 x 3 = 18 testing groups

32. Testing Groups Examples of testing groups: – Square Root of 2 > Bias > Perm-Comb – Relations > Control > Square Root of 2 – Relations > Training > Square Root of 2 These would have been TERRIBLE names Actual names for the above groups: – Iceberg – Sunflower – Fuchsia

33. Web-Based Content Key parts of the project were hosted on the Internet at my William and Mary webspace: http://djvill.people.wm.edu http://djvill.people.wm.edu This helped avoid the need to download lesson videos onto Learning Center computers In addition, this makes it easy to turn the experiment into a fully web-based one

34. Testing Sessions After students handed back consent forms, I used a random number generator to determine their testing group Students first loaded the URL http://djvill.people.wm.edu/%%_lesson.html http://djvill.people.wm.edu/%%_lesson.html This redirects the student to the page containing the first lesson video for that group At the end of the video, the student receives an assessment from a moderator

35. Testing Sessions, Cont’d Once the student completes the assessment, they are instructed to load http://djvill.people.wm.edu/%%_mid.html http://djvill.people.wm.edu/%%_mid.html This redirects the student to their group’s inter-lesson module Once that is complete, the student is instructed to load http://djvill.people.wm.edu/%%_lesson2.html http://djvill.people.wm.edu/%%_lesson2.html This redirects the student to their lesson 2

36. Testing Sessions, Cont’d Again, the student receives an assessment corresponding to lesson 2 Once the student is done with the second assessment, they are instructed to load http://djvill.people.wm.edu/LPQ.html http://djvill.people.wm.edu/LPQ.html This page is a Linguistic Profile Questionnaire (on Google Docs) – How many Math classes taken in college – Where parents are from – Childhood exposure to accents

37. Discussion Sessions In addition to testing sessions (collecting mostly quantitative data), I held discussion sessions to collect qualitative data Nine discussion sessions between October 17-20, 60-90 minutes apiece Between 4 and 11 participants in each session Segregated by testing group Sandwiches!

38. Discussion Sessions, Cont’d Three parts to session: – Reviewed ground rules and human subjects protections – Actual discussion itself – Nuts and bolts of project Discussion questions: – For starters, how many classes, if any, have you taken with a foreign-born professor? Have any been Math classes? – Did his or her accent ever hurt your understanding of the material? – Do you talk about professors’ accents a lot with your friends?

39. Discussion Questions More discussion questions: – Have you ever dropped a class or even changed your academic plans because the professor had a foreign accent? What about other students you know? – What do you think impedes communication between students and professors the most, regardless of accent? – What do you wish professors (or even the College) would do to deal with the issue of the communication gap between undergraduates and mathematics professors? – Do you feel that you gained anything from this process? – Do you think that you are now at least somewhat better equipped to deal with issues of accent in your instructors?

40. Discussion Session Ending In discussing methodology, I finished by walking the groups through the Training module This led to the final question: – Do you think that programs such as these would be effective in dealing with the communication gap?

41. Results/Discussion

42. Quantitative Results I haven’t yet been able to do a full statistical analysis of quantitative data – (cough, cough, Complex Analysis) However, preliminary analyses are not encouraging Variable Macro-group N Mean StDev Minimum Median Maximum IQR Improvement Bias 23 -1.09 6.99 -13.00 -2.00 10.00 12.00 Control 28 0.79 6.48 -15.00 0.50 12.00 9.50 Training 29 -0.79 6.24 -12.00 0.00 12.00 6.00

44. T-Tests Remember, our null is that μ C = μ B = μ T (But we want μ B < μ C < μ T ) Two-sample T for Improvement Macro-group N Mean StDev SE Mean Control 28 0.79 6.48 1.2 Training 29 -0.79 6.24 1.2 Difference = mu (Control) - mu (Training) Estimate for difference: 1.58 95% CI for difference: (-1.80, 4.96) T-Test of difference = 0 (vs not =): T-Value = 0.94 P-Value = 0.353 DF = 54 Two-sample T for Improvement Macro-group N Mean StDev SE Mean Bias 23 -1.09 6.99 1.5 Control 28 0.79 6.48 1.2 Difference = mu (Bias) - mu (Control) Estimate for difference: -1.87 95% CI for difference: (-5.71, 1.96) T-Test of difference = 0 (vs not =): T-Value = -0.98 P-Value = 0.330 DF = 45 Two-sample T for Improvement Macro-group N Mean StDev SE Mean Bias 23 -1.09 6.99 1.5 Training 29 -0.79 6.24 1.2 Difference = mu (Bias) - mu (Training) Estimate for difference: -0.29 95% CI for difference: (-4.05, 3.46) T-Test of difference = 0 (vs not =): T-Value = -0.16 P-Value = 0.875 DF = 44 Fail to reject null hypothesis

45. ANOVAs Source DF SS MS F P Macro-group 2 54.3 27.1 0.63 0.534 Error 77 3299.3 42.8 Total 79 3353.5 S = 6.546 R-Sq = 1.62% R-Sq(adj) = 0.00% Individual 95% CIs For Mean Based on Pooled StDev Level N Mean StDev ---------+---------+---------+---------+ Bias 23 -1.087 6.986 (-------------*------------) Control 28 0.786 6.483 (-----------*-----------) Training 29 -0.793 6.241 (-----------*-----------) ---------+---------+---------+---------+ -2.0 0.0 2.0 4.0 Pooled StDev = 6.546

46. ANOVAs Source DF SS MS F P Lesson Order 5 1995.4 399.1 21.74 0.000 Error 74 1358.1 18.4 Total 79 3353.6 S = 4.284 R-Sq = 59.50% R-Sq(adj) = 56.76% Individual 95% CIs For Mean Based on Pooled StDev Level N Mean StDev ---------+---------+---------+---------+ PC-R2 12 -5.000 3.464 (----*----) PC-Rel 16 1.563 4.647 (---*---) R2-PC 14 6.000 4.224 (----*----) R2-Rel 10 5.200 3.736 (----*-----) Rel-PC 12 -0.667 4.755 (----*----) Rel-R2 16 -7.438 4.442 (---*---) ---------+---------+---------+---------+ -5.0 0.0 5.0 10.0 Pooled StDev = 4.284

47. Why these results? Major discrepancy in difficulty of tests Not enough participants for the number of testing groups Training module needs improvement Individual 95% CIs For Mean Based on Pooled StDev Level N Mean StDev --+---------+---------+---------+------ PermComb 28 12.714 3.184 (----*----) Relations 28 14.857 2.663 (----*----) Root2 24 8.375 4.009 (-----*----) --+---------+---------+---------+------ 7.5 10.0 12.5 15.0

48. In the end… That’s just how it goes with research in the social sciences, especially with research on humans A project like this is often the first in a series of many Plenty of qualitative data gathered from Linguistic Profile Questionnaire and discussion sessions (about 6 ½ hours worth of participants’ responses)

49. Questions? Any questions or comments?

50. Homework As part of your homework, you will simulate being a participant in a testing session http://djvill.people.wm.edu/ _lesson.html and follow the instructions from there When you’re done Assessment 1, go to http://djvill.people.wm.edu/ _mid.html Finally, there’s a discussion question for you to answer on the BlackBoard discussion board (not for extra credit)

51. Works Cited (in order of appearance)  Mooney, C. J. (1989, January 25). Uncertainty is Rampant as Colleges Begin to Brace for Faculty Shortage Expected to Begin in 1990’s. Chronicle of Higher Education, A14-A17.  Bailey, K. M. (1984). The “Foreign TA Problem”. In K. Bailey, F. Pialorsi, J. Zukowski/Faust (Eds.), Foreign Teaching Assistants in U.S. Universities (3-15).  Watts, M., & Lynch, G. (1989). The Principles Course Revisited. The American Economic Review, 79, 236-241. Retrieved from http://www.jstor.org/stable/1827763. http://www.jstor.org/stable/1827763  Gass, S. M., & Selinker, L. (2001). Second Language Acquisition: An Introductory Course (2nd ed.). Mahwah, New Jersey: Lawrence Erlbaum Associates.  Derwing, T. (2003). What Do ESL Students Say About Their Accents? The Canadian Modern Language Review, 59, 547- 566.

52. Works Cited (in order of appearance)  Cassell, Edith Camilla (2007). Understanding community linguistic diversity: An ecological approach to examining language use patterns of international graduate students. Ph.D. dissertation, Purdue University, United States -- Indiana. Retrieved June 1, 2009, from Dissertations & Theses: Full Text database. (Publication No. AAT 3287301).  Plakans, B. (1997). Undergraduates' Experiences with and Attitudes toward International Teaching Assistants. TESOL Quarterly, 31. Retrieved April 27, 2009, from http://www.jstor.org/stable/3587976.  Davies, C. E., Tyler, A., & Koran, J. J., Jr. (1989). Face-to-Face with English Speakers: An Advanced Training Class for International Teaching Assistants. English for Specific Purposes, 8, 139-153.

53. Works Cited (in order of appearance)  Tyler, A. (1992). Discourse Structure and the Perception of Incoherence in International Teaching Assistants' Spoken Discourse. TESOL Quarterly, 26, 713-726. Retrieved from http://www.jstor.org/stable/3586870.  Young, R. (1989). Introduction. English for Specific Purposes, 8, 101- 107.  Halleck, G. B., & Moder, C. L. (1995). Testing Language and Teaching Skills of International Teaching Assistants: The Limits of Compensatory Strategies. TESOL Quarterly, 29, 733-758.  Gravois, J. (2005, April 8). Teach Impediment. Chronicle of Higher Education, 51, A10. Retrieved from http://chronicle.com/free/v51/i31/31a01001.htm. http://chronicle.com/free/v51/i31/31a01001.htm  Byrd, P., & Constantinides, J. (1992). The Language of Teaching Mathematics: Implications for Training ITAs. TESOL Quarterly, 26, 163-167. Retrieved from http://www.jstor.org/stable/3587384.  Gass, S., & Varonis, E. (1984). The effect of familiarity on the comprehensibility of nonnative speech. Language learning, 34, 65-89.