1. A Five-year project CALL Project Leads to the Analysis of Motivation with an Eye on Curriculum Reform By David W. Reedy and James W. Pagel Associate Professors School of Science and Engineering Aoyama Gakuin University Tokyo, Japan 14th International CALL Conference 08/20/2010

2. This presentation deals with: Our previous study on implementing an e- learning component in all English classes offered in our department; The current investigation of whether entrance routes have any affect on performance on post tests two years after university entrance, and whether there is any correlation between completion of the e-learning project and an increase in test scores.

3. A basic explanation may be necessary. This research is the culmination of the authors’ efforts to implement and e-learning component in all English classes in their department to comply with MEXT mandates. It is our goal to improve the department’s English curriculum.

4. The Ministry of Education, Culture, Sports, Science and Technology (MEXT) has set various targets. In 2001 the Rainbow plan was announced: Smaller classes, easier-to-understand lessons and Integration of IT into the classrooms In 2004 the agency stated its goal of having all eligible High School graduates be accepted to university In 2005 in its initiatives for higher education, called for a fierce competition among private universities, and for globalization, which meant: 1. Accepting more students from overseas; 2. Sending more students abroad; 3. Opening satellite campuses; 4. Offering joint degree projects with other universities.

5. Globalization is the key word, but… Many universities* reported that they lacked the money the staffing the infrastructure to implement e-learning on a large scale basis * 60 percent answering the survey by the National Institute of Multimedia Education Source: Japan Times “E-Learning: Making Inroads into Nation’s Colleges” May 7, 2007

6. Japanese students are not so interested in going overseas. Students are content with their situation “at home.” Internet research is sufficient. (And companies can no longer afford to send its workers overseas as in the past.) Students fear they lack the ability to succeed in foreign countries. Reverse Japan’s Insularity. Japan Times April 8, 2010 Forty-nine percent of new hires will not go abroad. Jiji Press August 13, 2010

7. This is the start of the study. Integrate commercial software in all departmental English classes. But only 36.3% of Japanese universities had offered e- learning classes as of 2006. Purchase of commercial software was “one-time only.” Analyze the merits of such software.

8. In short, we succeeded.

9. We needed to know how students reacted to such software.

10. Our survey results were inconclusive. 938

11. Why did they stop? 9a. Why did you give up? 2006 2007 Hard to maintain motivation 90 (40%) 91 (54%) Takes too much time 72 (32%) 33 (20%) “Typically, drop-out rates are reported to be 80 percent or more in e-learning environments where little or no classroom support is available. “ Evolution of CALL” by Lance Knowles http://www.dyned.com/pdf/Teacher-Guides/TGTHEORY.PDF

12. Here is some more information. 1. Why didn’t you try the material? 2006 2007 Too busy 454 (61%) 228 (65%) Bonus points 110 (15%) 25 (7%) too low Dislike English 68 (9%) 34 (9%)

13. Our target became how to identify students who completed the e-learning component and their motivation. Break down the students by department. (2010) Seek other ways of finding correlations. Reinvent the student survey.

14. We broke down students by department. N=564

15. So you are now up to speed. What is next? We could analyze completion of the e- learning component other ways. And why …(problem: survey answers were anonymous.) We could study how students are admitted to the university to better understand motivational factors in the study of English.

16. There are ten ways to get accepted. Departmental Exam Day One Passing on first cut-off 224 (39.7%) Departmental Exam Day One Passing on second cut-off 45 (8%) Departmental Exam Day Two Passing on first cut-off 100(17.7%) Departmental Exam Day Two Passing on second cut-off 4 (0.7%) National Exam First Term 20 (3.6%) National Exam Second Term + Departmental Exam 21 (3.7%) Internal High School Evaluation 30 (5.3%) Recommendation 120 (21.3%) International Students Exam Foreign Students 2 (0.4%) International Student Exam Students attending international high school 1 (0.2%)

17. We considered success on the basis the following. TOEFL ITP Pretest results averages Posttest results averages Best score change averages Percentage of students whose scores increased Percentage of students whose scores decreased e-learning Percentage of freshmen who completed the requirement Percentage of freshmen who completed the bonus component Percentage of sophomores who completed the requirement Percentage of sophomores who completed the bonus component

18. TOEFL ITP pretest scores University EntranceMeanSD 1. Departmental Exam Day One: Passing on first cut-off 422.432.62 2. Departmental Exam Day One: Passing on second cut-off 411.430.18 3. Departmental Exam Day Two: Passing on first cut-off 41626.49 4. Departmental Exam Day Two: Passing on second cut-off 39417.1 5. National Exam 442.630.24 6. National Exam + Departmental Exam 417.230.61 7. Internal High School Evaluation 405.313.49 8.Recommendation 398.729.32 9. International Students Exam: Foreign Students 383.53.5 10. International Student Exam: Japanese students attending foreign high school 5400

19. Pretest and Posttest scores University Entrance Pretest Mean SDPosttest Mean SD 1. Departmental Exam Day One Passing on first cut-off 422.432.62410.841.8 2. Departmental Exam Day One Passing on second cut-off 411.430.18392.745.1 3. Departmental Exam Day Two Passing on first cut-off 41626.49388.638.8 4. Departmental Exam Day Two Passing on second cut-off 39417.138913.4 5. National Exam 442.630.24428.144.8 6. National Exam + Departmental Exam 417.230.61405.439.9 7. Internal High School Evaluation 405.313.49401.435.4 8. Recommendation 398.729.3239036.5 9. International Students Exam Foreign Students 383.53.5XX 10. International Student Exam Japanese students attending foreign high school 54005900 Total412.6232.65401.2142.03

20. The Spearman rank-order correlation coefficient calculates to.82. University EntrancePretest RankingPost test Ranking 10.International Student Exam Japanese students attending foreign high school 1 (540)1 (590) 5. National Exam 2 (442.6)2 (428.1) 1. Departmental Exam Day One Passing on first cut-off 3 (422.4)3 (410.8) 6. National Exam + Departmental Exam 4 (417.2)4 (405.4) 3. Departmental Exam Day Two Passing on first cut-off 5 (416)9 (388.6) 2. Departmental Exam Day One Passing on second cut-off 6 (411.4)6 (392.7) 7. Internal High School Evaluation 7 (405.3)5 (401.4) 8. Recommendation 8 (398.7)7 (390) 4. Departmental Exam Day Two Passing on second cut-off 9 (394)8 (389) 9. International Students Exam Foreign Students 10 (383.5)XX

21. How did scores change? Entrance RouteAverage TOEFL Change Scored Higher Scored Lower No Change Students Absent 1. Departmental Exam Day One Passing on first cut-off -10 (-2.4%) 64 (39%) 99 (60.4%) 1 (0.6%) 59 (26.8%) 2. Departmental Exam Day One Passing on second cut-off -18.7 (-4.6%) 12 (36.4%) 21 (58.3%) 0 12 (26.7%) 3. Departmental Exam Day Two Passing on first cut-off -27.4 (-6.6%) 14 (23.3%) 46 (76.7%) 2 (3.3%) 38 (38%) 4. Departmental Exam Day Two Passing on second cut-off -5 (-1.3%) 1 (25%) 2 (50%) 1 (25%) 0 (0%) 5. National Exam -14.5 (-3.3%) 3 (18.6%) 12 (75%) 1 (6.3%) 4 (20%) 6. National Exam + Departmental Exam -11.8 (-2.8%) 6 (35.3%) 11 (64.7%) 0 4 (19.1%) 7. Internal High School Evaluation -3.9 (-1%) 6 (28.6%) 12 (57.1%) 3 (14.3%) 9 (30%) 8. Recommendation -12.4 (-3.1%) 39 (37.9%) 58 (56.3%) 6 (5.8%) 17 (14.2%) 9. International Students Exam Foreign Students XX 2 (100%) 10. International Student Exam Japanese students attending foreign high school +50 (+9.3%) 1 (100%) 000 Total-11.4 (-2.76%) 146 (34.68%) 261 (63%) 14 (3.33%) 145 (25.62%)

22. TOEFL achievements at a glance. Entrance Routepretestpost testupdownchange 1. Departmental Exam Day One Passing on first cut-off 33264 2. Departmental Exam Day One Passing on second cut-off 66458 3. Departmental Exam Day Two Passing on first cut-off 59899 4. Departmental Exam Day Two Passing on second cut-off 98723 5. National Exam 22987 6. National Exam + Departmental Exam 44575 7. Internal High School Evaluation 75642 8. Recommendation 87336 9. International Students Exam Foreign Students 10XX 10. International Student Exam Japanese students attending foreign high school 11111

23. Give your eyes a little break!

24. Are you now ready for more? I am David & on vacation

25. Our e-learning program at a glance. 1 st term2 nd term FreshmenRequirement 6 units Bonus Component 30 units SophomoresRequirement 6 units Bonus Component 30 units

26. They always do better as freshmen. Entrance Route1 st term req 1 st term bonus 2 nd term req 2 nd term bonus 1. Departmental Exam Day One Passing on first cut-off 89.7333.3375.8913.39 2. Departmental Exam Day One Passing on second cut-off 91.1134.1571.1122.22 3. Departmental Exam Day Two Passing on first cut-off 8144.446910 4. Departmental Exam Day Two Passing on second cut-off 7566.677550 5. National Exam 78.9526.6763.1610.53 6. National Exam + Departmental Exam 90.4826.3261.904.76 7. Internal High School Evaluation 93.3360.7186.6723.33 8. Recommendation 98.3352.5486.6730.83 9. International Students Exam Foreign Students 0000 10. International Student Exam Japanese students attending foreign high school 1000 Total 89.2235.3473.1417.67

27. Let’s focus on E-learning achievements. Let’s focus on E-learning achievements. Entrance Route1 st yr req 1 st yr bonus 2 nd yr req 2 nd yr bonus 1. Departmental Exam Day One Passing on first cut-off 6646 2. Departmental Exam Day One Passing on second cut-off 4565 3. Departmental Exam Day Two Passing on first cut-off 7478 4. Departmental Exam Day Two Passing on second cut-off 9152 5. National Exam 8787 6. National Exam + Departmental Exam 5899 7. Internal High School Evaluation 3224 8. Recommendation 2323 9. International Students Exam Foreign Students 109 10. International Student Exam Japanese students attending foreign high school 1911

28. Let’s tally up the two groups. Entrance RouteTOEFL total rank e-learning total rank 1. Departmental Exam Day One Passing on first cut-off 26 2. Departmental Exam Day One Passing on second cut-off 75 3. Departmental Exam Day Two Passing on first cut-off 97 4. Departmental Exam Day Two Passing on second cut-off 74 5. National Exam 58 6. National Exam + Departmental Exam 49 7. Internal High School Evaluation 33 8. Recommendation 52 9. International Students Exam Foreign Students 10. International Student Exam Japanese students attending foreign high school 11

29. We drew the following conclusions. Success can’t be determined by comparing entrance route and TOEFL scores. The Spearman rank-order correlation coefficient of TOEFL and e-learning achievements calculates to.44. Students entering via the internal high school evaluation system were most consistent (ranked 3 rd in both categories). More data analysis is necessary.

30. Our current investigations have led us to consider: Limiting comparison to top and bottom scoring students; Looking at individual departments and their students’ pre- test and posttest scores vs e- learning completion rates, considering the types of classes required); Revising the student survey tool to include more questions that would address motivation.

31. References not listed above Ministry of Education, Culture, Sports, Science and Technology. (2001). The education reform plan for the 21st century – the Rainbow Plan. Retrieved from http://www.mext.go.jp/english/topics/21plan/010301.htm Ministry of Education, Culture, Sports, Science and Technology. (2003). Regarding the establishment of an action plan to cultivate ‘Japanese with English abilities’. Retrieved from http://www.mext.go.jp/english/topics/03072801.htm Ministry of Education, Culture, Sports, Science and Technology. (2008). Recent MEXT’s Initiatives for Higher Education. Retrieved from http://www.jpf.go.jp/cgp/exchange/event/pdf/overview_ota.pdf Ministry of Education, Culture, Sports, Science and Technology. (2009). Launching the Project for Establishing Core Universities for Internationalization (Global 30) Retrieved from http://www.mext.go.jp/english/news/1283454.htm

32. Thank you for your attention! Any questions? James W. Pagel David W. Reedy jwpagel@yahoo.com davidr@cc.aoyama.ac.jp 14th International CALL Conference 08/20/2010