A Five-year project CALL Project Leads to the Analysis of Motivation with an Eye on Curriculum Reform By David W. Reedy and James W. Pagel Associate Professors School of Science and Engineering Aoyama Gakuin University Tokyo, Japan 14th International CALL Conference 08/20/2010

This presentation deals with: Our previous study on implementing an e- learning component in all English classes offered in our department; The current investigation of whether entrance routes have any affect on performance on post tests two years after university entrance, and whether there is any correlation between completion of the e-learning project and an increase in test scores.

A basic explanation may be necessary. This research is the culmination of the authors’ efforts to implement and e-learning component in all English classes in their department to comply with MEXT mandates. It is our goal to improve the department’s English curriculum.

The Ministry of Education, Culture, Sports, Science and Technology (MEXT) has set various targets. In 2001 the Rainbow plan was announced: Smaller classes, easier-to-understand lessons and Integration of IT into the classrooms In 2004 the agency stated its goal of having all eligible High School graduates be accepted to university In 2005 in its initiatives for higher education, called for a fierce competition among private universities, and for globalization, which meant: 1. Accepting more students from overseas; 2. Sending more students abroad; 3. Opening satellite campuses; 4. Offering joint degree projects with other universities.

Globalization is the key word, but… Many universities* reported that they lacked the money the staffing the infrastructure to implement e-learning on a large scale basis * 60 percent answering the survey by the National Institute of Multimedia Education Source: Japan Times “E-Learning: Making Inroads into Nation’s Colleges” May 7, 2007

Japanese students are not so interested in going overseas. Students are content with their situation “at home.” Internet research is sufficient. (And companies can no longer afford to send its workers overseas as in the past.) Students fear they lack the ability to succeed in foreign countries. Reverse Japan’s Insularity. Japan Times April 8, 2010 Forty-nine percent of new hires will not go abroad. Jiji Press August 13, 2010

This is the start of the study. Integrate commercial software in all departmental English classes. But only 36.3% of Japanese universities had offered e- learning classes as of Purchase of commercial software was “one-time only.” Analyze the merits of such software.

In short, we succeeded.

We needed to know how students reacted to such software.

Our survey results were inconclusive. 938

Why did they stop? 9a. Why did you give up? Hard to maintain motivation 90 (40%) 91 (54%) Takes too much time 72 (32%) 33 (20%) “Typically, drop-out rates are reported to be 80 percent or more in e-learning environments where little or no classroom support is available. “ Evolution of CALL” by Lance Knowles

Here is some more information. 1. Why didn’t you try the material? Too busy 454 (61%) 228 (65%) Bonus points 110 (15%) 25 (7%) too low Dislike English 68 (9%) 34 (9%)

Our target became how to identify students who completed the e-learning component and their motivation. Break down the students by department. (2010) Seek other ways of finding correlations. Reinvent the student survey.

We broke down students by department. N=564

So you are now up to speed. What is next? We could analyze completion of the e- learning component other ways. And why …(problem: survey answers were anonymous.) We could study how students are admitted to the university to better understand motivational factors in the study of English.

There are ten ways to get accepted. Departmental Exam Day One Passing on first cut-off 224 (39.7%) Departmental Exam Day One Passing on second cut-off 45 (8%) Departmental Exam Day Two Passing on first cut-off 100(17.7%) Departmental Exam Day Two Passing on second cut-off 4 (0.7%) National Exam First Term 20 (3.6%) National Exam Second Term + Departmental Exam 21 (3.7%) Internal High School Evaluation 30 (5.3%) Recommendation 120 (21.3%) International Students Exam Foreign Students 2 (0.4%) International Student Exam Students attending international high school 1 (0.2%)

We considered success on the basis the following. TOEFL ITP Pretest results averages Posttest results averages Best score change averages Percentage of students whose scores increased Percentage of students whose scores decreased e-learning Percentage of freshmen who completed the requirement Percentage of freshmen who completed the bonus component Percentage of sophomores who completed the requirement Percentage of sophomores who completed the bonus component

TOEFL ITP pretest scores University EntranceMeanSD 1. Departmental Exam Day One: Passing on first cut-off Departmental Exam Day One: Passing on second cut-off Departmental Exam Day Two: Passing on first cut-off Departmental Exam Day Two: Passing on second cut-off National Exam National Exam + Departmental Exam Internal High School Evaluation Recommendation International Students Exam: Foreign Students International Student Exam: Japanese students attending foreign high school 5400

Pretest and Posttest scores University Entrance Pretest Mean SDPosttest Mean SD 1. Departmental Exam Day One Passing on first cut-off Departmental Exam Day One Passing on second cut-off Departmental Exam Day Two Passing on first cut-off Departmental Exam Day Two Passing on second cut-off National Exam National Exam + Departmental Exam Internal High School Evaluation Recommendation International Students Exam Foreign Students XX 10. International Student Exam Japanese students attending foreign high school Total

The Spearman rank-order correlation coefficient calculates to.82. University EntrancePretest RankingPost test Ranking 10.International Student Exam Japanese students attending foreign high school 1 (540)1 (590) 5. National Exam 2 (442.6)2 (428.1) 1. Departmental Exam Day One Passing on first cut-off 3 (422.4)3 (410.8) 6. National Exam + Departmental Exam 4 (417.2)4 (405.4) 3. Departmental Exam Day Two Passing on first cut-off 5 (416)9 (388.6) 2. Departmental Exam Day One Passing on second cut-off 6 (411.4)6 (392.7) 7. Internal High School Evaluation 7 (405.3)5 (401.4) 8. Recommendation 8 (398.7)7 (390) 4. Departmental Exam Day Two Passing on second cut-off 9 (394)8 (389) 9. International Students Exam Foreign Students 10 (383.5)XX

How did scores change? Entrance RouteAverage TOEFL Change Scored Higher Scored Lower No Change Students Absent 1. Departmental Exam Day One Passing on first cut-off -10 (-2.4%) 64 (39%) 99 (60.4%) 1 (0.6%) 59 (26.8%) 2. Departmental Exam Day One Passing on second cut-off (-4.6%) 12 (36.4%) 21 (58.3%) 0 12 (26.7%) 3. Departmental Exam Day Two Passing on first cut-off (-6.6%) 14 (23.3%) 46 (76.7%) 2 (3.3%) 38 (38%) 4. Departmental Exam Day Two Passing on second cut-off -5 (-1.3%) 1 (25%) 2 (50%) 1 (25%) 0 (0%) 5. National Exam (-3.3%) 3 (18.6%) 12 (75%) 1 (6.3%) 4 (20%) 6. National Exam + Departmental Exam (-2.8%) 6 (35.3%) 11 (64.7%) 0 4 (19.1%) 7. Internal High School Evaluation -3.9 (-1%) 6 (28.6%) 12 (57.1%) 3 (14.3%) 9 (30%) 8. Recommendation (-3.1%) 39 (37.9%) 58 (56.3%) 6 (5.8%) 17 (14.2%) 9. International Students Exam Foreign Students XX 2 (100%) 10. International Student Exam Japanese students attending foreign high school +50 (+9.3%) 1 (100%) 000 Total-11.4 (-2.76%) 146 (34.68%) 261 (63%) 14 (3.33%) 145 (25.62%)

TOEFL achievements at a glance. Entrance Routepretestpost testupdownchange 1. Departmental Exam Day One Passing on first cut-off Departmental Exam Day One Passing on second cut-off Departmental Exam Day Two Passing on first cut-off Departmental Exam Day Two Passing on second cut-off National Exam National Exam + Departmental Exam Internal High School Evaluation Recommendation International Students Exam Foreign Students 10XX 10. International Student Exam Japanese students attending foreign high school 11111

Give your eyes a little break!

Are you now ready for more? I am David & on vacation

Our e-learning program at a glance. 1 st term2 nd term FreshmenRequirement 6 units Bonus Component 30 units SophomoresRequirement 6 units Bonus Component 30 units

They always do better as freshmen. Entrance Route1 st term req 1 st term bonus 2 nd term req 2 nd term bonus 1. Departmental Exam Day One Passing on first cut-off Departmental Exam Day One Passing on second cut-off Departmental Exam Day Two Passing on first cut-off Departmental Exam Day Two Passing on second cut-off National Exam National Exam + Departmental Exam Internal High School Evaluation Recommendation International Students Exam Foreign Students International Student Exam Japanese students attending foreign high school 1000 Total

Let’s focus on E-learning achievements. Let’s focus on E-learning achievements. Entrance Route1 st yr req 1 st yr bonus 2 nd yr req 2 nd yr bonus 1. Departmental Exam Day One Passing on first cut-off Departmental Exam Day One Passing on second cut-off Departmental Exam Day Two Passing on first cut-off Departmental Exam Day Two Passing on second cut-off National Exam National Exam + Departmental Exam Internal High School Evaluation Recommendation International Students Exam Foreign Students International Student Exam Japanese students attending foreign high school 1911

Let’s tally up the two groups. Entrance RouteTOEFL total rank e-learning total rank 1. Departmental Exam Day One Passing on first cut-off Departmental Exam Day One Passing on second cut-off Departmental Exam Day Two Passing on first cut-off Departmental Exam Day Two Passing on second cut-off National Exam National Exam + Departmental Exam Internal High School Evaluation Recommendation International Students Exam Foreign Students 10. International Student Exam Japanese students attending foreign high school 11

We drew the following conclusions. Success can’t be determined by comparing entrance route and TOEFL scores. The Spearman rank-order correlation coefficient of TOEFL and e-learning achievements calculates to.44. Students entering via the internal high school evaluation system were most consistent (ranked 3 rd in both categories). More data analysis is necessary.

Our current investigations have led us to consider: Limiting comparison to top and bottom scoring students; Looking at individual departments and their students’ pre- test and posttest scores vs e- learning completion rates, considering the types of classes required); Revising the student survey tool to include more questions that would address motivation.

References not listed above Ministry of Education, Culture, Sports, Science and Technology. (2001). The education reform plan for the 21st century – the Rainbow Plan. Retrieved from Ministry of Education, Culture, Sports, Science and Technology. (2003). Regarding the establishment of an action plan to cultivate ‘Japanese with English abilities’. Retrieved from Ministry of Education, Culture, Sports, Science and Technology. (2008). Recent MEXT’s Initiatives for Higher Education. Retrieved from Ministry of Education, Culture, Sports, Science and Technology. (2009). Launching the Project for Establishing Core Universities for Internationalization (Global 30) Retrieved from

Thank you for your attention! Any questions? James W. Pagel David W. Reedy 14th International CALL Conference 08/20/2010