Conclusion Our prediction model did a good job at predict 8 th grade math proficiency. It can be used to estimate 10 th grade score fairly well, too. But we are disappointed  that it can not do better than MCAS even we got more data. Students learn from ASSISTments while being assessed. Follow-up questions: How much do students learn? How much do students learn? Will this raise state test score? Will this raise state test score? The ASSISTment Project Blends ASSISTance and assessMENT By Mingyu Feng In collaboration with Neil Heffernan, Joseph Beck & Kenneth Koedinger  The ASSISTment System is a web-based system that attempts to blend both computer-based tutoring and assessing. It offers instruction to students while providing a more detailed evaluation of their ability to teachers.  More than 3000 Worcester middle school students used ASSISTments as part of their math class. Background on ASSISTments Goal II: Teach students effectively How does ASSISTment work Student proficiency score correlates significantly with MCAS8 (r =.731) The interaction data matters a lot. A better prediction is obtained using assistance metrics (r =.864, reliably higher than.731) Our model did as well as MCAS on predicting exam scores two years later. CollaboratorsSponsors Q: How do we predict student end-of-year exam score? Collect data while students working in ASSISTment Student proficiency score – how student performed on main questions Assistance metrics – How student interacted with the system # attempts Help seeking behavior Speed performance on scaffolding questions On-taskness Build backwards linear regression model to predict real MCAS score  Students will first be administered a problem (main question). If they got it right, they will get a new one; otherwise, they are provided with a small “tutoring” session where they are forced to answer a few questions (called scaffolding questions) that break the problem down into steps.  On-demand hint messages and context sensitive buggy messages Q: Are we doing a good job at predicting? The Rasch model: MCAS8MCAS10 r = MCAS8’ r = r = Predicted score Assistance metrics Student proficiency predicts A sample learning opportunity group (Skill: Area) Goal I: Assess students accurately Q: Do students learn in ASSISTment? Item response theory (IRT) approach IRT model assumes no learning during tests Students who learn will be underestimated Modeling process Train Rasch model and compute residuals Do student analysis and item analysis to compare residuals at different opportunities Results Students learned mostly from 1 st opportunity Learning slows down afterwards Feng, Mingyu, Beck, J,. Heffernan, N. & Koedinger, K. (In preparation) Can an intelligent tutoring system predict math proficiency as well as a standardized test? To be submitted to the 1st International Annual Conference on Education Data Mining. Montréal Feng, Mingyu, Heffernan, N. (In preparation). Do students learn within ASSISTments? To be submitted to the 1st International Annual Conference on Education Data Mining. Montréal Feng, M., Heffernan, N.T. (2007). Towards live informing and automatic analyzing of student learning: Reporting in the Assistment system. Journal of Interactive Learning Research (JILR) 18(2) pp Chesapeake, VA: AACE. Feng, M., Heffernan, N. & Koedinger, K. (2006a). Predicting state test scores better with intelligent tutoring systems: developing metrics to measure assistance required. In Ikeda, Ashley & Chan (Eds.). Proceedings of the Eighth International Conference on Intelligent Tutoring Systems. Springer-Verlag: Berlin. pp Feng, M., Heffernan, N. T., & Koedinger, K.R. (2006b). Addressing the testing challenge with a Web-based e-assessment system that tutors as it assesses. Proceedings of the Fifteenth International World Wide Web Conference (WWW-06). New York, NY: ACM Press. Pardos, Z., Feng, M. & Heffernan, N. T. & Heffernan-Lindquist, C. (2007) Analyzing fine-grained skill models using bayesian and mixed effect methods. In Luckin & Koedinger (Eds.) Proceedings of the 13th Conference on Artificial Intelligence in Education. IOS Press. Razzaq, Feng, Heffernan, Koedinger, Nuzzo-Jones, Junker, Macasek, Rasmussen, Turner & Walonoski (2007). A Web-based authoring tool for intelligent tutors: Assessment and instructional assistance. In Nadia Nedjah, et al. (Eds.) in Intelligent Educational Machines. Intelligent Systems Engineering Book Series. Springer.