1. Learn about research involving this online formative assessment system and hear about how it is used in classrooms. Worcester Polytechnic Institute Introduces Neil Heffernan, a computer science professor about his research using this tool. In this Webinar you will hear from: Matt Militello, a professor of school leadership on how school leaders can use such a system. Courtney Mulcahy, an 8 th grade teacher who uses ASSISTment in her formative assessment process every day. Cristina Heffernan, a professional developer who trains teachers to use the system and incorporate it into their classrooms. And Russ Rapose, District curriculum coordinator for Milbury Public Schools

2. If Old McDonald Had An Assessment… Matthew Militello Assistant Professor

3. The Press… The Push The Press: More than ever we are TOLD, MANDATED, and SANCTIONED to use data to inform education decisions. Specifically, assessment data. Means state tests. The Push: These data are not timely, not informative for teachers; That is, these data points are autopsy data. TMI: Too Much Information.

4. In Search of the Right Data! Central Office Student/School Growth School Leaders Teacher Effectiveness Teachers Student Cognition

5. Formative Assessment Systems (FAS) There is Demand…. Supply Followed Multi-Million Dollar Industry- Cost usually based on per student use Not all FAS are the SAME!

6. How Do Districts Select, Support, and Use FAS? Three DistrictsThree FAS Technical FeaturesImplementation Actual Use

7. Framework

8. Fit is defined by the INTENT of the districts selection (e.g. teacher use (student-diagnostic data) or district use (student growth model data) and the PURPOSE of the assessment system—including validity.

9. Framework Readiness is defined as: (1) building capacity in regard to both knowledge of the assessment product and (2) the motivation to use the product in practice.

10. Framework Coherence is how educators used the assessment systems fit into the context and the standard operating procedures of the district. For instance, the creation of trigger mechanisms from review of data reports.

11. Woodbury Schools Good fit for Central Office, but not teachers Teachers ill-equipped Growth data forced on daily practice FAS- NWEA-MAP

12. Can’t Get Beyond FIT!

15. Needle in a Haystack: A FAS with Real-time, Student-Level Data! Fit! Finally, Formative Assessments for TEACHERS Technical Features Tutoring System for Real-Time Data and Diagnostic Data: PURPOSE Teacher-Level Use- Desire For This Data: INTENT

16. Using Intelligent Tutoring Systems to Help Math Teachers in the Formative Assessment Process. Prof. Neil Heffernan Worcester Polytechnic Institute

17. Who Am I?  Teacher for America : teacher of middle school math and science in Baltimore City  PhD in Computer Science and Cognitive Science –Licensed to Carnegie Learning Inc.  Tenure professor at Worcester Polytechnic  Grant funded research on developing interventions

18. Quick Project History - 2003  Teacher are being asked to use data to inform instruction, but every minute in the classroom spent on assessment is a minute lost to instruction.  US Department of Education (2003) Funded us to build a assessment system for teachers that did not take away teacher’s time.

19. Lindeman, Carolina Ruiz, George Heineman, Murali Mani, Ryung Kim and others 30 others

20. Definitions  A diagnostic assessment is an assessment designed to uncover any misconceptions or gaps in knowledge or skills of individual students so that teachers can base their instruction on the current state of students’ knowledge, skills, or understanding.  Formative assessment is a process used by teachers and students during instruction that provides feedback to adjust ongoing teaching and learning to improve students’ achievement of intended instructional outcomes.

21. Research Questions for US Dept of Ed 1.Does ASSISTments work as an assessment tool? a.Reliably predict state test performance? b.Reliably advise teachers & students on what knowledge to focus on? 2.Does ASSISTments help enhance student learning? a.Does the System effectively teach as it assesses? b.Can teachers use the advice of this System to lead to higher student learning?

22. The ASSISTment System  A web-based assessment system, designed to collect formative assessment data on student math skills  Usage has grown as our content has grown.  Currently, thousands of students use the system.

28. You Just Saw One ASSISTment  A question within this “tutor” is what we call an ASSISTment.  We have 2,000 such ASSISTments for 7 th and 8 th grade. (66 hours of instruction)  We have another 1,000 for 6 th grade.  We have 1000 for 9-10 th grade math,  There are Three other types of content. –Teacher created questions –Mastery Learning questions (track 130 skills in 8 th grade math) –Connected Math matching units (500 ASSISTments with tutoring)

29. Many Different Types of Reports  State Test Report  Knowledge component reports (based upon 130 middle school skills)  Item Report (e.g. for small formative assessments)  Mastery Learning Report (which students have reached mastery?)

30. Names Obscured on Purpose

31. Report to a Teacher on Their Own State Test

35. Research Questions for US Dept of Ed 1.Does ASSISTments work as an assessment tool? a.Reliably predict state test performance? b.Reliably advise teachers & students on what knowledge to focus on? 2.Does ASSISTments help enhance student learning? a.Does the System effectively teach as it assesses? b.Can teachers use the advice of this System to lead to higher student learning?

36. Report to a Teacher on Their State

37. Predict Student Knowledge as Measured by State Test Scores  We have state scores from 2004 to 2009 for thousands of students. –We showed our error in predicting the state test is as low as the test itself. –We showed we can even predict their 10 th grade math score from their 8 th grade data. –We have shown we can more reliably predict their state test score if we use how much assistance they need. Feng, M., Heffernan, N.T., & Koedinger, K.R. (2009). Addressing the assessment challenge in an Intelligent Tutoring System that tutors as it assesses. The Journal of User Modeling and User-Adapted Interaction. Vol 19: p243-266.Addressing the assessment challenge in an Intelligent Tutoring System that tutors as it assesses. Feng, M., Beck, J,. Heffernan, N. & Koedinger, K. (2008) Can an Intelligent Tutoring System Predict Math Proficiency as Well as a Standardized Test? In Baker & Beck (Eds.). Proceedings of the 1st International Conference on Education Data Mining. Montreal, Canada. pp.107-116. ISBN - 13: 9780615306292 Can an Intelligent Tutoring System Predict Math Proficiency as Well as a Standardized Test? Feng, M., Heffernan, N. T., & Koedinger, K. R. (2006b). Addressing the testing challenge with a Web-based e-assessment system that tutors as it assesses. Proceedings of the Fifteenth International World Wide Web Conference (WWW-06). New York, NY: ACM Press. ISBN:1-59593-332-9. pp. 307-316. [Nominated for Best Student Paper] Addressing the testing challenge with a Web-based e-assessment system that tutors as it assesses.

38. 1.Does ASSISTments work as an assessment tool? a.Reliably predict state test performance? b.Reliably advise teachers & students on what knowledge to focus on? 2.Does ASSISTments help enhance student learning? a.Does the System effectively teach as it assesses? b.Can teachers use the advice of this System to lead to higher student learning? Research Questions for US Dept of Ed

39. We have shown that our transfer model is good enough to improve assessment.  Using finer-grained skill model can improve accuracy of the assessment of student knowledge.  Feng, M., Heffernan, N. T., Mani, M., & Heffernan, C. (2007). Assessing students’ performance longitudinally: Item difficulty parameter vs. skill learning tracking. The National Council on Educational Measurement 2007 Annual Conference, Chicago.Assessing students’ performance longitudinally: Item difficulty parameter vs. skill learning tracking.  Feng, M, Heffernan, N., Heffernan, C. & Mani, M. (2009) Using Mixed-Effects Modeling to Analyze Different Grain-Sized Skill Models. Journal of Technology, Learning, and Assessment. (Featured Article of the year!)Using Mixed-Effects Modeling to Analyze Different Grain-Sized Skill Models.  Pardos, Z. A., Heffernan, N. T., Anderson, B. & Heffernan, C. (2007). The effect of model granularity on student performance prediction using Bayesian networks. The International User Modeling Conference 2007.The effect of model granularity on student performance prediction using Bayesian networks.

44. 1.Does ASSISTments work as an assessment tool? a.Reliably predict state test performance? b.Reliably advise teachers & students on what knowledge to focus on? 2.Does ASSISTments help enhance student learning? a.Does the System effectively teach as it assesses? b.Can teachers use the advice of this System to lead to higher student learning? Research Questions for US Dept of Ed

45. Can teachers use the advice of this system to lead to higher student learning?  Quasi Experiment –Results: Student in schools that used ASSISTments did better on state test than students in control schools. –Results: Gaps were closed in achievement  Recruitment for a large 60 school field trial is being organized. $25 million dollar grant “i3” Investing in Innovation Fund grant line If you are interested talk to me.

46. Stories of Formative Assessment Online Exit Card Mastery Learning Book Homework Online Courtney Mulcahy, an 8 th grade teacher who uses ASSISTment in her formative assessment process every day.

47. The Formative Assessment Cycle Objective Lesson Assessment Teacher Student

48. The Cycle can move through two or three iterations depending on the situation. ObjectiveLessonObjective LessonAssessment ObjectiveLesson Assessment

49. Online Exit Card Objective Find the side of a triangle using the Pythagorean Theorem Lesson Use Connected Math Book Assessment Three question Problem Set online on ASSISTment

50. Assessment Three question Problem Set online on ASSISTment

51. Assessment Three question Problem Set online on ASSISTment Teacher Studen t There were 43 students on this item report, not all are shown.

52. INCORRECT a 2 +b 2 = c 2 12 2 +14 2 = x 2 144 + 196 = x 2 340= x 2 18.44 = x CORRECT a 2 +b 2 = c 2 12 2 +x 2 = 14 2 144 + x 2 = 196 x2 = 54 x = 7.21

53. Online Exit Card Objective Find the side of a triangle using the Pythagorean Theorem Lesson Use Connected Math Book Assessment Three question Problem Set online on ASSISTment Objective Find the leg of a right triangle using the Pythagorean Theorem. Lesson Discussed the different elements of the Pythagorean Theorem.

54. Mastery Learning Objective Find Mean Median Mode Range from data in a Stem and Leaf Plot Lesson Lessons in 7 th Grade Assessment Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots

55. Assessment Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots Students work until they get 3 (or another chosen number) right in a row. The are very involved in their learning.

56. Mastery Learning Objective Find Mean Median Mode Range from data in a Stem and Leaf Plot Lesson Lessons in 7 th Grade Assessment Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots Objective Find Mean Median Mode Range from data in a Stem and Leaf Plot Lesson Tutoring in ASSIST- ment.

57. Lesson Tutoring in ASSIST- ment. Students gets instant feedback and support.

58. Mastery Learning Objective Find Mean Median Mode Range from data in a Stem and Leaf Plot Lesson Lessons in 7 th Grade Assessment Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots Objective Find Mean Median Mode Range from data in a Stem and Leaf Plot Lesson Tutoring in ASSIST- ment. Assessment Students must work until they get 3 right in a row. Can give a static post test.

59. Assessment Students must work until they get 3 right in a row. Can give a static post test.

60. Book Homework Online Objective Students will be able to solve questions involving exponential functions. Lesson Use CMP Book Assessment Homework from the Book, with questions online. One question asks about linear equations.

61. Homework was Page 13 # 22 and 23 Page 16 # 34

62. Assessment Homework from the Book, with questions online. One question asks about linear equations. From Connected Mathematics Project Page 16 of Growing Growing Growing.

64. Assessment Homework from the Book, with questions online. One question asks about linear equations. This data informed both teacher and students of the need for the next objective

65. Book Homework Online Objective Students will be able to solve questions involving exponential functions. Lesson Use CMP Book Assessment Homework from the Book, with questions online. One question asks about linear equations. Objective While studying exponential functions students will be able to answer questions about linear relationships. Read what is being asked carefully. Lesson Discuss the linear equation question in class. Assessment Next nights homework had multiple questions about comparing linear and exponential functions.

66. From Connected Mathematics Project Page 14 of Growing Growing Growing. Assessment Next nights homework had multiple questions about comparing linear and exponential functions.

67. Homework was Page 14 – 17 # 24 – 30 and 35 - 38

68. Stories of Formative Assessment Online Exit Card Mastery Learning Book Homework Online

69. Does It teach as It Assesses?  We have done 6 different randomized controlled experiment, and ASSISTments consistently leads to statistically meaningful learning results.  Two examples 1.A homework study 2.A classroom study

70. Comparing Traditional Paper and Pencil Homework with Web-Based Homework Leena Razzaq, Michael Mendicino & Neil Heffernan Worcester Polytechnic Institute & West Virginia University Intelligent Tutoring Systems 2008 Mendicino, M., Razzaq, L. & Heffernan, N. T. (2009). Improving Learning from Homework Using Intelligent Tutoring Systems. Journal of Research on Technology in Education (JRTE). Published by the International Society For Technology in Education (ISTE). Spring 2009 issue. 41:3 p 331-346.Improving Learning from Homework Using Intelligent Tutoring Systems. Journal of Research on Technology in Education (JRTE).

71. Digital Divide Is Real. Worcester is an urban district with only about 70% of homes with Internet. Students already get weekly homework that requires the Internet. Worcester needs supports for students who don’t have computers at home. –School libraries –Printable worksheet –Before school and homeroom These studies prepare us for a future where kids can get the benefit of computers.

72. Maine Lap Top Initiative  Every student in the state get a lap top. Over half the districts let students take the lap top home.  State scores have gone up in English but stayed flat in Math. (Prof. David Silvernail)  Richmond Virginia (Henrico Country)

73. Is Homework Important? Etta Kralovec, author of The End of Homework: How Homework Disrupts Families, Overburdens Children, and Limits Learning "When work goes home, teachers have little control over who does the work. Teachers [are] unable to scaffold new knowledge for students, and [are] unaware of each student's true educational progress.” http://www.hno.harvard.edu/gazette/2000/09.28/homework.html Hattie(1999) meta-analysis across over 100 studies showed that homework has a.4 effect size.

74. Literature Review: Other Comparisons of PPH to Tutoring Systems  Little prior work but the best of it is at the college level –Mastering Physics – (Warnakulasooriya & Pritchard, 2005) –Andes (VanLehn et al., 2005) produced a mean effect size of 0.6 over paper-and-pencil homework. 74

75. Possible Advantages of Web Based Homework Better learning for students –Immediate feedback to students with tutoring More help for teachers –Time saver –Common errors analyzed –Formative assessment—Teachers use the data to guide instruction more effectively. –Homework help—Students take homework more seriously because they know it will be graded and the grade recorded.

76. Possible Disadvantages Many of these systems do not take students’ work into consideration. Students may try to do more math in their heads. Teacher may spend less time looking at student work. Cheating may be easier among students.

77. Experimental Design Purpose: to determine if students can learn more by doing their math homework with the ASSISTment System than when doing traditional paper-and-pencil homework Control: Student did their work on paper. The next day teachers did a 10 minute review of the questions. Teachers used the same feedback the ASSISTments system gave. Materials: Pretest and Posttest of 10 items Homework was also 10 questions but with different numbers. Topic A: Number Scene Topic B: Mixed 77

78. 1. Which of the following is closest to the product of 397.8 * 10.3? o A) 3,000 o B) 30,000 o C) 400 o D) 4,000 2. Which of the following shows the numbers in order from least to greatest? o A) 0.452, 0.51, 0.432 o B) 0.452, 0.432, 0.51 o C) 0.432, 0.51, 0.452 o D) 0.432, 0.452, 0.51 4. What is 15/60 as a percent? 5. Write 10/25 as a decimal. 6. What is the value of the following expression? 7 * 6 + 3 7. Destinee has a total of 12 fish in her aquarium. Exactly 9 of the fish are goldfish. What percent of the fish in the aquarium are goldfish? Sample “Number Sense”

79. Experimental Design Continued  We did a repeated measures analysis where each student participated in each condition but in different orders. This allows us to deal with the large variation in different students learning rates. We eliminate the need to model the variation between teachers.  We counter-balanced the homework assignments in a Latin square design. The order of seeing the topics were randomly assigned to different classrooms.  West Virginia- 4 math teachers

80. Experimental Design Continued DayComputer first groupPaper-and-Pencil First Group Class AClass BClass CClass D Mon Pretest Number Sense Intro to ASSISTments ASSISTments Pretest Mixed Intro to ASSISTments ASSISTments Pretest Mix PPH Pretest Number Sense PPH Tues Post-test Number Sense Post-test MixedReview PPH Post-test Mix Review PPH Post-test Number Sense Wed Pretest Mixed PPH Pretest Number Sense PPH Pretest Number Sense Intro to ASSISTments ASSISTments Pretest Mix Intro to ASSISTments ASSISTments Thurs Review PPH Post-test Mixed Review PPH Post-test Number Sense Post-test Mix 80

81. Experimental Design Continued To deal with the fact that not everyone had internet at home we only analyzed students that did their homework both nights in both conditions. Left us with 28 students You can’t ask students if they internet at home 1.Its embarrassing 2.They don’t know if they will be able to have access

82. Results Students learned significantly in both conditions Students learned reliably more with ASSISTments (2.3 problems verse 1.14 problems) with 0.6 effect size (p = 0.05). 82

83. Results Not Due to a Few Students Top 5 gain scores were in the web based homework condition. Our worst gain score is in our WBH condition.

84. Conclusions  Students do learn significantly (.6 standard deviations more) more from using the ASSISTment System compared to traditional paper-and-pencil homework. 84

85. Implications Results of this study are promising in that kids can learn a good deal more with homework. Only study of its kind I am aware of in K12 that isolated the benefit of web based homework. Could be important to policy makers, particularly considering the popularity of one-to-one computing initiatives Will become be more relevant as the digital divide narrows.

86. Does an Intelligent Tutor lead to more learning than traditional classroom practice?  Most math classrooms give a lecture and then a period of time for practice.  We showed that kids learn a lot more when getting immediate feedback from the ITS than the “business as usual” control.  Results: Students learned more when using the computer compared to the traditional classroom. –Razzaq, L., Mendicino, M. & Heffernan, N. (2008) Comparing classroom problem-solving with no feedback to web-based homework assistance. In Woolf, Aimeur, Nkambou and Lajoie (Eds.) Proceeding of the 9th International Conference on Intelligent Tutoring Systems. pp. 426 -437. Springer-Verlag: Berlin.Comparing classroom problem-solving with no feedback to web-based homework assistance

87. 1.Does ASSISTments work as an assessment tool? a.Reliably predict state test performance? b.Reliably advise teachers & students on what knowledge to focus on? 2.Does ASSISTments help enhance student learning? a.Does the System effectively teach as it assesses? b.Can teachers use the advice of this System to lead to higher student learning? Research Questions for US Dept of Ed

88. Can teachers use the advice of this system to lead to higher student learning?  Quasi Experiment –Results: Student in schools that used ASSISTments did better on state test than students in control schools. –Results: Gaps were closed in achievement  Recruitment for a large 60 school field trial is being organized. $25 million dollar grant “i3” Investing in Innovation Fund grant line If you are interested talk to me.

89. We Have Users in Over a Dozen Towns  Worcester, Auburn, Fitchburg—Classroom formative assessment  Shrewsbury—Nightly homework & Mastery Learning  Grafton—Open response explanation writing  Milbury—Benchmark Assessment every 9 weeks, teacher reports  Maine: Mastery Learning for homework

90. What have I not told you about?  Parent Notification System  How teacher and schools put in their own content.  Stick around for the next 30 minutes to learn how to –Create our own account, –Let kids log in, and –Build your own content. Cristina Heffernan

91. Thank you

98.  Extra slides

99. Cognitive Tutor – Carnegie Learning Inc.

100. Replicated Field Studies  Controlled, full year classroom experiments  Replicated over 3 years in urban schools  In Pittsburgh & Milwaukee  Results: 50-100% better on problem solving & representation use. 15-25% better on standardized tests. Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8.

101. An Intelligent Tutoring System Success  Cognitive Tutor Algebra  Most widely used ITS –1000+ schools across the country –Marketed by local spin-off company Carnegie Learning  “Exemplary Curricula” by US Dept of Ed. In What Works Clearing House  Most cited Journal of AI-ED paper –Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.......