1. 1 Andes Tutoring: Freedom, Support, and Accelerated Learning as Students Solve Complex Physics Problems Kurt VanLehn & Brett van de Sande School of Computing, Informatics and Decision Engineering Arizona State University

2. 2 Andes development team u Developers –Brett van de Sande u Instructors/designers –Bob Shelby –Don Treacy u Experimenters –Scotty Craig –Bob Hausmann –Sandy Katz –Tim Nokes –Michael Ringenberg u Instructors 4-year colleges Jim Culbertson – Arizona State University, AZ John Fontinella – US Naval Academy, MD David Guerra – St. Anselm College, NH Troy Hacker – US Air Force Academy, CO Andrew Heckler – Ohio State University, OH Gerd Kortemeyer – Michigan State University Ted McClanahan – US Naval Academy, MD Mary Wintersgill – US Naval Academy, MD 2-year colleges Tom Wilbur – Ann Arundel Com. College, MD High Schools Sophia Gersham – Watchung Hills Reg. HS, NJ Paul Perkins – Belleview Christian School, WA David Richardson – Packer Academy, NY

3. 3 Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next

4. 4 Why physics? u Required for the other sciences u Major source of attrition u Well-studied in cognitive science u Other courses are similar

5. 5 A typical physics course Repeat: u Read chapter u Attend lectures u Solve problems u Do lab u Review for exam u Take chapter exam Take finals: Standardized & conceptual Tutoring needed

6. 6 Why focus on problem solving? u Students spend most of their time on problem solving u Frustrating; May causes attrition u Bad habits can develop –“Symbol pushing” instead of deep understanding u Done right, it can elicit deep, conceptual understanding u Professional human tutors focus on it

7. 7 What do professional tutors do? u Pick a good problem –Neither trivially easy nor impossibly difficult –Problem targets a weakness of the student u Help the student solve it –Let student try each step –Give immediate feedback if step is wrong –Give hints sparingly u Reflective debriefing –What were main principles? –What did you learn? Andes could, but doesn’t Andes does Under development

8. 8 How to evaluate Andes? Repeat: u Read chapter u Attend lectures u Solve problems u Do lab u Review for exam u Take chapter exam 1. Paper & human graders 2. Paper & grading service 3. Andes 4. Professional human tutor Take finals: Standardized & Conceptual Comparison conditions Just these 2 as of today

9. 9 Main goals: Andes should be a “workbook” where: u Instructors select & assign Andes problems –Need lots of problems, covering all textbooks u Students solve problems on Andes, not paper –getting immediate feedback and hints u Students learn more, compared to paper

10. 10 More goals u Instructors agree with all of Andes’ advice u Instructors no longer need to grade homework u Students prefer Andes to paper u Andes is as effective as human tutors

11. 11 Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next

12. 12 A typical physics problem u University introductory physics courses u High school physics courses A 2000 kg car at the top of a 20º inclined driveway 20 m long slips its parking brake and rolls down. If we ignore friction and drag, what is the magnitude of the velocity of the car when it hits the garage door?

13. 13 Andes user interface Read a physics problem Type in equations Draw vectors Type in answer

14. 14 Andes feedback and hints “What should I do next?” Green means correct Red means incorrect “What’s wrong with that?” Dialogue & hints

15. 15 Frequently asked questions (slide 1 of 3) u Why is Andes sometimes called an intelligent tutoring system? –Tutors usually replace classes; Andes doesn’t. –Andes should be called a “step-based homework helper” u Is Andes the same as web-based grading services such as WebAssign, Mastering Physics & LON-CAPA? –They give feedback & hints on answers only –Andes gives feedback & hints on steps leading to answers

16. 16 Most existing tutoring systems and web-based grading services accept only the answer W = 25 Answer What is the value of angle w? 45° w°w° x°x° 30° 40° z°z° y°y° u°u°

17. 17 With step-based systems (e.g., Andes), students enter steps leading up to the answer Step Answer Step 40+30+x=180 x=110 x=z w+45+z=180 w+45+110=180 w=180-155 w=25 Step What is the value of angle w? 45° w°w° x°x° 30° 40° z°z° y°y° u°u°

18. 18 Step-based homework helpers usually give immediate feedback on each step w = 40 OPPS! OK 45° w°w° x°x° 30° 40° z°z° y°y° u°u° What is the value of angle w?

19. 19 Hints start general… w = 40 Lines that look parallel often are not. OK 45° w°w° x°x° 30° 40° z°z° y°y° u°u° What is the value of angle w?

20. 20 Hints become more specific. w = 40 Try summing the angles of the triangle that include angle w. (see pg. 212) OK 45° w°w° x°x° 30° 40° z°z° y°y° u°u° What is the value of angle w?

21. 21 Usually, the last hint tells the student exactly what to enter. w = 40 You should apply the triangle sum rule by entering 45+z+w=180. OK 45° w°w° x°x° 30° 40° z°z° y°y° u°u° What is the value of angle w?

22. 22 Frequently asked questions (slide 2 of 3) u Does Andes require students to do all the steps? –No. But skipping conceptual steps lowers the score. u Does it require doing steps in a particular order? –No, except that variables must be defined before being used in equations –The only way to define a vector component is to draw the vector u Are there correct solutions that Andes doesn’t accept. –No (we hope).

23. 23 Frequently asked questions (slide 3 of 3) u Why does Andes focus on problem solving? Problem solving doesn’t improve conceptual understanding [sic]. Instruction really should focus on __________ instead of problem-solving. –Andes requires more conceptual steps than paper. –Reflective debriefing is being added –Perhaps instructors should let Andes to handle problem-solving so they can focus on __________.

24. 24 Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next

25. 25 Two methods for accessing Andes problems u Stand-alone –Downloads as a standard Windows application u Open-Learning Initiative (Web-based LMS)

26. 26 Standalone opening screen is a menu of physics topics (apprx. chapters)

27. 27 After clicking on “Translational dynamics,” get menu of problems

28. 28 Two methods for accessing Andes problems u Stand-alone –Downloads as a standard Windows application u Open-Learning Initiative (Web-based LMS) –Free & open usage »OLI keeps no record of student’s usage –Authenticated member of a registered class »OLI keeps a grade book for the instructor Next

29. 29 OLI top level screen Andes

30. 30 OLI top level of Andes is a menu of topics (apprx. Chapters)

31. 31 After clicking on “Translational Dynamics” module, get menu of problems

32. 32 OLI grade book u Looks/acts like a spreadsheet –One row per student –One column per problem –Cell has student’s score on the problem u Can export to Excel or database u Clicking a cell displays student’s solution in instructor’s Andes. –Can check student’s work –Good for office hours

33. 33 Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next

34. The major components of an ITS P(mastery) for each knowledge compo. Problem to be solved All correct steps in all orders Response pattern for each student step Expert Assessor Helper User interface

35. The expert’s computation u Expert can be authors, students or an expert system u Solve the problem in all acceptable ways u Record steps taken u Record knowledge components used at each step P(mastery) for each knowledge compo. Problem to be solved All correct steps in all orders Response patterns for each student step Expert Assessor Helper

36. Andes uses an expert system u Knowledge base organized by “problem solving methods” –One per major physics principle –Often called as “principle schemas” –Approximately 100 principles covered in 1-year course u Can solve more that 500 physics problems u For each, finds all acceptable solutions –Merges them into a solution graph u Pre-generates solution graphs & saves on disk –Allows regression testing (compare new to old) –Allows instructors to inspect 36

37. Problem solving method for Newton’s second law To apply Newton’s second law to at along : u Draw a free-body diagram for at including –For each force on at, draw it. –Draw the acceleration of at u Write the F=m*a in terms of components along u For each vector (i.e., acceleration and each force), write a projection equation for the vector along u For each minor principle of the form = where the force is one of the ones on at, write the minor principle’s equation. 37

38. 38 Problem & its solution graph A 2000 kg car at the top of a 20º inclined driveway 20 m long slips its parking brake and rolls down. If we ignore friction and drag, what is the magnitude of the velocity of the car when it hits the garage door? Draw coord. axes @ 20 deg Define given quantities d=20 m; m= 200kg Apply conservation of energy vf^2 = 2*g*sin(20 deg) Apply translational kinematics vi^2 = 2*a*d Apply Newton’s second law a= -g*cos(200 deg) Solve equations for final velocity vf= 11.59 m/s Enter answer: 11.59 m/s Draw body for car 1. Draw free-body diagram 2. Apply F=m*a along x-axis 3. Project vectors onto x-axis 4. Apply weight law W=m*g Draw weight force Draw normal force Draw acceleration

39. 4 popular designs for the Expert u Hand-author all possible solutions per problem –AutoTutor, CTAT u Rule-based AI problem solver + problem  all possible solutions –Andes, Cognitive tutors u Hand-author one solution & use constraints to generalize to all possible solutions –Constraint-based tutors e.g., SQL Tutor u Given log files from students, induce shortest/best paths –iLisp, Barnes tutor Problem to be solved All correct steps in all orders Expert

40. The helper’s computation u When the student enters a step, match it to a correct step u Give feedback & hints as necessary u Record response pattern P(mastery) for each knowledge compo Problem to be solved All correct steps in all orders Response patterns for each student step Expert Assessor Helper

41. How to tell if student’s equation is correct? u Color by numbers algorithm u Given student’s equation “Fw_x+m*g*sin(20 deg)” u Substitute values from solution point –Fw_x  6704 N –g  9.8 m/s^2 –m  2000 kg u Check arithmetic. –If checks, equation is correct u See paper for dealing with variables that don’t have values in the solution point 41

42. When the student needs a hint, how to choose next step? u Which solution branch is student probably following? u First PSM along path that is not finished? u First step inside the PSM that is not yet done? u But how to tell which equation-writing steps have been done already? 42

43. Matching equations via the indy check algorithm u Student enters “Fw_x+m*g*sin(20 deg)” –Row S below u Which equations in solution were combined? –Rows A, B, C, D –Because S’s gradient is a linear combination of their gradients 43 Function f  f/  m  f/  g  f/  Fw  f/  Fw_x gradient AFw_x –Fw*cos(250º)00–cos250º1(0, 0, 0.342, 1) BFw – mc*g–g–mc10(–9.8,–2000, 1,0) Cmc – 20001000(1, 0, 0, 0) Dg – 9.80100(0, 1, 0, 0) SFw_x+mc*g*sin(20º)g*sin20ºmc*sin20º01(3.352, 684, 0, 1)

44. 3 main design issues for the Helper u Matching student’s step to correct steps –Natural language: Use LSA, keywords, Atlas… –Math expressions: Substitute numbers for variables –Physical actions: Fuzzy, Bayesian u Handling pedagogically important student errors u Managing the student-tutor dialogue –Immediate feedback + hint sequences –Delayed feedback + student or tutor controlled debriefing –Adaptive, especially decision theoretic & fading All correct steps in all orders Response patterns for each student step Helper

45. The assessor’s computation u Given –Response patterns for each step taken by the student –Old P(mastery) for each knowledge component u Calculate –New P(mastery) P(mastery) for each knowledge compo. Problem to be solved All correct steps in all orders Response patterns for each student step Expert Assessor Helper Currently, Andes doesn’t do such assessment

46. Where’s the scale-up bottleneck? Number of problems Code size Expert Helper Assessor User interface

47. Other scaling up issues = same as other reforms u Coordination with curriculum & standards u Teacher buy-in and training u Support u Etc…

48. Implementation: Summary u Expert –Expert system, not humans u Helper –Matching student equations is challenging u Assessor –Dynamic Bayesian networks P(mastery) for each knowledge component Problem to be solved All correct steps in all orders Response patterns for each student step Expert Assessor Helper

49. 49 Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next

50. 50 Prior work with answer-only homework helpers u Compared to ordinary paper-based homework –Modest effect size: 0.42 (Kulik et al., 1983) u Compared to paper-based homework that is collected & graded –No benefits (Pascarella, 2002; Dufresne, Mestre & Rath, 2002) u Interpretation –Motivating students to do their homework provides benefits –But answer-only feedback & hints provides no additional benefits

51. 51 Prior work with step-based homework helpers u Lisp Tutor (Corbett, 2001) and many others –Same homework problems & text –Experimenter’s exams only –Not a whole semester (only 5 lessons) u Cognitive Tutors (Koedinger et al; Carnegie Learning) –Whole year of high-school algebra, geometry –Both experimenter’s exams & standard exams –Curriculum confounded with tutoring system

52. 52 Evaluation of Andes at the US Naval Academy u Fall semesters 2000, 2001, 2002 & 2003 u Only the homework modality was varied: Andes vs. paper-based –Same textbook –Similar lectures, labs, recitations –Similar homework problems –Same exams u Students were motivated to do paper-based homework –Either collected and graded –Or 1 homework problem on each quiz

53. 53 Exams u Midterm exam –1 hour, 4 problems –Scored on derivation & answer »Drawings (30%) »Variable definitions (20%) »Equations (40%) »Answers (10%) u Final exam –3 hours, 50 problems –Multiple choice

54. 54 Subjects; non-random assignment u Prior competence equal, all 4 years –Grade-point averages equal –Distribution of majors equal 2000200120022003 N Andes14012993 N control135445344

55. 55 Midterm exam results (All differences reliable, p <.01) Effect size: 0.61

56. 56 Final exam: Methodological details u Andes coverage of the course increased to 70% by 2003, so used only that year’s final exam u Non-random sample –89 Andes students (3 sections) –823 non-Andes students (rest of course) –GPAs, Majors reliably different u Regressed out incoming GPA, Major

57. 57 Final exam results Difference is reliable (p = 0.028) Effect size = 0.25

58. 58 Effect sizes for subscores of midterm exam The more conceptual the subscore, the greater the benefit

59. 59 Benefits same regardless of GPA

60. 60 Benefits varied by major on final exam but not on midterm exam

61. 61 Interpretation of results u Engineering & science majors knew red path & preferred it for answer-only –Andes didn’t affect their final exam scores u Other majors did not have red path, so they used the blue path on answer-only –Andes increased their final exam scores u Everyone used blue path on midterms –Andes increased everyone’s midterm exam scores –Biggest benefit for diagrams & variables –Smaller on equations; none on answer Problem Diagram & variables Equations Answer Andes Prior math & physics

62. 62 Summary of results u Main result: Replacing graded paper homework with Andes provides benefits –Midterm exam effect size: 0.61 –Final exam effect size: 0.25 u Andes helps students learn conceptual skills –Effect sizes on conceptual subscores: 1.21 & 0.69 –Effect sizes on calculational subscores: 0.11 & -0.08 u Engineering & Science majors appear to have a non-conceptual method for solving problems –Competes with the conceptual method taught by Andes –They use it on the (answer-only) final exam –This dilutes the benefit of Andes on final exam

63. 63 Andes’ effect sizes are typical of other step-based homework helpers

64. 64 Take-home message on homework helpers u Baseline: –Paper-based with light or no human grading u Better: –Answer-based e.g., WebAssign u Best: –Step-based e.g., Andes –

65. 65 Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Andes – technology u Evaluation u Why does Andes succeed? Next

66. 66 Why these results? Hypotheses: u Baseline: –Paper-based with light or no human grading –Students often do not do their homework u Better: –Answer-based e.g., WebAssign –Students do homework, but with non-optimal methods u Best: –Step-based e.g., Andes –Students do homework with optimal method

67. 67 3 methods for doing homework 1.Get answers from friends 2.Copy & edit another problem’s solution 3.Generate each step oneself

68. 68 1. Getting answers from friends u No learning, so should be discouraged u Even if the numbers in a problem are randomly generated, students circulate spreadsheets that calculate answers u Andes requires that students “show their work” –Andes can analyze time per step, too. –So Andes can make this kind of cheating very difficult

69. 69 2. Copy and edit another problem’s solution u Students learn general solution schemas –E.g., for pulley problems, a = g(m1-m2)/(m1+m2) –Good for learning to solve algebra word problems –High math students tend to use it (cf. USNA) –But should learn physics principles, not problems u Andes mildly discourages –Must close current problem in order to open an old one –Example solutions are videos, not paper u Andes should implement “fading”

70. 70 3. Generate each step oneself u If stuck, get hints (from Andes), or refer to textbook or videos of examples –Should focus on learning the principle, not just getting unstuck –Hint sequences start vague, become specific u Causes learning of principles, not problems

71. 71 Hypothesized distributions of solution methods explains results

72. 72 Take-home message, again u Baseline: –Paper-based with light or no human grading –Students often do not do their homework u Better: –Answer-based e.g., WebAssign –Students do homework, but with non-optimal methods u Best: –Step-based e.g., Andes –Students do homework with optimal method

73. 73 Outline u Goals of the Andes project u Andes – the core u Andes – the surroundings u Evaluation u Why does Andes succeed? u Future work Next

74. 74 Future work u Andes3 –Client/server (no downloading necessary) –Client runs in any browser & any OS –User interface ≈ PowerPoint (for familiarity) –Compatible with WebAssign & other services u Help others to author Andes problems u Support conceptual problem solving

75. 75 Quantitative problem solving is necessary but not sufficient for deep understanding u Necessary –Avoids inert knowledge u Not sufficient –Question: »If a massive truck and a lightweight car have a head- on collision, on which vehicle is the impact force greater? –Students who got A’s say: The car –Correct answer: Equal (Newton’s third law) u Many concept inventories (exams) exist

76. 76 Does Andes improve scores on concept inventories? u Learning principles instead of problems should help u But not enough –USNA: FCI not significantly different

77. 77 Why so little transfer from quantitative to conceptual problem solving? Conceptual prob. solving knowledge Quantitative prob. solving knowledge Traditional assumption  lots of transfer Quantitative prob. solving knowledge Cognitive modeling  little transfer Conceptual prob. solving knowledge

78. 78 Solution: Add conceptual problem solving practice u Question: In the problem you just solved, what would be different if the car had no mass? Quantitative prob. solving knowledge Conceptual prob. solving knowledge W=m*g If no mass, then no weight

79. 79 Katz & Connelly’s experimental addition to Andes u After students have finished an Andes problem, use a natural language dialogue system to discuss the problem’s solution u System asks lots of conceptual questions

80. 80 Students first solve an Andes problem

81. 81 Andes then asks about principles, alternative situations… u Andes: Which major principle did you need to solve the problem? u Student: Newton's 2nd law u Andes: Yes. The main difference between this problem and most of the previous Newtons 2 nd law problems is the fact that the direction of acceleration is __________. (fill in the blank) u Student: centripetal u Andes: Correct. All else being the same, how would the velocity at the lowest point of the swing change if the length of the rope were doubled to 4.0 m? u Student: it would also double u Andes: Well, it would increase, but it would not double. Recall that the formula for centripetal acceleration involves the square of the velocity. …

82. 82 Results of Katz/Connelly expt.  Successes  Treatment students significantly outperformed control students on a mostly qualitative post-test  Instructors would like reflective dialogue to become a permanent feature of Andes  Limitations  Currently covers only 8 weeks of Andes  No significant effect on FCI at the end of the semester  Next steps  Revise content to target FCI  Cover whole semester

83. 83 Thanks for your attention! u At www.andestutor.org you can…www.andes –Download the stand-alone version of Andes –Try the OLI version of Andes –Download papers on Andes –View videos of Andes being used

84. Questions? 84 P(mastery) for each knowledge compo. Problem to be solved All correct steps in all orders Response pattern for each student step Expert Assessor Helper User interface