1. Introduction to the Cognitive Tutor Authoring Tools (CTAT) and Example-Tracing Tutors Bruce McLaren Systems Scientist, Co-Manager of the CTAT Project Team Human-Computer Interaction Institute Carnegie Mellon University

2. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 Kinds of Computer Tutors Intelligent tutoring systems e.g., Sherlock Model-tracing tutors e.g., Andes Cognitive tutors e.g., Algebra Tutoring systems CAI e.g., Microsoft’s Personal Tutor Constraint- based tutors e.g., SQL Tutor

3. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 Cognitive Tutors: Real-world success of Intelligent Tutoring Systems technology Cognitive Tutor Courses –Computer-based one-to-one tutoring Based on computational models of student thinking Based on the concept of Model Tracing Used by many students in many schools –Algebra Cognitive Tutor: In over 2,000 schools in the USA, 300,000 students per year –Geometry Cognitive Tutor: In 350 schools –Most widely used intelligent tutoring system University created a company, Carnegie Learning, to disseminate

4. Algebra Cognitive Tutor Sample Use graphs, graphics calculator Analyze real world problem scenarios Use table, spreadsheet Use equations, symbolic calculator Tutor learns about each student Tutor follows along, provides context-sensitive Instruction

5. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 ACT-R: A Cognitive Theory of Learning and Performance Big theory … key tenets: –Learning by doing, not by listening or watching –Production rules represent performance knowledge: These units are: Instruction implications: modular context specific isolate skills, concepts, strategies address "when" as well as "how" Anderson, J.R., & Lebiere, C. (1998). The Atomic Components of Thought. Erlbaum.

6. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 Cognitive Model: A system that can solve problems in the various ways students can Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c = d/a Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d Cognitive Tutor Technology: Use ACT-R theory to individualize instruction

7. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology: Use ACT-R theory to individualize instruction Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

8. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology: Use ACT-R theory to individualize instruction Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction Hint message: “Distribute a across the parentheses.” Bug message: “You need to multiply c by a also.” Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing Known? = 85% chanceKnown? = 45%

9. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 CTAT motivation: Make tutor development easier and faster! Development costs of instructional technology are, in general, quite high –E.g., ~300 dev hours per hour of instruction for Computer Aided Instruction (Murray, 1999) Cognitive Tutors: –Large student learning gains as a result of detailed cognitive modeling –~200 dev hours per hour of instruction (Koedinger et al, 1997) –Requires PhD level cog scientists and AI programmers Solution: Easy to use Cognitive Tutor Authoring Tools (CTAT) Murray, T. (1999). Authoring Intelligent Tutoring Systems: An Analysis of the state of the art. The International Journal of Artificial Intelligence in Education, 10, 98-129. Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. The International Journal of Artificial Intelligence in Education, 8, 30-43.

10. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 How to reduce the Authoring Cost? Less programming, more automation –Drag & drop interface construction –Demonstration-based programming Human-Computer Interaction methods –User studies, summer schools, informal & formal comparison studies Exploit tools already in use –Component-based architecture & standard inter- process communication protocols

11. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 Authoring with CTAT Cognitive Tutors –Difficult to build; for programmers –General for a class of problems Example-Tracing Tutors –Much easier to build; for non-programmers –Limited to a single problem instance New: “Mass Production” feature greatly facilitates authoring of multiple problem instances (in Excel) Typical Approach –Build Example-Tracing Tutor first Cognitive Task Analysis Rapid prototypes - perhaps sufficient for the problem –Develop Cognitive Tutor from Example-Tracing Tutors Examples guide planning Serve as semi-automated test cases

12. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 Where does CTAT fit into the Realm of Authoring Tools? Murray survey (1999) discusses 7 categories of authoring tools for intelligent tutoring systems (ITS), including: Device Simulation - RIDES (Munro et al 1997) Tutoring Strategies - REDEEM (Ainsworth et al 2003; Major et al 1997) Domain Expert System - Demonstr8 (Blessing 1997) Cognitive Tutor Authoring Tools: “Domain Expert System” category of authoring systems No commercially successful tools for authoring model- tracing, Cognitive Tutors First to provide functionality to simplify & speed cog. tutor development

13. Behavior Recorder Data Shop CTAT’s Modular Architecture Student Interface (or external problem- solving environment) Cool Modes CyclePad Java Swing Flash Learner Management System LMS Tutor Shop Cognitive Model Development Tools TDK (Lisp-based) Jess tools (Java- based) Editor Eclipse Tutor Engine Tertle (Lisp- based) + Model Tracer Jess (Java-based) + Model Tracer Example Tracer (Java-based) Example Tracer (Flash-based) GUI Builder IntelliJ Dreamweaver Code Warrior Netbeans Flash MX 2004

14. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 CTAT’s Track Record CTAT-based tutors have been used in experiments in Geometry, Chemistry, Chinese, and French Over 200 users Three releases in past year (1.4, 1.5, 1.6) 13 papers published during 2005-2006 CTAT Web site –36,000 unique visitors in 2005 – 8,800 so far in 2006 CTAT Downloads –840 in 2005 (27% CMU & Pitt) –952 so far in 2006 (4.6% CMU & Pitt)

15. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 http://ctat.pact.cs.cmu.edu

16. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 PSLC Example-Tracing Tutors built with CTAT Chinese - tone study Chinese - listen and read French culture French Chemistry (stoichiometry) Geometry (self-assessment) Geometry (on-line testing)

17. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 PSLC Case study - Chemistry (Stoichiometry) Goals: oTest hypotheses about personalization and worked examples oExercise and test the Cognitive Tutor Authoring Tools (CTAT) in a live, web-based experiment Results: oStudents learned: Significant difference between pre and post oNo sig differences due to personalization or worked examples McLaren, B. M. et al. (2006). Studying the Effects of Personalized Language and Worked Examples in the Context of a Web- Based Intelligent Tutor. In the Proceedings of the 8 th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, June 26-30, 2006.

18. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 PSLC Case study - Chemistry (Stoichiometry) Behavior Recorder Flash authoring environment with tutor interface shown Web browser with Example-Tracing tutor embedded in HTML

19. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 CTAT Features used in Case Study Web delivery –Use of Macromedia Flash MX 2004 to build Student Interface –Tutor Shop handles problem sequencing on the web Template-based tutor development –Use of Excel lowers skill threshold even more –More efficient authoring –Easier maintenance Support for experiments –Pre/post tests adminstered and graded automatically –CTAT-built tutors log all student-tutor interactions –PSLC Data Shop provides storage and data analysis facilities

20. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 Development time comparison Past estimates of ITS development 200:1 for robust beta versions used in real classrooms Example-Tracing Tutor Development 25:1 for initial alpha versions: (Koedinger et al, 2004) If creating a robust beta doubles dev time, we would have ~50:1 ratio -- a reduction of 4 times! Koedinger, K., Aleven, V., Heffernan, N., McLaren, B. M., and Hockenberry, M (2004). Opening the Door to Non-Programmers: Authoring Intelligent Tutor Behavior by Demonstration;. In the Proceedings of the Seventh International Conference on Intelligent Tutoring Systems (ITS-2004).

21. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 CTAT Example-Tracing Tutor Web-Delivery Options Java –WebStart –Utility may depend on the situation –We are still gaining experience with this; can be a bit tricky Flash (from Macromedia) –With plug-in, compatible with virtually all browsers –Easy to deploy

22. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 Flash or Java for Authoring and Delivery? CostWeb Delivery Look-and- Feel of Student Interface MaturityAvailability of Developers Flash - License must be purchased from Macromedia (Academic: $100, $150, for authoring only) + More compatible across platforms & browsers + Emphasis on presentation; Easy to include movies, graphics, other visuals (Currently 6 widgets) - First released as part of CTAT in Feb 2005; Much newer than Java - Less established programming environment, so less programmers available Java + NetBeans and Java are free downloads - Less compatible; Use of apps like WebStart for web delivery - Visual elements not as crisp; more difficult to include other media (Currently 15+ widgets) + Available for over 2 1/2 years + More established programming language, so more programmers available

23. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 Movie Showing How an Example- Tracing Tutor is built

24. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 Hands-On Assignment: Build your own fraction addition Example-Tracing Tutor To get started: –Start Flash MX2004 (From Start menu: All Programs:Programming Tools:MX2004) –Start the Behavior Recorder (Double click the “Behavior Recorder for Flash” icon on the desktop) –In Flash open the partially built fraction-addition GUI (W:/Cognitive Tutor Authoring Tools/Projects/Examples/Exercise1- ExampleTracingTutor/fractionAddition.fla) To finish the GUI: Follow the hand-out “Building a student interface for fraction addition” (Because you have a partially built user interface, you can just review the first three pages; begin work in the middle of page 4) To build the Example-Tracing Tutor: Follow the hand-out: “Building a student interface for fraction addition”

25. PSLC Summer School July-August, 2006© Vincent Aleven and Bruce McLaren, 2006 THE END (Of “ Introduction to CTAT and Example-Tracing Tutors ”)