1. An iterative approach to designing scalable and adaptive computer-based science instruction Mari Strand Cary, David Klahr, Stephanie Siler, Cressida Magaro, Kevin Willows, Junlei Li Carnegie Mellon University & University of Pittsburgh

2. 2 Goal of this EdBag? Discuss this iterative research process and its role/prominence in curriculum design, HCI, etc. –Usefulness to researchers and designers (and, more importantly, end-users) –How different is it from “design research” –Difficulties we’ve encountered –Other?

3. 3 Overview of the TED project Curriculum: Experimental design, evaluation, and interpretation Age: 5th-8th grade students Schools: 6 inner city –4 low SES & challenging classroom environments –2 mid-high SES End goal: Computer-based adaptive tutor –1 student : 1 computer in classroom environment –Provides individualized, adaptive instruction –Supplements (does not replace!) teacher

4. 4 “Experimental design” = CVS (Control of Variables Strategy) 1.Simple procedure for designing unconfounded experiments (Vary one thing at a time) 2.Conceptual basis for making valid inferences from data (Isolating the causal path)

5. 5 Does SURFACE affect how far balls roll? VariableRamp 1Ramp 2 SurfaceSmoothRough Track lengthShortLong HeightHighLow BallGolfRubber Ramp 1Ramp 2 SmoothRough Short High Golf ConfoundedUnconfounded

6. 6 What’s the best way to teach CVS? As a society (educators, researchers, and legislators), we don’t know Our research team knows of one effective way…

7. 7 Our basic CVS instruction (used in various forms in past studies) Students design experiments Students answer questions Instructor provides explicit instruction about CVS One domain Short instructional period

8. 8 But… Our brief, focused CVS instruction is differentially efficient and effective for different student populations, settings, and transfer tasks. We want to reach ALL students! To improve our instruction for the entire student population, we must engage in modification & individualization

9. 9 A computer tutor could facilitate differentiated instruction Instruction that suited for full-class setting could be provided by teacher Remaining instruction could be provided by computer tutor –Individualized & self-paced –Provides instruction, practice, and feedback –Teacher freed to provide coaching as needed

10. 10 How are we building our tutor? 4 development phases & Iterative design process

11. 11 4 development phases: 1.Information gathering: What are the novice models students hold and how can we address those? 2.Refining the basic instruction and “going virtual” 3.Building a computer tutor with a few “paths” 4.Building an adaptive computer tutor with a “web” of paths

12. 12 Improve current version & Inform next version Compare against previous version Our iterative design process: Version n Pilot testing Delayed post assessment One-on-one human tutoring Classroom validation study (+ pre, post, and formative assessments)

13. 13 An evolving CVS computer tutor Version 1Version 2Version 3Version 4 Instructional mode Class (teacher)2a) Class (teacher)Class (teacher) Individual (computer) Class (teacher) Individual (computer) Inflexible Flexibility Limited flexibility (differentiation points) Flexible (multiple paths) Adaptive (“web” of paths) Stimuli Simulations Computer interface Physical apparatus Overhead transparencies Simulations Computer interface Simulations Computer interface Instructional components (domain) Procedural & Conceptual (Ramps) Prereq. skills (Auto sales) Procedural (Study habits) Conceptual (Ramps) TBD Discussion Feedback Discussion, paper exchange, researchers Discussion, Computer, researchers TBD 2b) Small groups (us!)

14. 14 What are we learning from each version that will help us design the final, adaptive tutor? VERSION 1 (Completed) Initial list of student biases, misconceptions, errors & areas of difficulty Inventory of successful tutoring approaches familiar domains instruction in prerequisite skills step-by-step approach Student-friendly terminology, definitions, and phrasing Requiring explicit articulation by student

15. 15 A sampling of what students do wrong: Common errors: Vary everything Hold target variable constant and vary other variables Partially confounded Nothing varied (identical) Common justifications: “I don’t know” You told me to test x! Describe their set-up Want to see if x happens Want to see if this setup is better than that setup

16. 16 Why? By accident –misread question –working carelessly Are led astray –by saliency of physical apparatus (e.g., ramps) –don’t understand written representations (e.g., tables) On purpose –different goals (e.g., “engineering”) –misconception of experimental logic –think other variable(s) don’t matter Just guessing

17. 17 VERSION 2a (Complete) & 2b (Fall 2007) Information regarding: Addressing most common problems in full-class instruction using successful tutoring approaches Instructional effectiveness of switching domains effect of emphasizing domain-generality interface usability worksheet usability 2b: Implementation of successful tutoring approaches with small groups (groups will differ in the “paths” they take)

18. 18 VERSION 3 (being developed) Information regarding: division of instruction between teacher and tutor individual tutor usability and pitfalls comparative efficacy of set learning paths efficacy of immediate computer feedback

19. 19 The adaptive tutor will include: Pre-testing and ongoing monitoring of student knowledge Self-paced instruction Diverse topics matching student’s interests An interactive and engaging interface Teacher-controlled and/or computer-controlled levels of difficulty Level of scaffolding, feedback, and help aligned with student’s needs Computerized assessments Logging capability (“level” of output TBD)

20. 20 Discuss! Discuss this iterative research process and its role/prominence in curriculum design, HCI, etc. –Usefulness to researchers and designers (and, more importantly, end-users) –How different is it from “design research” (and would it have gotten funded if we had labeled it that?) –Difficulties we’ve encountered –Other?

21. Questions? Comments? MariStrandCary@cmu.edu Klahr@cmu.edu Funding provided by: Institute of Education Sciences (IES _____)

23. 23 V1 learning examples: VERSION 1 Database of student biases, misconceptions & areas of difficulty Inventory of successful tutoring approaches familiar domains instruction in prerequisite skills step-by-step approach Student-friendly terminology, definitions, and phrasing Requiring explicit articulation of understanding and reasoning Ignore the data or Biased by expectations Create “best” outcome or Most dramatic difference Learn about all variables at once Pets, Sports drinks, Cars, Study habits, Running races Variable vs. Value Experiment Result vs. Conclusion Read carefully, Identify question, Identify variables… Good vs. Fair vs. Informative vs. True “Variable” = something that can change Table format Remembering the target variable Drawing conclusions based on the experiment

24. 24 Stand-alone, detailed lesson plan with visual aids Examples of exp. designs (good and bad) Assessments (formative and summative) Students designing experiments Asks students to explain, justify, and infer Feedback Every version

25. 25 Increasing complexity and adaptiveness Physical apparatus  Virtual simulations Full class  Full class & individual computer use Inflexible  Individually-adaptive & self-paced One domain  Multiple domains

26. 26 What if later versions are less effective than earlier versions? “Stop the presses!” Look for obvious reasons Examine lesson components individually Consider what is missing

27. 27 “Procedures” Test one variable at a time 1.Make the values for the variable you’re testing be DIFFERENT across groups. 2.Make the values for the variables you’re not testing be the SAME across groups.

28. 28 “Concepts” You need to use different values for the variable you’re testing in order to know what effect those different values have. You need to use the same value for all the other variables (hold all the other variables constant; “control” the other variables) so that they can’t cause difference in the outcome. If you use CVS, you can know that only the variable you’re testing is causing the outcome/result/effect.

29. 29 Beyond our classroom instruction… Where on the contextual / abstract continuum should this type of instruction be focused? When? Single vs. multiple domains? Static pictures vs. simulations vs. tabular representations Best mix of explicit instruction, exploration, help, feedback, etc.