1. Microsoft Faculty Research Summit 2007 Adaptive Book: A Platform for teaching, learning and student modeling Ananda Gunawardena School of Computer Science Carnegie Mellon University & Vince DeStasio CIO and Associate Professor in Chemistry Grove City College Pennsylvania

2. Microsoft Faculty Research Summit 2007 Outline Introduction What is Adaptive Book? Adaptive Book Authoring Tools Just in time learning modules Adaptive Markup Repository Assessment of Student Reading Pilot Results Future Work

3. Microsoft Faculty Research Summit 2007 Adaptive Book Project Tablet PC is an ideal platform for electronic textbooks Ability to use the pen to annotate, highlight and share markups Online textbooks cannot be –Just the pdf versions of their printed versions Flexible but Limited use Cheap but hard to read Our efforts is to design the textbook of the future

4. Microsoft Faculty Research Summit 2007 Introduction Effective teaching is not confined to the classroom—it is successful when it enhances interest and generates thinking beyond the classroom setting. A great deal of customization and personalization of the content is needed to generate the interest Adaptive Book is a platform for customization, personalization and understanding of student behavior

5. Microsoft Faculty Research Summit 2007 Adaptive Book A software platform developed in C# /.net –Developed by TextCentric,Inc (CMU Spin-off) Research Base at Carnegie Mellon –Usability (Human Computer Interaction) –Adaptive (Machine Learning) Adaptive Book User Interface –Navigation, search, markup tools (highlighting, annotating, and linking), book marking –Labeling, archiving and searching markups –Adaptive Book combines textbook content with all other related material

6. Microsoft Faculty Research Summit 2007 Adaptive Book UI

7. Microsoft Faculty Research Summit 2007 Survey Slide 1 What do you think the students initial reaction to Adaptive Book was? –A) They remained skeptical –B) They embraced it whole-heartedly and could immediately see the benefit –C) They liked the idea, but it had too many "kinks" to be immediately useful –D) All of the above –E) none of the above

8. Microsoft Faculty Research Summit 2007 Creating an Adaptive Book Adaptive Book is a thin client program Any SCORM/IMS content package can be imported into Adaptive Book A content authoring tool creates these content packages. In other words anyone can create an Adaptive Book package using their own content and/or textbook chapters

9. Microsoft Faculty Research Summit 2007 Instructor content Publisher content professor Rights and Royalty Management Adaptive Book

10. Microsoft Faculty Research Summit 2007 Adaptive Book Authoring Tool Uses a HTML/XML content repository to find chapters of the book and supplements Each chapter of the book is tagged as a Sharable Content Object (SCO) Uses a simple drag and drop menu to select the chapters and supplements needed to create the custom book Program generates the table of content and package the book as a SCO

11. Microsoft Faculty Research Summit 2007 Importing a SCO

12. Microsoft Faculty Research Summit 2007 Survey Slide 2 Do you think combining textbook chapters with your own material is a useful way to package a course –A) Strongly Agree –B) Agree –C) Neutral –D) Disagree –E) Strongly Disagree

13. Microsoft Faculty Research Summit 2007 Markup Concept Research shows marking up a text while reading enhances the learning Markup is defined as a semantically related set of objects consists of highlights, annotations and web links. Markups automatically generate its own metadata as well as anyone can add other metadata to markups before saving Markups are stored in a searchable repository. Search and find markups related to a certain concept –What is the “best” markup to learn topic A? –A search algorithm specifically designed to find related markups Assign Rights to markups –Private, public, protected

14. Microsoft Faculty Research Summit 2007 Sample Markup

15. Microsoft Faculty Research Summit 2007 Students Sign up for Markup Services

16. Microsoft Faculty Research Summit 2007 Students can create buddy groups

17. Microsoft Faculty Research Summit 2007 Students Define their buddy group or Who has the access to their markups

18. Microsoft Faculty Research Summit 2007 Mark-up List

19. Microsoft Faculty Research Summit 2007 Assigning Rights to Markups

20. Microsoft Faculty Research Summit 2007 Reading a markup

21. Microsoft Faculty Research Summit 2007 Survey Slide 3 If you have the ability to markup the textbook and related material, how would you use the feature –A) I would use it to make notes for myself –B) I would markup the text and send it to students before class –C) I would markup the text as an answer to a frequently asked question and share –D) I would create a repository of reusable markups that addresses various concepts –E) I would encourage students to markup the text as they read the book –F) I do not find this feature useful –F) Other

22. Microsoft Faculty Research Summit 2007 Just in time Learning Modules Markups are important learning tools Students learn better from marked up text Worksheets to reading the book –This is a compromise Instructors can create just in time learning modules using markup tools Select content, highlight, place sticky notes and URL’s and package them as a markup object Save the markup to a customized learning objects (CLO) repository Others can search and find the markup, disassemble and customize to their needs

23. Microsoft Faculty Research Summit 2007 A Just-in-time learning module with highlights, and digital ink notes

24. Microsoft Faculty Research Summit 2007 More on Markups Markups with learning paths can be created by individual instructors Search and find the Markup, and import to Adaptive Book

25. Microsoft Faculty Research Summit 2007 Learning Objects Repository Three types of learning objects stored in the repository Book Chapters that can be used to build a custom book Individual markups packaged as IMS/SCORM with access rights Customized Learning Objects (CLO’s) that can be retrieved and modified using Adaptive Book

26. Microsoft Faculty Research Summit 2007 Modeling Student Behavior One of the focuses of our research –Do students read the book and other notes? –If so, what do they read? not pages, what specific content? –How do they construct knowledge? Does the Tablet PC help students move from informal sketching to formalization? –What can we do to capture that?

27. Microsoft Faculty Research Summit 2007 Modeling Student Behavior If we are successful in capturing an accurate assessment of student thinking, what can we do with that data? Our pilots involve constructing activities that “requires” the student to show specific things in the textbook and notes that they found useful Student markup is then compared to an “expert” markup Our preliminary data show some interesting correlation between student performance and their perception of what was important in completing the assignment More work is needed to accurately model student behavior This is ongoing research

28. Microsoft Faculty Research Summit 2007 Pilot Results We have used Adaptive Book and markup repository with several institutions and middle schools Many positive Student Comments but lot needs to be done Interesting results on what students read Sample comments…

29. Microsoft Faculty Research Summit 2007 Some Student Comments “I like the ability to find a relevant markup that can help solve some of the programming problems” “It is nice to receive markups from the instructor before the lecture” “Good thing about the Adaptive Book is that instructor can create a markup linking textbook concepts and relevant programming examples” “receiving an answer to a FAQ as a markup is a very useful feature” “Adaptive Book search feature needs improvements”

30. Microsoft Faculty Research Summit 2007 Future Work More focus on the impact of Tablet PC and Adaptive Book Technology Making Adaptive Book a research platform for modeling student behavior Creating a dynamic sketch environment for conceptual understanding and automatic code generation Informal to formal stages of learning Better markup analysis and classification

31. Microsoft Faculty Research Summit 2007 Ink-based Technology Fundamental Questions –Do ink-based technologies improve teaching and learning? What are the differences and similarities with standard PC- based technology in education? How do ink-based technologies compare/compete with paper? Two promises –Better document management Electronic repositories –Sketch recognition Smart paper Real time feedback on ink Perhaps a better platform for interacting with electronic documents –ebooks, note taking etc..

32. Microsoft Faculty Research Summit 2007 Carnegie Mellon Projects HP Technology for Teaching Grant – 2004 –21 HP 1100 Tablet PC’s –Other equipment HP Technology for Teaching Leadership grant - 2005 –42 HP 4200 Tablet PC’s –Other equipment Qatar Foundation Grant - 2005 –25 HP 4200 Tablet PC’s and Research funding Microsoft Research Grant - 2006 –$150,000 funding for a 2-year impact study –With Grove City College Tablet PC program

33. Microsoft Faculty Research Summit 2007 HP Technology for Teaching Award 2004 2004-2005 study of the integration of Tablet PC’s 38 Computer Science Freshman participated in the study Students “owned” the tablet PC for the duration of the pilot (typically one semester) –85% of the CMU students who participated in the pilot owned laptops –60% of the students used “tablet PC” as their primary machine

34. Microsoft Faculty Research Summit 2007 HP Technology for Teaching Award 2004 Ink-based Software used –Adaptive Book Technology (CMU-TextCentric) –MS Windows Journal Primary activities –Reading and annotating the book –Creating and saving worksheets –Sharing worksheets using a repository Motivation –CMU bought 1 Tablets PC that was raffled at the end of the semester –Those who contributed to the ink-based worksheet repository received extra coupons for the raffle.

35. Microsoft Faculty Research Summit 2007 HP Technology for Teaching Award 2004 Research Findings –38 students surveyed –60% of the students used Tablet as their primary device –We used a control group of 20 students who did not have tablet PC’s –taught by the same professor (Don Slater), same introductory programming course –Final analysis showed slight improvement in grades in the Tablet group. But not statistically significant to draw conclusions –85% of the students liked the idea of electronic note taking Easy organization, storage and retrieval –45% of the students would consider buying a tablet PC, next time they shop for a PC Smaller screen and performance were cited as reasons for not buying a tablet

36. Microsoft Faculty Research Summit 2007 HP Technology for Teaching Leadership Award 2005-2006 42 HP Tablet PC’s Target Group –Sophomore/Junior students taking data structures and algorithm courses –Initial survey done to select 35 students with varied criteria's Major and GPA Ownership of PC General attitude towards electronic material –Control group is 40 students who were not selected or did not want to participate Mac’s, unix owners etc.

37. Microsoft Faculty Research Summit 2007 HP Technology for Teaching Leadership Award 2005-2006 Ink based Software tools –Adaptive Book with Markup analysis tools –Classroom presenter –Windows journal –Data structure visualizer Customization of MIT Physics illustrator project Status of the project –Ongoing –Preliminary survey data available –Student markup analysis data available

38. Microsoft Faculty Research Summit 2007 HP Technology for Teaching Leadership Award 2005-2006 Support tools –Configured with CMU’s wireless cluster network –All Tablets belong to a virtual cluster network Pilot study in understanding use of mobile devices as computer cluster machines (Gates building) Automated configurations –Network isolation –Software access privileges –Automated backup scripts All related journal notes collected for analysis

39. Microsoft Faculty Research Summit 2007 HP Technology for Teaching Leadership Award 2005-2006 Research Activities –Pre-lecture markup –Post-lecture markup –Automated analysis of markups With an expert as well as with community Various data mining techniques employed Research Question –What is the impact of pen-based technology in learning? Ink versus paper More reading and markup activity Electronic note taking Methods –Surveys of users and non-users of tablet technology –Analysis of pre and post lecture markup activities

40. Microsoft Faculty Research Summit 2007 Qatar Foundation Grant 2005- 25 HP Tablet PC’s at CMU-Qatar campus in Education City Entire CS sophomore class had the tablets Observations –Immediate acceptance of the technology Only 1 out of 20 students took paper notes Had a hard time getting the machines back at the end of the year Students used electronic note taking in all classes –Classroom presenter used to create interaction Send/receive annotations – professor-students Great for middle east –Adaptive Book was used to increase Textbook reading and interaction Reading comprehension

41. Microsoft Faculty Research Summit 2007 Qatar Foundation Grant 2005- Informal Study Results Potential benefits –Teaching Ink grading and archiving of homework Archiving of all classwork Pittsburgh TA’s used/uses IM tools with ink –Learning Use of classroom presenter tool Ability to annotate English textbooks in Arabic Improving reading comprehension Results

42. Microsoft Faculty Research Summit 2007 Microsoft Research 2-year assessment study 2006-2008 Collaborative project –Carnegie Mellon University –Grove City College Research subjects from Grove City College –2400 HP 4200 Tablet PC’s Data from over 1200 students Wide range of disciplines –CS, Economics, English, Business Software –Adaptive Book (unlimited licenses) –DyKnow vision (600 licenses per year)

43. Microsoft Faculty Research Summit 2007 Other related Projects Ellis School 8 th Grade Geometry project –Adaptive Book (CMU/TCI) –Geometers sketch pad (Key Press) –Classroom presenter (UW) –references

44. Microsoft Faculty Research Summit 2007 What is the future of textbooks Textbook is an integral component of a course –It often gives a different viewpoint with more details –Extra examples, problems and solutions –It is one place students can find answers to most of the questions However, future of textbooks are threatened by online content –Wikipedia’s –Google’s

45. Microsoft Faculty Research Summit 2007 What is the future of textbooks? It is clear that use of online content by students are growing It is also clear that textbooks demand is going down –Publishers increase the price –Create extra online content and sell the book even higher prices

46. Microsoft Faculty Research Summit 2007 Electronic Books At Carnegie Mellon we are designing the future textbook using Human Computer Interaction principles We see textbook of the future as a platform We focus on three things –Usability –Adaptability –Community Online textbooks can be extremely useful and much superior to its printed version, when all three conditions are met Tablet PC’s and new fonts make reading much closer to the experience of reading a printed book

47. Microsoft Faculty Research Summit 2007 Usability Markup tools –Highlights –Short notes –Digital Review Sheets Navigation and Search tools –Search by keyword –Search by concepts –Search by markups Multiple Views –Typical textbook view –Question view –Concept view

48. Microsoft Faculty Research Summit 2007 Adaptability How good is my markup? How do I know I am reading what I am supposed to read? Answer: Use the markup analysis tool –Create a markup –Submit (or synchronize with) to a website –Compare to an “expert” markup –Compare to a “community” markup Lots of applications –Measuring the reading comprehension

49. Microsoft Faculty Research Summit 2007 Community Electronic textbooks are platforms. Find out exactly what other people are reading Comment about content online Find community markups to help study Classify markups –Private –Protected –public

50. Microsoft Faculty Research Summit 2007 Security of Content Digital Rights Management –Publishers biggest worry Tablet PC’s and other ink based devices can help establish the authenticity of the user –Digital ink signatures –Early research

51. Microsoft Faculty Research Summit 2007 Adaptive Book The Thin Client Professor 3a.Definitions 3b. Important 3c. Concepts Mark-ups Sticky Notes Student Textbooks Research VPN* Student Assignments 3. Professor mark up Content 4. Professor ask questions, make assignments. 2. Professor’s Lesson Plan 6. Students Read Content (don’t see highlighted text) 1. Content Test 7. Students learn Content with AB Professor monitor’ individual student’s progress. Self-paced remediation without reprisal! Case Studies *Virtual Private Network 8. Students are Tested 3d. Key Words Professor uses highlighter pen for develop learning paths (Mark-ups), Dean

52. Microsoft Faculty Research Summit 2007 Case Study CMU Computer Science 17 Students Fall Semester 2006

53. Microsoft Faculty Research Summit 2007 The Ellis Survey 7 Actual Users

54. Microsoft Faculty Research Summit 2007 The Ellis School

55. Microsoft Faculty Research Summit 2007 The Survey 37 Questions 37. The lease important feature, the ability to change the text font size. 1. The Tablet PC is a Notebook PC and More 35. Sharing Markups 36. Using Digital Books 2. Highlight text 3. Extends use

56. Microsoft Faculty Research Summit 2007 Adaptive Book & Tablet Math Whiz Ananda Gunawardena School of Computer Science Carnegie Mellon University

57. Microsoft Faculty Research Summit 2007 Tablet Math Whiz certified remediate Adaptive book Markup repository pretest professor group

58. Microsoft Faculty Research Summit 2007 Adaptive Book A software platform developed in C# /.net –Developed by TextCentric,Inc (CMU Spin-off) Research Base at Carnegie Mellon –Usability (Human Computer Interaction) –Adaptive (Machine Learning) Adaptive Book User Interface –Navigation, search, markup tools (highlighting, annotating, and linking), book marking –Labeling, archiving and searching markups –AB combines textbook content with all other related material

59. Microsoft Faculty Research Summit 2007 Adaptive Book UI

60. Microsoft Faculty Research Summit 2007 Creating an Adaptive Book Adaptive Book is a thin client program Any SCORM/IMS content package can be imported into Adaptive Book An authoring tool creates these content packages. Instructors create their custom Adaptive Book Anyone can create an Adaptive Book package using their own content and/or textbook chapters

61. Microsoft Faculty Research Summit 2007 Importing a SCO

62. Microsoft Faculty Research Summit 2007 Markup Concept Markup is a semantically related set of objects consists of highlights, annotations and web links. Markups automatically generate its own metadata as well as anyone can add other metadata to markups before saving Markups are stored in a searchable repository. Search and find markups related to a certain concept –What is the “best” markup to learn topic A? –Google type search engine Assign Rights to markups –Private, public, protected

63. Microsoft Faculty Research Summit 2007 Sample Markup

64. Microsoft Faculty Research Summit 2007 Submit your Markup

65. Microsoft Faculty Research Summit 2007 Get your markup Graded

66. Microsoft Faculty Research Summit 2007 Questions: guna@cs.cmu.edu Thank Youguna@cs.cmu.edu

67. Microsoft Faculty Research Summit 2007 Current Collaborators: Andrew Owens (Cornell) Jay Heo, Dan Horbatt & Chantelle Humphreys(CMU) Research Partially supported by: Qatar Foundation, Microsoft, HP,CMU Ananda Gunawardena, Carnegie Mellon University, CS Department John Barr, CMU-Qatar & Ithaca College

68. Microsoft Faculty Research Summit 2007 Background My semester in Qatar Teaching Data Structures and Algorithms Regular Challenges in the course –Challenging concepts –Broad curriculum –Highly Technical –Large programming assignments Qatar Challenges –Student readiness to accept 15-211 –English reading difficulties

69. Microsoft Faculty Research Summit 2007 Assignments –Frequent Reading and Writing assignments Technology Framework –Tablet PC’s –Adaptive Book Process –Reading Strategies –Reading Comprehension Implementation –Fall 2005 – Ananda Gunawardena –Fall 2006 – John Barr –Fall 2007 – Bob Monroe

70. Microsoft Faculty Research Summit 2007 Put more emphasis on reading in the course Reading assignments based on concepts or problem sets Coordinating with Reading Strategies –Pre-Lecture(submit once before the lecture) –Post-Lecture (after lecture) Ask the student to create a markup “Compare” with instructor markup –Similarity score Improve the markup iteratively Meet a reading threshold

71. Microsoft Faculty Research Summit 2007 Sample Reading Assignments –BST –Graphs –Hashing –Sorting –Heaps Demo –How to setup the assignment –How to submit an expert markup –How do students read in the course –Marking up what they read –Submitting so they can get a score –Iteratively improving the score

72. Microsoft Faculty Research Summit 2007 Observations Using reading as part of the course –English speaking abilities vary among Qatari students –reading technical texts in English is difficult –very resistant to being forced to read –Initially didn't like their reading being evaluated.

73. Microsoft Faculty Research Summit 2007 Changing the perception changes about using a book –By the end of the course good students felt that reading helped them. –The initial reaction seems to have been partly a result of a fear that they couldn't do the reading adequately. –By the end of the course, they realized that they had opportunities to improve their reading score, had figured out how to do the reading correctly –realized that doing the reading helped them in other parts of the course. –Not ALL students felt this way. –The better students, who were more careful about doing the readings, in general had more positive responses than poor students who saw the reading assignments as chores.

74. Microsoft Faculty Research Summit 2007 Specific challenges in reading technical material –Acceptable general reading skills –poor technical English skills. This makes reading time consuming. students may read, highlight, and outline material and not really understand it. –Combine reading assignments with questions that force students to act on the material that they've read. –This reinforces the material in the student's mind, forces them to think about what they've read, and helps make the book material concrete.

75. Microsoft Faculty Research Summit 2007 The role of the instructor in encouraging students to read in technical courses –Students in Qatar will, for the most part, not read without direct intervention of the instructor. –They would much rather use powerpoint slides, lecture notes, and one-on-one conversations with the instructor in office hours to learn the material. –We found that only when the reading was tested in some manner students would do the readings.

76. Microsoft Faculty Research Summit 2007 Role of reading in technical courses –Technical courses, especially in computer science, consist of important concepts, theories, and arcane programming details. –It is relatively easy to convey concepts, theories, and broad programming principles in class. It is hard to discuss details unless students already have a high-level grasp of the material. –Under these circumstances, reading plays three parts in technical courses. students need a source where they can gain at least an overview of the material before class. Forcing them to read the relevant material before coming to class In Qatar, students primarily rely on class notes and powerpoint slides. But textbooks can provide a much richer resource for review. The last role of reading is provide a source for the details that were not covered in class. In this sense, books provide a reference.

77. Microsoft Faculty Research Summit 2007 How best to integrate reading assignments into CS courses –Readings should be made part of the course –Students must be forced to read relevant material before class. This can be done by requiring them to submit markups or by short in-class quizes. This ensures that students are somewhat familiar with the material before class begins. –It's also useful to have students redo their reading after class. This will encourage them to look to the textbook when reviewing the material instead ofthe class lecture notes or slides. –Require students to submit a markup before class. They could be allowed to only submit one markup, no revisions. Then after the class, they could be allowed to submit revisions. Their final grade on the markup would be some combination of the initial and final submissions.

78. Microsoft Faculty Research Summit 2007 Results

79. Microsoft Faculty Research Summit 2007