Lecturing with Digital Ink Richard Anderson University of Washington

Lessons learned from the Classroom Presenter project Classroom Pedagogy Teaching with ink HCI Ink based presentation Multimedia Analysis of lecture artifacts

Classroom Presenter Integration of slides and digital ink using Tablet PC Key ideas: Ink overlay on images Distributed application Many other systems also support ink and slides

Classroom Presenter as a distributed application Designed as distributed application for distance learning Enables many scenarios Mobility Walking and talking Sharing materials with students Note taking Classroom interaction Student submissions

Deployments Estimated use in at least 100 courses Wide use inside of computer science Push for adoption outside of CS Lecture archives from UW Professional Master’s Program Several hundred hours of recorded audio, video, and ink.

Distance Learning Classes

“Typical ink usage”

Planning for ink usage

Ink use in presentation Cognitive load Only limited attention available for computer while lecturing Linkage with speech Close tie between ink and speech

Cognitive load Limited feature use Even color change unusual User interface must be simple (and robust) Cannot give feedback to user Many actions appear to minimize mental effort Color change only for contrast Reliance on screen erase

Understanding Attentional Marks Properties Brief, simple markings Occur with speech Augment meaning of speech Ad hoc form Is there a linguistic context in which to understand these marks?

Spontaneous Hand Gestures Spontaneous Hand gestures [McNeill]: are synchronous w/speech are co-expressive w/speech lack standard of form Attentional marks share these properties.

Gesture Types: Iconic

Gesture Types: Deictic & Cohesive

Analysis of digital ink Understand ink usage Motivation: inform development of ink based applications Archiving Search, Summarization, Transcription Lecture based Improved rendering, note taking, accessibility

Ink classification Textual Diagrammatic Attentional % of strokes% of episodes BCB+CBC Attentional Diagram Writing Other Coding of six hours of lecture

Goals Understand usage “in the wild” Cannot expect lecturers to modify behavior Determine opportunities for automatic analysis Identify challenges

Methodology Study of recorded classes Best data set: Professional Master’s Program Distance courses Audio, Video, Ink archives HCI, Compilers, Programming Languages, AI, Transaction Processing

Attentional ink Problem – content matching Identify slide content referred to by ink Study Implement basic algorithms to match attention marks to slide content Compare results with human coders

Attentional ink Determine the lecturer’s intent: Determine level to parse the content

Attentional ink Challenges Recognition of attentional ink on text Difficult example:

Handwriting How well does handwriting recognition work on “typical” instructor writing? Domain has many challenges

Recognition Study Studied isolated words/phrases written on slides Removed non-textual ink Fed through the Microsoft Handwriting Recognizer No training

Recognition Examples The Good: The Bad: The Ugly:

Handwriting Reco Results ExactAlternateCloseNone Prof. A 16 (88%)1 (6%)0 (0%)1 (6%) Prof. B 146 (59%)26 (10%)6 (2%)71 (29%) Prof. C 18 (42%)5 (11%)1 (3%)19 (44%) Prof. D 262 (61%)45 (11%)9 (2%)111 (26%) Prof. E 408 (79%)46 (9%)2 <(1%)58 (11%) Total 850 (68%)123 (10%)18 (1%)260 (21%)

Joint Writing and Speech Recognition Can we use handwriting recognition with speech recognition together to improve accuracy? Co-expression of ink and speech Are written words spoken as well? Can speech disambiguate handwriting? Can handwriting disambiguate speech?

Examples Difficult for Speech and Ink Recognition Difficult Written Abbreviations Speech/Ink Used to Disambiguate Ink/Speech

Experiment Examined instances of isolated word writing Selected word writing episodes at random but uniformly from the various instructors Generated transcripts manually from the audio Checked whether the instructor spoke the exact word written Measured the time between the written and spoken word

Speech/Text Co-occurrence Results

Activity Recognition Identifying slide corrections

Example Results

Diagrammatic ink How do instructors use diagrams Basic legibility Observed behaviors Diagram phasing Locality of expression

Typical diagram Basic, irregular shapes Difficult labels Attentional ink

More examples

Zipf diagram

Stroke order

Diagram phasing

More phasing

Top arrows: “Not there” Separate wins indicated together Locality in diagrams

Summary Pedagogy with ink How is ink used in conjunction with content and speech to express information Presentation with ink Low attention task Analysis of ink usage Extracting meaning from archived lectures

Resources cs.washington.edu/education/dl/presenter/ Software Downloads Papers Contact info Richard Anderson, Ruth Anderson, Craig Prince,