1. Hyper-Hitchcock: Authoring Interactive Videos and Generating Interactive Summaries Frank Shipman Texas A&M University Andreas Girgensohn Lynn Wilcox FX Palo Alto Laboratory, Inc.

2. Outline Introduction Introduction Detail-on-demand video Detail-on-demand video Hyper-Hitchcock hypervideo editor Hyper-Hitchcock hypervideo editor Generating hypervideo summaries Generating hypervideo summaries Experience with Authoring Experience with Authoring Conclusions Conclusions

3. Computational Media Computation-based mediums of communication are growing Computation-based mediums of communication are growing E-mail, IM, Blogs, MMORPG, … E-mail, IM, Blogs, MMORPG, … Text, images, animation, … Text, images, animation, … TV/DVD/PC merger (Sony PS3, Media Center PCs) TV/DVD/PC merger (Sony PS3, Media Center PCs) Interactive video Interactive video Vision: Seamlessly interact with characters in the show Vision: Seamlessly interact with characters in the show Reality: Difficult to author even simple interactive videos Reality: Difficult to author even simple interactive videos

4. Support for Hypervideo Authoring Links in video can lead to other video segments Links in video can lead to other video segments Short main video with branches providing additional detail Short main video with branches providing additional detail Hyperlinks to branches just like in Web pages Hyperlinks to branches just like in Web pages Making of a scene in a movie, biography of an actor, different camera angle Making of a scene in a movie, biography of an actor, different camera angle General hypervideo difficult to author General hypervideo difficult to author Simple hypervideo format with only a single active link Simple hypervideo format with only a single active link Novel approach: use automatic video analysis, create an easy-to-use interface, and support simple hypervideo format Novel approach: use automatic video analysis, create an easy-to-use interface, and support simple hypervideo format

5. Uses of Hypervideo Hypervideo well-suited for training video Hypervideo well-suited for training video Overview of topic with links to more detail Overview of topic with links to more detail Viewers can choose video content based on their prior knowledge and current task Viewers can choose video content based on their prior knowledge and current task Home video is more enjoyable if viewers can select content Home video is more enjoyable if viewers can select content Only a small portion of the video is really “usable” Only a small portion of the video is really “usable” Difficult to watch long home videos Difficult to watch long home videos Customize viewing experience for different viewers Customize viewing experience for different viewers More detail on sports events for some family members, more scenes with children for others More detail on sports events for some family members, more scenes with children for others

6. Detail-on-demand Hypervideo General hypervideo General hypervideo Links from objects in video to other video Links from objects in video to other video Requires object tracking Requires object tracking Requires interface for indicating and selecting from multiple links Requires interface for indicating and selecting from multiple links Detail-on-demand video Detail-on-demand video Single link from any video segment Single link from any video segment No anchor regions to simplify viewing and authoring No anchor regions to simplify viewing and authoring

7. Hierarchical Video with Links Video sequences are represented as a containment hierarchy of video elements Video sequences are represented as a containment hierarchy of video elements Elements are video clips or composites grouping other video elements Elements are video clips or composites grouping other video elements Elements are played in sequence Elements are played in sequence Each element can be link anchor or link destination Each element can be link anchor or link destination Anchor for innermost element is available while element is playing Anchor for innermost element is available while element is playing After link destination video is played, play-back continues at the link anchor After link destination video is played, play-back continues at the link anchor

8. Detail-on-demand Links Any video clip or composite can be link anchor or link destination Any video clip or composite can be link anchor or link destination Optional link offsets into destination Optional link offsets into destination Links have labels Links have labels Link return behaviors control the purpose of the link Link return behaviors control the purpose of the link Play from where the viewer left the video Play from where the viewer left the video Play from the end of the source anchor sequence Play from the end of the source anchor sequence Play from beginning of the source anchor sequence Play from beginning of the source anchor sequence Stop playback Stop playback Different behaviors for destination completion or aborted playback Different behaviors for destination completion or aborted playback

9. Hyper-Hitchcock Editor & Player Goals Hyper-Hitchcock editor designed to Hyper-Hitchcock editor designed to Reduce cost of producing interactive video Reduce cost of producing interactive video Make resulting video useful to wider audience Make resulting video useful to wider audience Simple video player to simulate DVD viewing experience Simple video player to simulate DVD viewing experience

10. Hyper-Hitchcock Editor Hyper-Hitchcock evolved from Hitchcock video editor Hyper-Hitchcock evolved from Hitchcock video editor Video clips grouped in piles by similarity (e.g., recording time) Video clips grouped in piles by similarity (e.g., recording time) Workspace to arrange clips Workspace to arrange clips Resize keyframes to trim clips Resize keyframes to trim clips Clips ordered as horizontal or vertical lists Clips ordered as horizontal or vertical lists Place links between clips Place links between clips Group clips into composites Group clips into composites Tree view to visualize containment hierarchy of composites Tree view to visualize containment hierarchy of composites

11. Determining Suitable Video Clips Unsuitability score Unsuitability score Single score for video features such as camera motion and brightness Single score for video features such as camera motion and brightness Estimate camera pan by shifting frames against each other Estimate camera pan by shifting frames against each other Require minimum brightness Require minimum brightness Determine clip boundaries Determine clip boundaries Select clips that fall in “valleys” between unsuitability peaks Select clips that fall in “valleys” between unsuitability peaks Look for areas completely above unsuitability threshold Look for areas completely above unsuitability threshold Peak is clip boundary candidate Peak is clip boundary candidate Enforce the minimum length requirement Enforce the minimum length requirement Trimming clips Trimming clips Select portion of clip with minimal area under the curve Select portion of clip with minimal area under the curve Expand the area for longer requested portions Expand the area for longer requested portions

12. Selected Clip Portions

13. Trimming Clips in the Workspace Best five seconds of clip selected by default Best five seconds of clip selected by default Resizing keyframe changes length of clip Resizing keyframe changes length of clip Picks the best portion around initial five-second portion Picks the best portion around initial five-second portion Start and end can jump to sentence boundary silence Start and end can jump to sentence boundary silence Clip start and/or end can be locked in timeline Clip start and/or end can be locked in timeline Locked ends can be dragged Locked ends can be dragged Audio energy visualized in timeline to spot words and sentences Audio energy visualized in timeline to spot words and sentences

14. Video clips to be grouped into a composite (keyframe area proportional to clip length) Composite visualizations cross (area error minimized) sliding dividers (area proportional to clip length) Visualizing Video Composites

15. Attaching Links to Clips and Composites Link anchors and destinations can be clips, composites, or elements inside composites Link anchors and destinations can be clips, composites, or elements inside composites Color-coding and position indicates link attachment in workspace Color-coding and position indicates link attachment in workspace Links in and out of composite Blue: attached to composite Red: attached to element Dashed: between composite and element

16. Hypervideo Player Video player with controls for following and returning from links Video player with controls for following and returning from links Several improvements based on user feedback Several improvements based on user feedback First version indicated links in timeline and showed the label for the active link First version indicated links in timeline and showed the label for the active link Next version showed labels in timeline Next version showed labels in timeline Current version includes keyframes for active link and for link history Current version includes keyframes for active link and for link history User study suggests further improvements User study suggests further improvements

17. Navigation Aids in Hypervideo Player Keyframe list for navigation history Keyframe list for navigation history Shows followed links with more remote navigation indicated by size of keyframe Shows followed links with more remote navigation indicated by size of keyframe Label of followed link as video caption Label of followed link as video caption Link indicator in timeline (in blue) Link indicator in timeline (in blue) Link labels and keyframes in timeline Link labels and keyframes in timeline Keyframe grows, label completes, and link recolored when active Keyframe grows, label completes, and link recolored when active

18. Impressions from Users Pilot editing study with two participants Pilot editing study with two participants Subjects own video Subjects own video report on a trip to Japan report on a trip to Japan family outing to mountain bike race family outing to mountain bike race Few problems but link return behavior was confusing for one participant Few problems but link return behavior was confusing for one participant Study of hypervideo player (6 subjects) Study of hypervideo player (6 subjects) Plumbing training video re-authored as hypervideo Plumbing training video re-authored as hypervideo All participants were able to find answers quickly in the hypervideo (9 - 20 minutes in 60-minute video) All participants were able to find answers quickly in the hypervideo (9 - 20 minutes in 60-minute video) Navigation through video can be confusing. As player was altered to better support navigation, playback becomes less like video. Navigation through video can be confusing. As player was altered to better support navigation, playback becomes less like video.

19. Generating Hypervideo Summaries Locating content in video is time consuming. Locating content in video is time consuming. Much effort into generating “good” video summaries. Much effort into generating “good” video summaries. But what is good for one task is not good for another. But what is good for one task is not good for another. Generate hypervideo summaries that allow users to determine the level of detail viewed. Generate hypervideo summaries that allow users to determine the level of detail viewed.

20. Process for Generating Summaries Determine number of summary levels Determine number of summary levels Based on length of source video Based on length of source video Select clips to include in each summary level Select clips to include in each summary level Clips found by subdividing takes by camera motion Clips found by subdividing takes by camera motion Select clips via clip distribution, take distribution, or best-first algorithms Select clips via clip distribution, take distribution, or best-first algorithms Add links between summary levels Add links between summary levels Group clips by takes Group clips by takes Links between clips from the same take using simple take-to-take or take-to-take with offsets algorithms Links between clips from the same take using simple take-to-take or take-to-take with offsets algorithms

21. Selecting n clips out of m candidate clips Selecting n clips out of m candidate clips Evenly distribute in among candidate clips Evenly distribute in among candidate clips Selects more clips from takes with many recognized clips. Selects more clips from takes with many recognized clips. Good when take includes more than one topic/activity. Good when take includes more than one topic/activity. Bad when lots of clips are for a single topic/activity. Bad when lots of clips are for a single topic/activity. Clip Distribution Selection

22. Evenly distribute in time and takes Evenly distribute in time and takes Divide video duration into n time segments Divide video duration into n time segments Select clip nearest center of segment in take not already represented. Select clip nearest center of segment in take not already represented. Bad when take includes more than one topic/activity. Bad when take includes more than one topic/activity. Good when lots of clips are for a single topic/activity. Good when lots of clips are for a single topic/activity. Take Distribution for Clip Selection

23. Best-first Clip Selection Assumes human or automated ordering of value of clips Assumes human or automated ordering of value of clips Simply selects n best clips Simply selects n best clips Good in cases of edited video – not currently applicable for unedited video. Good in cases of edited video – not currently applicable for unedited video. Best can be introductory material Best can be introductory material Best can be highlights of material Best can be highlights of material

24. Links in Video Summary All clips from a take are grouped into composite All clips from a take are grouped into composite Single clip from take or take composite is used as link anchor and destination Single clip from take or take composite is used as link anchor and destination Simple take-to-take algorithm Simple take-to-take algorithm Links take composite to take composite Links take composite to take composite Best when single activity divided into multiple clips. Best when single activity divided into multiple clips. Take-to-take with offsets algorithm Take-to-take with offsets algorithm Each clip from take links to take composite in next level with offset to temporally closest clip. Each clip from take links to take composite in next level with offset to temporally closest clip. Best when clips portray multiple activities in take. Best when clips portray multiple activities in take. Complete takes for entire video Complete takes for entire video

25. Automatically Generated Summary 0:33 3:35 14:44 60:42 A four-level hypervideo summary of a one-hour source video. Lower levels provide more detail. Time

26. Future Systems Research MPEG-4 players support hypervideo MPEG-4 players support hypervideo No good authoring tools No good authoring tools Demand for easy-to-use tools for quickly editing hypervideo Demand for easy-to-use tools for quickly editing hypervideo Research problems for hypervideo player Research problems for hypervideo player Navigation: easy to get lost; how to visualize place Navigation: easy to get lost; how to visualize place Viewing: user interface for TV remote control; other input devices such as data gloves Viewing: user interface for TV remote control; other input devices such as data gloves Keyframes for links represent a first step towards solutions Keyframes for links represent a first step towards solutions Research problems for hypervideo summaries Research problems for hypervideo summaries Different summaries for different genres (e.g., home video, produced training video) Different summaries for different genres (e.g., home video, produced training video) Clip selection and link placement dependent on the genre Clip selection and link placement dependent on the genre

27. Hypervideo in Practice

28. Experience with Use Nine hypervideos each authored by 1-2 students in a Computers and New Media class Nine hypervideos each authored by 1-2 students in a Computers and New Media class Recorded up to 1 hour using a DV camera then authored hypervideo in HH Recorded up to 1 hour using a DV camera then authored hypervideo in HH Students had about 1 week for authoring activity. Students had about 1 week for authoring activity.

29. Riding Down University Drive Most directly maps geographic structure Links are choices to stop in at sites along road.

30. Perspectives on Bridges Shows bridges between College Station and Austin. Presents different perspectives (roadway, construction, wildlife) in order. Links are to seeing different perspective.

31. “In Danger” Buildings Visits to buildings identified for demolition on TAMU campus. Links are to seeing interior and details.

32. International Dance Festival Shows bits of performances. Brief “how-to use hypervideo player” at start. Links are to seeing more and to next performance.

33. Game Walkthrough Shows what happens while playing game. Links represent choice points in game.

34. Preparing for the In-Laws How-to for cleaning and cooking, includes constant audio commentary. Links are to greater detail.

35. How To Make a Sandwich Presents choices in sandwich creation. Includes video of options. Links are to select another option.

36. Music Hypervideo public-domain video, remixed audio

37. Home Hypervideo idiosyncratic structure

38. Feedback on Hyper-Hitchcock

39. Evaluation of Hyper-Hitchcock Easy to use hypervideo player, to work with video clips, to create composites. Easy to use hypervideo player, to work with video clips, to create composites. Not as easy to manipulate composites and create links. Not as easy to manipulate composites and create links. Spatial parser correctly interpreted composites. Spatial parser correctly interpreted composites. Link labels were valued by many. Link labels were valued by many. Audio energy graph was only valued by a few. Audio energy graph was only valued by a few. Automatic segmentation of source video into clips and automatic selection of in/out points did not help. Automatic segmentation of source video into clips and automatic selection of in/out points did not help.

40. Structuring Problems Authoring the two separate structures confused some students Authoring the two separate structures confused some students Composition of video clips into linear videos Composition of video clips into linear videos Links between linear videos Links between linear videos Need for greater explanation of how the two structures interact Need for greater explanation of how the two structures interact

41. Suggested Features Ability to have more than one link active at a time. Ability to have more than one link active at a time. Ability to specify for video to stop playing at decision point. Ability to specify for video to stop playing at decision point. Separate audio editor Separate audio editor Text overlays and video effects Text overlays and video effects

42. Roles of Links Detail Links Detail Links Prerequisite Links Prerequisite Links Related Information Links Related Information Links Alternate View Links Alternate View Links Action Choice Links Action Choice Links

43. Hypervideos Mimicking Hypertext Detail links resulted in sieves. Detail links resulted in sieves. Some sieves included links from one detailed section to the next (shortcuts). Some sieves included links from one detailed section to the next (shortcuts). Related information links also created tangles. Related information links also created tangles. Alternate view links likely to generate mirrorworld or counterpoint structures. Alternate view links likely to generate mirrorworld or counterpoint structures.

44. Hypervideo Differences Split/joins and mirrorworlds used to merge alternate courses of actions or perspectives. Split/joins and mirrorworlds used to merge alternate courses of actions or perspectives. Return behaviors used to cause merging paths, thus the structure looks like a sieve. Return behaviors used to cause merging paths, thus the structure looks like a sieve.

45. Summary Hyper-Hitchcock used to author documentary, how-to, music, and home hypervideos. Hyper-Hitchcock used to author documentary, how-to, music, and home hypervideos. Links in hypervideos used for: details, prerequisites, alternate views, action choices, and related information. Links in hypervideos used for: details, prerequisites, alternate views, action choices, and related information. Structures in hypervideo were impacted by the inclusion of return behaviors. Structures in hypervideo were impacted by the inclusion of return behaviors.

46. Conclusions Detail-on-demand video well-suited for training and home video Detail-on-demand video well-suited for training and home video Simple interaction style appropriate for DVD-player interfaces Simple interaction style appropriate for DVD-player interfaces Enables wide range of authors due to emphasis on ease of learning and use over richness of interaction Enables wide range of authors due to emphasis on ease of learning and use over richness of interaction Early user study results of both authors and viewers very encouraging Early user study results of both authors and viewers very encouraging Hypervideo summaries Hypervideo summaries Remove need for single context-free video summary Remove need for single context-free video summary Multiple clip selection and link generation algorithms Multiple clip selection and link generation algorithms Act as starting point for human-authored summary Act as starting point for human-authored summary