1. 2003.04.14 - SLIDE 1IS246 - SPRING 2003 Lecture 21: Research Multimedia Systems IS246 Multimedia Information (FILM 240, Section 4) Prof. Marc Davis UC Berkeley SIMS Monday and Wednesday 2:00 pm – 3:30 pm Spring 2003 http://www.sims.berkeley.edu/academics/courses/is246/s03/

2. 2003.04.14 - SLIDE 2IS246 - SPRING 2003 Today’s Agenda Feedback on Last Time Final Projects Update Research Multimedia Systems –FotoFile –Informedia –VideoMAP and VideoSpaceIcon –Discussion Questions Action Items for Next Time

3. 2003.04.14 - SLIDE 3IS246 - SPRING 2003 Today’s Agenda Feedback on Last Time Final Projects Update Research Multimedia Systems –FotoFile –Informedia –VideoMAP and VideoSpaceIcon –Discussion Questions Action Items for Next Time

4. 2003.04.14 - SLIDE 4IS246 - SPRING 2003 Today’s Agenda Feedback on Last Time Final Projects Update Research Multimedia Systems –FotoFile –Informedia –VideoMAP and VideoSpaceIcon –Discussion Questions Action Items for Next Time

5. 2003.04.14 - SLIDE 5IS246 - SPRING 2003 Final Project Ideas iCoach –A hi-fi prototype of an interactive multimedia system to assist job seekers improve their speaking skills in a job interview by streamlining the audio/video capturing, editing, audio analysis, and performance report generating processes Media Annotation in Mobile Phones –A detailed design specification of a system in which the user can input metadata information about a media element she has captured using her mobile phone camera or microphone

6. 2003.04.14 - SLIDE 6IS246 - SPRING 2003 Final Project Ideas Active Capture Authoring –A prototype to support the creation of automated directing applications by developing a set of higher- level state machine primitives to facilitate the design of Active Capture interactions Public Domain Content –A paper reviewing technology and technology policies that contribute to broadening the content in the public domain and facilitating access to public domain content

7. 2003.04.14 - SLIDE 7IS246 - SPRING 2003 Final Project Ideas Picture Book Storytelling over Video Phone –A prototype of a consumer application which incorporates both live video and automatic editing of the live interaction for on-demand viewing that structures the interaction around a “picture book” activity, in which an adult will read the child a story, accompanied by pictures Interactive Multimedia Maps –An interactive low-fi prototype of a software application that gives users the ability to retrieve, use, produce, and distribute multimedia files using GPS mapping technology and an online multimedia database

8. 2003.04.14 - SLIDE 8IS246 - SPRING 2003 Today’s Agenda Feedback on Last Time Final Projects Update Research Multimedia Systems –FotoFile –Informedia –VideoMAP and VideoSpaceIcon –Discussion Questions Action Items for Next Time

9. 2003.04.14 - SLIDE 9IS246 - SPRING 2003 Multimedia Research: ACM MM CFP The technical program will consist of plenary sessions and talks with topics of interest in three main tracks: –Content Processing Multimedia analysis, processing, and retrieval, including multimedia content description and indexing, multimedia semantics modeling, audio/video/multi-modal processing, multimedia digital rights management, content-based retrieval, MPEG-7/-21 standards. –Systems Multimedia networking and system support: including peer-to-peer streaming, audio/video streaming, multimedia content distribution, wireless multimedia, energy-aware streaming, Internet protocols, multimedia servers, operating systems, middleware and QoS. –Applications Multimedia tools, end-systems, and applications: including hypermedia systems, user interfaces, authoring, multi-modal interaction and integration, multimedia in education, entertainment, and virtual environments. We specifically solicit submissions in new and emerging areas in the field.

10. 2003.04.14 - SLIDE 10IS246 - SPRING 2003 Today’s Agenda Feedback on Last Time Final Projects Update Research Multimedia Systems –FotoFile –Informedia –VideoMAP and VideoSpaceIcon –Discussion Questions Action Items for Next Time

11. 2003.04.14 - SLIDE 11IS246 - SPRING 2003 Building a Multimedia Album in FotoFile

12. 2003.04.14 - SLIDE 12IS246 - SPRING 2003 “Scraplets” in the Story-Editing Window

13. 2003.04.14 - SLIDE 13IS246 - SPRING 2003 Face Recognition and Annotation

14. 2003.04.14 - SLIDE 14IS246 - SPRING 2003 Automatic Video Album Generation

15. 2003.04.14 - SLIDE 15IS246 - SPRING 2003 Hyberbolic Tree for Content Index

16. 2003.04.14 - SLIDE 16IS246 - SPRING 2003 Today’s Agenda Feedback on Last Time Final Projects Update Research Multimedia Systems –FotoFile –Informedia –VideoMAP and VideoSpaceIcon –Discussion Questions Action Items for Next Time

17. 2003.04.14 - SLIDE 17IS246 - SPRING 2003 Informedia Project (CMU) http://www.informedia.cs.cmu.edu/ Informedia-I: Integrated speech, image & language understanding for creating digital video libraries Informedia-II: Video information summarization & visualization

18. 2003.04.14 - SLIDE 18IS246 - SPRING 2003 Content-Based Thumbnails

19. 2003.04.14 - SLIDE 19IS246 - SPRING 2003 Filmstrip and Video Playback Windows

20. 2003.04.14 - SLIDE 20IS246 - SPRING 2003 Today’s Agenda Feedback on Last Time Final Projects Update Research Multimedia Systems –FotoFile –Informedia –VideoMAP and VideoSpaceIcon –Discussion Questions Action Items for Next Time

21. 2003.04.14 - SLIDE 21IS246 - SPRING 2003 VideoMAP

22. 2003.04.14 - SLIDE 22IS246 - SPRING 2003 VideoSpaceIcon

23. 2003.04.14 - SLIDE 23IS246 - SPRING 2003 Views of VideoSpaceIcon

24. 2003.04.14 - SLIDE 24IS246 - SPRING 2003 Construction of VideoSpaceIcon

25. 2003.04.14 - SLIDE 25IS246 - SPRING 2003 Making a Moving VideoSpaceIcon

26. 2003.04.14 - SLIDE 26IS246 - SPRING 2003 Today’s Agenda Feedback on Last Time Final Projects Update Research Multimedia Systems –FotoFile –Informedia –VideoMAP and VideoSpaceIcon –Discussion Questions Action Items for Next Time

27. 2003.04.14 - SLIDE 27IS246 - SPRING 2003 Discussion Questions Catherine Lai on “FotoFile” –FotoFile incorporates some automatic image analysis to help with annotations, for example, face recognition. But can we and how can we take more advantages of the context to determine appropriate situation for annotation? For example, if one picture is about a karaoke contest, there is a good chance that subsequent pictures taken within a three-hour span and close to the same location will describe the same event. –How is the use of narrative structure to help organize content so much different from annotating content manually? Does annotating contents mean to do data entry of predefined metadata like date and location as opposed to entering captions?

28. 2003.04.14 - SLIDE 28IS246 - SPRING 2003 Discussion Questions David Warthen on “FotoFile” –In the short section on the “Use of Narrative Structure” the authors assert they can use the “narrative structure… as a source of their [photos] organization and annotation”, thereby turning annotation into a “more enjoyable” activity. However, they do not really explore this further. How might the system exploit this approach to capturing annotation? (e.g. from an actual narrative presentation) How might the system facilitate one of the envisioned usages, where “the storyteller shares groupings of photos and annotations over the internet”?

29. 2003.04.14 - SLIDE 29IS246 - SPRING 2003 Discussion Questions David Warthen on “FotoFile” –The authors pay a great deal of attention to annotating personal photos, which they note requires significant effort. However, they do not attempt any cost/benefit analysis to see whether this effort pays off with reuse, or potential reuse, of the photos. If one considers the task from this point of view: How would one measure the ROI of investing in annotation? Might a “80/20” rule apply where it is only beneficial to invest in annotating the (loosely) 20% of pictures accessed 80% of the time? Might this imply introducing workflow annotation opportunities to the user upon first attempted re-use rather than upon entry in the system (beyond the basic “trip to Greece” or “Sam’s baby pictures” type of annotation)? Or, alternatively, would this present an insurmountable barrier to accessing the pictures for re-use at all?

30. 2003.04.14 - SLIDE 30IS246 - SPRING 2003 Discussion Questions David Warthen on “FotoFile” –The authors have organized FotoFile to use scrapbook/photo album metaphors (enhanced with keyword retrieval and browsing). These metaphors have the benefit that consumers are familiar and comfortable with them, and root the use of FotoFile in the context of existing human activities. However, such metaphors can also be limiting. Let’s explore alternatives, including: Why separate limit FotoFile to pictures, video, and audio, rather than all data on the computer? These might include written documents, e-mail, contact lists, etc. Why limit FotoFile to the personal pictures on the computer, as opposed to tying them into online resources as well (encyclopedias, picture searches and picture archives, etc.)? What alternatives to scrapbook narrative presentation might be compelling? Perhaps a hyperlinked approach? Or a free-form associative approach?

31. 2003.04.14 - SLIDE 31IS246 - SPRING 2003 Discussion Questions Lily Chen on “VideoMAP and VideoSpaceIcon” –Since the current version of VideoSpaceIcon cannot be simply applied when the camera operation is tracking or dollying, in which the camera position changes, I am very curious at how the "3D space model" the author defined for this approach will be constructed--how will they represent this 3D icon?

32. 2003.04.14 - SLIDE 32IS246 - SPRING 2003 Discussion Questions Lisa Wang on “VideoMAP and VideoSpaceIcon” –How has spatial mapping of video advanced in the last ten years? –Has there been further research on three dimensional space modeling?

33. 2003.04.14 - SLIDE 33IS246 - SPRING 2003 Today’s Agenda Feedback on Last Time Final Projects Update Research Multimedia Systems –FotoFile –Informedia –VideoMAP and VideoSpaceIcon –Discussion Questions Action Items for Next Time

34. 2003.04.14 - SLIDE 34IS246 - SPRING 2003 Readings for Next Time Wednesday 04/16 Guest Lecture on Commercial Media Asset Management Systems –Bradley Horowitz, Virage Founder and Chief Technology Officer Bradley Horowitz co-founded Virage in 1995 and currently serves as Chief Technology Officer, determining and managing the technical direction of the company. As director of Virage's Advanced Technology Group, he leads research efforts that result in new technologies, products and services for Virage. Prior to joining Virage, Bradley was a PhD. candidate at the MIT Media Lab. While at the Media Lab, he worked on a number of topics related to computer vision, computer graphics and image processing. His work there resulted in a patented new technique for the recovery of structure, motion and camera parameters from video sequences. He received an MS in Media Science from MIT in 1991 under advisor Alex P. Pentland, and a BS in Computer Science from the University of Michigan in 1989. Check out Virage product materials at –http://www.virage.com