Presentation is loading. Please wait.

Presentation is loading. Please wait.

Human-Centric Multimedia Research: Research Opportunities

Similar presentations

Presentation on theme: "Human-Centric Multimedia Research: Research Opportunities"— Presentation transcript:

1 Human-Centric Multimedia Research: Research Opportunities
Nuria Oliver, PhD Telefonica Research Multimedia and Data Mining & User Modeling Scientific Director

2 Explosion of Digital Data
Information Created, Captured and Replicated 6-Fold Growth in Four Years 2010 988 Exabytes 2006 161 Exabytes Source: IDC, 2007

3 Exabytes? Photo Movie Book
All words ever spoken by human beings (5 exa) Photo MEGA GIGA TERA PETA EXA ZETTA YOTTA Print Collection US Library of Congress All printed material (multimedia) Movie Book

4 Who will create all this data?
988 Exabytes 563 Exabytes User* Generated Content 692 Exabytes Organizational** Touch Content 296 Exabytes *Consumers and Workers creating, capturing or replicating personal informaton **Transported, hosted managed or secured

5 Human-Centric Multimedia
Search & Discovery Consume Media Media Produce On-the-go Access Media, tags, ratings, comments Prosumer Context & Content Social


7 Multi-Modality: Content + context
Multi-modal approaches are needed to construct novel methodologies to fuse multi-modal content and context information Multi-modal Multimedia Content Analysis: Feature Extraction Similarity Metrics Ontology definition Indexing schemes…. Multimedia Context: Higher level knowledge generated by users (tags, comments…) User interaction data Wisdom of the crowd Fusion of content and context-based features Creation of large collections of labeled training data Noise filtering by aggregation of contextual information Improved search results

8 Multimedia Tagging Paris France Vacation Eiffel Tower June 2009

9 Multimedia Tagging User generated content is rarely annotated  really difficult, if not impossible to later find it When annotated, it is typically done in batch, per session, not per item Tags significantly improve search results alone or combined with content-based techniques Need for novel interfaces to encourage users to annotate content Games with a Purpose Annotations at the time of capture More research on tag expansion and automatic tagging 70% of participants did NOT have a retrieval strategy that used TAGS or metadata

10 Multimedia Information Overload

11 Multimedia Information Overload
Retrieval accuracy is not sufficient due to vast amounts of available information  too many relevant results New orthogonal dimensions need to be used to extend the notion of relevance and improve retrieval performance  e.g., aesthetics User generated content is of varying quality Need for user centric approaches Multi-disciplinary approaches: Computer scientists, psychologists, human-computer interaction researchers Computer Vision, Pattern Recognition, Machine Learning, Human Perception, Human Activity Recognition

12 Example: Near-Duplicate Videos
Video and Audio Feature extraction Signature Creation Duplicate Detection How do users perceive near-duplicate videos? Do they care about them? Which features are important when defining near-duplicates? M. Cherubini, R. de Oliveira, and N. Oliver, “Understanding near-duplicate videos,” in Proceedings of ACM MM’09, (Beijing, China), pp. 35–44, ACM Press, October

13 Example: Multimedia Aesthetics
“The interest that a photograph, video or audio piece generates when perceived by human observers, and that incorporates both objective and subjective factors”

14 The Importance of Aesthetics
“Paris Louvre Night”

15 The Importance of Aesthetics
User generated content has a wide range of quality and aesthetic value for the same content Aesthetics influence our perception of content Highly disregarded in state-of-the-art multimedia retrieval systems Need for computational models of the aesthetic value of multimedia content Need for ground truth databases on image, audio and video aesthetics Need for deeper understanding of The role of aesthetics on user preferences and satisfaction Universal vs personal aesthetics Domain-dependent aesthetics Quality vs aesthetics

16 Personalization, Recommendation and Exploratory Search

17 Personalization, Recommendation and Exploratory Search
Future multimedia search and retrieval systems will need to take into account the user’s preferences, interests and task at hand in order to return relevant content Huge amounts of multimedia data Need for recommendations rather than direct search Automatic discovery of relevant information to the users More research should be devoted to user modeling, personalization and recommendations of multimedia content Untapped research challenge: Role that the task at hand plays in determining the optimal multimedia content to retrieve for the user

18 Multimedia Storytelling
“The conveying of events with words, images and sounds, often with embellishement. “ Stories or narratives have been shared in every culture and in every land as a means of entertainment, education, preservation of culture and in order to instill moral values. Crucial elements of stories and storytelling include plot and characters, as well as the narrative point of view.

19 Multimedia Storytelling
Despite capturing large amounts of digital multimedia content, most users rarely access the content again Sharing the multimedia content is one of the main reasons why users capture it Lack of efficient and scalable tools for browsing, finding and selecting the desired content Multimedia Storytelling: User-friendly, semi-automatic and scalable (space and time) multimedia tools that enable users to Easily retrieve desired multimedia content Create and share the story they want to create from their content

20 Exemplary Workflow for MM Storytelling
Face Detection Face Recognition Smile Detection Aesthetics Reranking Clustering Near-duplicate Detection Tag expansion Automatic Classification Multimedia Analysis Tools User Authentication User id’s saved on Image Management Platform database User interaction: semi-automatic approach Final Story Slideshow is an XML file with URL references to flickr images Create Story Slideshow: “Madrid Christmas 2009” Identify main Actors Identify main Chapters Complement content with external content Select images based on *Story Length *Target Audience *Target Device Storytelling UI Storytelling Algorithms Flickr communication With story slideshow

21 New Multimedia Experiences

22 New Multimedia Experiences
Users are increasingly seeking new ways to experience multimedia content Research opportunities combining: Music + Video: High-quality visual musical experiences Video + 3D reconstruction: 3D video Images + Context : Mobile Augmented Reality Research challenges in: Multimedia analysis: Machine learning, pattern recognition, computer vision User Modeling Human-computer interaction


Download ppt "Human-Centric Multimedia Research: Research Opportunities"

Similar presentations

Ads by Google