Presentation is loading. Please wait.

Presentation is loading. Please wait.

Human-Centric Multimedia Research: Research Opportunities Nuria Oliver, PhD Telefonica Research Multimedia and Data Mining & User Modeling Scientific Director.

Similar presentations


Presentation on theme: "Human-Centric Multimedia Research: Research Opportunities Nuria Oliver, PhD Telefonica Research Multimedia and Data Mining & User Modeling Scientific Director."— Presentation transcript:

1 Human-Centric Multimedia Research: Research Opportunities Nuria Oliver, PhD Telefonica Research Multimedia and Data Mining & User Modeling Scientific Director

2 Explosion of Digital Data 6-Fold Growth in Four Years Information Created, Captured and Replicated 2006 161 Exabytes 2010 988 Exabytes Source: IDC, 2007

3 Exabytes? Book Photo Movie Print Collection US Library of Congress All printed material (multimedia) All words ever spoken by human beings (5 exa)

4 Who will create all this data? User* Generated Content 692 Exabytes User* Generated Content 692 Exabytes Organizational** Touch Content 296 Exabytes *Consumers and Workers creating, capturing or replicating personal informaton **Transported, hosted managed or secured 2010 Data 988 Exabytes 563 Exabytes

5 Human-Centric Multimedia Prosumer On-the-go Access Context & Content Social Produce Search & Discovery Consume Media, tags, ratings, comments Media

6 A FEW RESEARCH CHALLENGES AND OPPORTUNITIES

7 Multi-Modality: Content + context Multi-modal approaches are needed to construct novel methodologies to fuse multi- modal content and context information Multi-modal Multimedia Content Analysis: Feature Extraction Similarity Metrics Ontology definition Indexing schemes…. Multi-modal Multimedia Content Analysis: Feature Extraction Similarity Metrics Ontology definition Indexing schemes…. Multimedia Context: Higher level knowledge generated by users (tags, comments…) User interaction data Wisdom of the crowd Multimedia Context: Higher level knowledge generated by users (tags, comments…) User interaction data Wisdom of the crowd Fusion of content and context-based features Creation of large collections of labeled training data Noise filtering by aggregation of contextual information Improved search results Fusion of content and context-based features Creation of large collections of labeled training data Noise filtering by aggregation of contextual information Improved search results

8 Multimedia Tagging Paris Eiffel Tower France Vacation June 2009

9 Multimedia Tagging User generated content is rarely annotated really difficult, if not impossible to later find it When annotated, it is typically done in batch, per session, not per item Tags significantly improve search results alone or combined with content-based techniques Need for novel interfaces to encourage users to annotate content – Games with a Purpose – Annotations at the time of capture More research on tag expansion and automatic tagging

10 Multimedia Information Overload

11 Retrieval accuracy is not sufficient due to vast amounts of available information too many relevant results New orthogonal dimensions need to be used to extend the notion of relevance and improve retrieval performance e.g., aesthetics User generated content is of varying quality Need for user centric approaches Multi-disciplinary approaches: – Computer scientists, psychologists, human-computer interaction researchers – Computer Vision, Pattern Recognition, Machine Learning, Human Perception, Human Activity Recognition

12 Video and Audio Feature extraction Signature Creation Duplicate Detection Example: Near-Duplicate Videos How do users perceive near- duplicate videos? Do they care about them? Which features are important when defining near-duplicates? M. Cherubini, R. de Oliveira, and N. Oliver, Understanding near-duplicate videos, in Proceedings of ACM MM09, (Beijing, China), pp. 35–44, ACM Press, October 19-24 2009.

13 Example: Multimedia Aesthetics The interest that a photograph, video or audio piece generates when perceived by human observers, and that incorporates both objective and subjective factors

14 The Importance of Aesthetics Paris Louvre Night

15 The Importance of Aesthetics User generated content has a wide range of quality and aesthetic value for the same content Aesthetics influence our perception of content Highly disregarded in state-of-the-art multimedia retrieval systems Need for computational models of the aesthetic value of multimedia content Need for ground truth databases on image, audio and video aesthetics Need for deeper understanding of – The role of aesthetics on user preferences and satisfaction – Universal vs personal aesthetics – Domain-dependent aesthetics – Quality vs aesthetics

16 Personalization, Recommendation and Exploratory Search

17 Future multimedia search and retrieval systems will need to take into account the users preferences, interests and task at hand in order to return relevant content Huge amounts of multimedia data – Need for recommendations rather than direct search – Automatic discovery of relevant information to the users More research should be devoted to user modeling, personalization and recommendations of multimedia content Untapped research challenge: Role that the task at hand plays in determining the optimal multimedia content to retrieve for the user

18 Multimedia Storytelling The conveying of events with words, images and sounds, often with embellishement. Stories or narratives have been shared in every culture and in every land as a means of entertainment, education, preservation of culture and in order to instill moral values.narratives entertainmentculturemoral Crucial elements of stories and storytelling include plotplot and characters, as well as the narrative point of view.charactersnarrative point of view

19 Multimedia Storytelling Despite capturing large amounts of digital multimedia content, most users rarely access the content again Sharing the multimedia content is one of the main reasons why users capture it Lack of efficient and scalable tools for browsing, finding and selecting the desired content Multimedia Storytelling: User-friendly, semi-automatic and scalable (space and time) multimedia tools that enable users to – Easily retrieve desired multimedia content – Create and share the story they want to create from their content

20 Exemplary Workflow for MM Storytelling Storytelling Algorithms Multimedia Analysis Tools Create Story Slideshow: Madrid Christmas 2009 Storytelling UI Flickr e-mail communication With story slideshow Face Detection Face Recognition Smile Detection Aesthetics Reranking Clustering Near-duplicate Detection Tag expansion Automatic Classification Identify main Actors Identify main Chapters Complement content with external content Select images based on *Story Length *Target Audience *Target Device

21 New Multimedia Experiences

22 Users are increasingly seeking new ways to experience multimedia content Research opportunities combining: – Music + Video: High-quality visual musical experiences – Video + 3D reconstruction: 3D video – Images + Context : Mobile Augmented Reality Research challenges in: – Multimedia analysis: Machine learning, pattern recognition, computer vision – User Modeling – Human-computer interaction

23 http://research.tid.es/multimedia nuriao@tid.es


Download ppt "Human-Centric Multimedia Research: Research Opportunities Nuria Oliver, PhD Telefonica Research Multimedia and Data Mining & User Modeling Scientific Director."

Similar presentations


Ads by Google