1. Human-Centric Multimedia Research: Research Opportunities
Nuria Oliver, PhD
Telefonica Research
Multimedia and Data Mining & User Modeling
Scientific Director

2. Explosion of Digital Data
Information Created, Captured and Replicated
6-Fold Growth in Four Years
2010
988 Exabytes
2006
161 Exabytes
Source: IDC, 2007

3. Exabytes? Photo Movie Book
All words ever spoken by human beings (5 exa)
Photo
MEGA
GIGA
TERA
PETA
EXA
ZETTA
YOTTA
Print Collection
US Library of Congress
All printed material (multimedia)
Movie
Book

4. Who will create all this data?
988 Exabytes
563 Exabytes
User* Generated Content
692 Exabytes
Organizational**
Touch
Content
296 Exabytes
*Consumers and Workers
creating, capturing or replicating
personal informaton
**Transported, hosted
managed or secured

5. Human-Centric Multimedia
Search & Discovery
Consume
Media
Media
Produce
On-the-go
Access
Media, tags,
ratings, comments
Prosumer
Context &
Content
Social

6. A FEW RESEARCH challenges and OPPORTUNITiES

7. Multi-Modality: Content + context
Multi-modal approaches are needed to construct novel methodologies to fuse multi-modal content and context information
Multi-modal Multimedia Content Analysis:
Feature Extraction
Similarity Metrics
Ontology definition
Indexing schemes….
Multimedia Context:
Higher level knowledge generated by users
(tags, comments…)
User interaction data
Wisdom of the crowd
Fusion of content and context-based features
Creation of large collections of labeled training data
Noise filtering by aggregation of contextual information
Improved search results

8. Multimedia Tagging
Paris
France
Vacation
Eiffel Tower
June 2009

9. Multimedia Tagging
User generated content is rarely annotated  really difficult, if not impossible to later find it
When annotated, it is typically done in batch, per session, not per item
Tags significantly improve search results alone or combined with content-based techniques
Need for novel interfaces to encourage users to annotate content
Games with a Purpose
Annotations at the time of capture
More research on tag expansion and automatic tagging
70% of participants did NOT have a retrieval strategy that used TAGS or metadata

10. Multimedia Information Overload

11. Multimedia Information Overload
Retrieval accuracy is not sufficient due to vast amounts of available information  too many relevant results
New orthogonal dimensions need to be used to extend the notion of relevance and improve retrieval performance  e.g., aesthetics
User generated content is of varying quality
Need for user centric approaches
Multi-disciplinary approaches:
Computer scientists, psychologists, human-computer interaction researchers
Computer Vision, Pattern Recognition, Machine Learning, Human Perception, Human Activity Recognition

12. Example: Near-Duplicate Videos
Video and Audio Feature extraction
Signature Creation
Duplicate Detection
How do users perceive near-duplicate videos?
Do they care about them?
Which features are important when defining near-duplicates?
M. Cherubini, R. de Oliveira, and N. Oliver, “Understanding near-duplicate videos,” in Proceedings of ACM MM’09, (Beijing, China), pp. 35–44, ACM Press, October

13. Example: Multimedia Aesthetics
“The interest that a photograph, video or audio piece generates when perceived by human observers, and that incorporates both objective and subjective factors”

14. The Importance of Aesthetics
“Paris Louvre Night”

15. The Importance of Aesthetics
User generated content has a wide range of quality and aesthetic value for the same content
Aesthetics influence our perception of content
Highly disregarded in state-of-the-art multimedia retrieval systems
Need for computational models of the aesthetic value of multimedia content
Need for ground truth databases on image, audio and video aesthetics
Need for deeper understanding of
The role of aesthetics on user preferences and satisfaction
Universal vs personal aesthetics
Domain-dependent aesthetics
Quality vs aesthetics

16. Personalization, Recommendation and Exploratory Search

17. Personalization, Recommendation and Exploratory Search
Future multimedia search and retrieval systems will need to take into account the user’s preferences, interests and task at hand in order to return relevant content
Huge amounts of multimedia data
Need for recommendations rather than direct search
Automatic discovery of relevant information to the users
More research should be devoted to user modeling, personalization and recommendations of multimedia content
Untapped research challenge: Role that the task at hand plays in determining the optimal multimedia content to retrieve for the user

18. Multimedia Storytelling
“The conveying of events with words, images and sounds, often with embellishement. “
Stories or narratives have been shared in every culture and in every land as a means of entertainment, education, preservation of culture and in order to instill moral values.
Crucial elements of stories and storytelling include
plot and characters, as well as the narrative point of view.

19. Multimedia Storytelling
Despite capturing large amounts of digital multimedia content, most users rarely access the content again
Sharing the multimedia content is one of the main reasons why users capture it
Lack of efficient and scalable tools for browsing, finding and selecting the desired content
Multimedia Storytelling: User-friendly, semi-automatic and scalable (space and time) multimedia tools that enable users to
Easily retrieve desired multimedia content
Create and share the story they want to create from their content

20. Exemplary Workflow for MM Storytelling
Face Detection
Face Recognition
Smile Detection
Aesthetics Reranking
Clustering
Near-duplicate Detection
Tag expansion
Automatic Classification
Multimedia Analysis Tools
User Authentication
User id’s saved on Image Management Platform database
User interaction: semi-automatic approach
Final Story Slideshow is an XML file with URL references to flickr images
Create Story Slideshow:
“Madrid Christmas 2009”
Identify main Actors
Identify main Chapters
Complement content
with external content
Select images based on
*Story Length
*Target Audience
*Target Device
Storytelling UI
Storytelling
Algorithms
Flickr communication
With story slideshow

21. New Multimedia Experiences

22. New Multimedia Experiences
Users are increasingly seeking new ways to experience multimedia content
Research opportunities combining:
Music + Video: High-quality visual musical experiences
Video + 3D reconstruction: 3D video
Images + Context : Mobile Augmented Reality
Research challenges in:
Multimedia analysis: Machine learning, pattern recognition, computer vision
User Modeling
Human-computer interaction

23. THANK YOU!
QUESTIONS?