Presentation is loading. Please wait.

Presentation is loading. Please wait.

© Arjen P. de Vries 2001 1 Arjen P. de Vries Fascinating Relationships between Media and Text.

Similar presentations


Presentation on theme: "© Arjen P. de Vries 2001 1 Arjen P. de Vries Fascinating Relationships between Media and Text."— Presentation transcript:

1 © Arjen P. de Vries Arjen P. de Vries Fascinating Relationships between Media and Text

2 © Arjen P. de Vries From Video to a List of Images… …

3 © Arjen P. de Vries Query image

4 © Arjen P. de Vries Feature Space Indexing N-dimensional space Feature extraction Ranking Display

5 © Arjen P. de Vries Complicating Factors What are Good Feature Models? What are Good Ranking Functions? Subjective Queries

6 © Arjen P. de Vries Mirror Approach (1/2) Gather User’s Knowledge –Introduce semi-automatic processes for selection and combination of feature models Local Information –Relevance feedback from a user Global Information –Thesauri constructed from all users

7 © Arjen P. de Vries Mirror Approach (2/2) N-dimensional space Feature extraction ClusteringRanking Concepts Display Thesauri

8 © Arjen P. de Vries Visual Thesaurus Correct cluster representing ‘Tree’, ‘Forest’ Glcm_47 ‘Incoherent’ cluster Fractal_23 Mis-labeled cluster Gabor_20

9 © Arjen P. de Vries Mirror Results Thesaurus Only After Feedback

10 © Arjen P. de Vries ‘Explaining’ the Results Paivio’s dual coding theory conjectures that the human brain processes textual terms (logogens) as well as image terms (imagens) Term frequency and inverse document frequency intuitively express the amount of surprise of observing the event represented by the term Also matches similar music: grunge, house, … Even works for predicting avalanches!

11 © Arjen P. de Vries Query Articulation Problem N-dimensional space Feature extraction How to articulate the query?

12 © Arjen P. de Vries What is the query semantics ?

13 © Arjen P. de Vries More than Sub-Image Search Irrelevant Background Relevant Colors Distinguishing Shapes …

14 © Arjen P. de Vries Irrelevant Background QueryResult

15 © Arjen P. de Vries Known Item

16 © Arjen P. de Vries Query

17 © Arjen P. de Vries Results …

18 © Arjen P. de Vries Query

19 © Arjen P. de Vries Results …

20 © Arjen P. de Vries Problem Video TREC query images are quite different from the (low-quality) video data in the collection The current translation model from collection objects to query objects is incorrect Figure out better ways, possibly using: –Color thesaurus, transitions between colors –Explicit background and foreground models –…

21 © Arjen P. de Vries Summarizing Don’t give up… But… Stay Realistic!


Download ppt "© Arjen P. de Vries 2001 1 Arjen P. de Vries Fascinating Relationships between Media and Text."

Similar presentations


Ads by Google