1. ECE160 Spring 2009Lecture 201 ECE160 / CMPS182 Multimedia Lecture 20: Spring 2009 Image Recognition and Retrieval

2. ECE160 Spring 2009Image Recognition2 National Research Priorities Energy Technologies Fuel efficient engines Replacement energy to fossil fuels Lighter, longer-duration batteries Bioengineering/Bioinformatics Genes  disease Disease  medicine Search with Multimedia Content Video surveillance Photo interpretation

3. ECE160 Spring 2009Image Recognition3 Multimedia Recognition Video surveillance Photo interpretation

4. ECE160 Spring 2009Image Recognition4 Wide-area Surveillance advertisement of objectvideo.com

5. ECE160 Spring 2009Image Recognition5 Surveillance Scenarios (1) Intrusion Detection (2) Passenger Screening (3) Perimeter Monitoring Z Z Use biometric facial recognition to identify individuals of interest through existing closed circuit TV surveillance Monitor and alert on tailgating, loitering, exit/closed entry, other unauthorized access Object tracking and biometric facial recognition to determine vehicles and humans exhibiting suspicious behavior (4) Unattended Baggage Copyright © 2004 Proximex Corp. Identify unattended baggage (or other objects) left for long periods of time

6. ECE160 Spring 2009Image Recognition6 Multimedia Recognition Video surveillance Photo interpretation

7. ECE160 Spring 2009Image Recognition7 How to Organize these Photos?

8. ECE160 Spring 2009Image Recognition8 Keyword-based Manual labeling is subjective, cumbersome The aliasing problem Content-based Promising for general semantics: outdoor, landscape, flowers, people, etc. Not enough for wh-queries (where, who, when, or what) Image Organization & Retrieval

9. ECE160 Spring 2009Image Recognition9 EXTENT TM = cont EXT + cont ENT Context Spatial (location) Temporal Social Others Content Perceptual features, such as color, texture, and shape Holistic features and local features

10. ECE160 Spring 2009Image Recognition10 EXTENT TM Content SpatialTemporal SocialOthers

11. ECE160 Spring 2009Image Recognition11 Augmented Images + Cameraphones with high-quality lens can record location, time, camera parameters, and voice =

12. ECE160 Spring 2009Image Recognition12 Context from Space/Time GPS or CellID data Into place names Time-based grouping Into meaningful “events” From place names and time Time of day Weather

13. ECE160 Spring 2009Image Recognition13 Example of Using Three Pieces of Information Content SpatialTemporal

14. ECE160 Spring 2009Image Recognition14 Maui Sunsets can be obtained from Space/Time

15. ECE160 Spring 2009Image Recognition15 Use content for verification

16. ECE160 Spring 2009Image Recognition16 Use content to transfer metadata

17. ECE160 Spring 2009Image Recognition17 Summarize of the example Derived from Context Derive time of the day Obtain weather Verify content Use of Content Verify context Transfer context Much more…

18. ECE160 Spring 2009Image Recognition18 Are They Similar?

19. ECE160 Spring 2009Image Recognition19 Are They Similar?

20. ECE160 Spring 2009Image Recognition20 Are They Similar? In terms of what? What is the user’s perception?

21. ECE160 Spring 2009Image Recognition21 Conveying Perception Image Databases Conveyed via Examples Use a sunset picture (or pictures) to find more sunset images Where does the perfect example come from?

22. ECE160 Spring 2009Image Recognition22 Conveying Perception Internet Searches Conveyed via Keywords

23. ECE160 Spring 2009Image Recognition23 Keyword Retrieval Pros A user-friendly paradigm Cons Annotation is a laborious process Annotation quality can be subpar Annotation can be subjective Synonyms

24. ECE160 Spring 2009Image Recognition24 Conveying Perception Image Databases Conveyed via Examples Use a sunset picture (or pictures) to find more sunset images Where does the perfect example come from?

25. ECE160 Spring 2009Image Recognition25 Are They Similar?

26. ECE160 Spring 2009Image Recognition26 Are They Similar?

27. ECE160 Spring 2009Image Recognition27 Are They Similar? In terms of what? What is the user’s perception?

28. ECE160 Spring 2009Image Recognition28 Recogintion of Content

29. ECE160 Spring 2009Image Recognition29 Recognition

30. ECE160 Spring 2009Image Recognition30 Recognition

31. ECE160 Spring 2009Image Recognition31 clouds vs. waves

32. ECE160 Spring 2009Image Recognition32 Web 1.0 vs. Web 2.0 User Content Content (Image/Video) Content Content (text) Content User

33. ECE160 Spring 2009Image Recognition33 Web 2.0 Content + Users + Interactions Collect rich, organized content Attract users & interactions To provide metadata To provide new content Improve search quality With new metadata and data Via social-network structure

34. ECE160 Spring 2009Image Recognition34 User management Photo uploading Single/multiple Upload wizard Photo uploading Building social network Social networks Event management Photo search Metadata collection Metadata fusion Image annotation Metadata collection - contextual - content Metadata fusion Annotate photos External functionalities Internal functionalities Fotofiti