1. Contextual Augmentation of Ontology for Recognizing Sub-Events Setareh Rafatirad and Ramesh Jain srafatir@ics.uci.edu, jain@ics.uci.edu ICSC2011, Sep 19-21, Palo Alto Donald Bren School of Information and Computer Science University of California Irvine 1

2. People query their personal photos very frequently. 2 University of California Irvine

3. Visual Data Human Semantics User Queries are High-Level 3 University of California Irvine

4. … add more “semantic” tags SubEvent: Visiting Ghost Town During Trip to Arizona 4 University of California Irvine People are interested in events and SubEvents! Subevents as recall cues Very important in retrieval What is the current trend?

5. Problem Given photos with EXIF metadata for an event E, we partition them into its subevents. 5 University of California Irvine

6. But photos contain limited data Common meta-data include: ◦ Time ◦ Camera Params (exposure time, aperture, focal length, ISO..) ◦ More recently, GPS Tags. How to tackle this limitation? ◦ External Data Sources  Sensory Data  Web 2.0  Ontologies 6 University of California Irvine

7. Visual Data Human Semantics high relevance to smart- phone applications. Contextual model visual content features only when required Multi-modal context 7 University of California Irvine

8. Outline Introduction Related Work Problem Formulation System Overview Experimental Results (the fun part ) Summary 8 University of California Irvine

9. From Conceptual Model to Contextual Model Conceptual Model Multi-layered abstract domain model Flexibility for multiple Classifications Avoid replication Leads to complex processing Mereological,Relative/Absolute temporal, spatial Relations Contextual Model Visual Semantics Contextual Semantic e.g. time and location of events 9 University of California Irvine Hindu

10. Inspired by Binford Hierarchical Object Model 10 University of California Irvine

11. R-Ontology TRIP scenario Individual: Types: Facts:, Individual: Types: Facts: “hotel”, {domain} 21.13, {R-Onto} 79.06 {R-Onto} Individual: Types: Facts:, Individual: Types: Facts: 39.908094, 116.444083 11 University of California Irvine

12. Related Work Object-based systems ◦ Describing geometric objects [Kokarand et.al.] ◦ Pixel-based ontology ◦ Description rather than Recognition!  COMM,ABC Event-based systems ◦ Only ontology creation e.g. F-Model,E-Model [Schertp et.al, Westerman et.al.] ◦ Activity recognition 12 University of California Irvine

13. Problem Formulation Given photos P : for an event E, we partition them into its subevents. R-Ontology ◦ How can O r be employed for partitioning P? 13 I: set of instances of the classes in domain ontology CI v : context R: set of relationships between the instances University of California Irvine

14. Ontology Store(upper and domain) R-Ontology (-ies) Ontology instance Modifier Modification Particular Context Ontology Augmentation Input Context Instantiation Content Descriptor Extractor Imaging Feature Extractor Metadata Extractor (time, coordinate, camera parameters) Metadata Extractor (time, coordinate, camera parameters) Feature Extraction........................ Agglomerative SpatioTemporal Clustering WWW P: Media Photos Filtering (Content and context features) Filtering (Content and context features) 14 Request System Overview University of California Irvine

15. My friend’s Indian Wedding, [02-12-09 02-13-09], Nagpur-India 15 University of California Irvine

16. Taking portrait Wedding party My friend’s marriage Groom arrival Indian Ceremony My friend’s Indian Wedding, [02-12-09 02-13-09], Nagpur-India 16 University of California Irvine

17. Visiting forbidden city Having Dinner Ordering Dinner Serving Dinner My Trip, [07-06-10 07-12-10], Beijing-China Shopping at twins mall 17 University of California Irvine

18. Summary Ontologies for Trip,Wedding Sub-Events as important Tags Future work ◦ Extend the employment of context and content features of photos ◦ Sensors on smart phones 18 University of California Irvine

19. Q & A And Thanks for listening. Contact Information: ◦ srafatir@ics.uci.edu ◦ jain@ics.uci.edu 19 University of California Irvine

20. Back up Slides 20 University of California Irvine

21. Hindu Wedding Variable Context Constant Context Web and other sources 21 {srafatir,jain}@ics.uci.edu

22. Vacation-Trip Constant Context Variable Context Web and other sources Vacation Trip Process Professional Activity Eating Lunch Shopping Visiting has-subevent subClassOf has-subevent has-processingUnit has-subevent Dinn er subClassOf Conference Process- L Serving food has-subevent Ordering food before restaurant,cafe Mall,plaza,shopping center has-locationType Process- Schedule has-processingUnit hotel,university Trip to Beijing Process Professional Activity Eating Lunch Shopping Visiting has-subevent subClassOf has-subevent has-processingUnit has-subevent Dinner subClassOf ACM Confernce Process-L Serving food Ordering food before restaurant,cafe Mall,plaza,shopping center has-locationType Process- Schedule has-processingUnit hotel,university Keynote: 8-9 am Speaker:… … has-value m1 m2 started-by finished-by 3:00 pm 11:59 pm has-value Delta- dinner occurs-during Beijing restaurant occurs-at bound-1 has-boundary lower boun d upper boun d s-contains ‘116.02..’ has-longitude ‘39.02..’ has-latitude ‘116.01..’ has-longitude ‘39.01..’ has-latitude 22 {srafatir,jain}@ics.uci.edu

23. Agglomerative Clustering 23 {srafatir,jain}@ics.uci.edu

24. Implementation and Results TRIP scenario Individual: Types: Facts:, Individual: Types: Facts: “hotel”, {domain} 21.13, {R-Onto} 79.06 {R-Onto} Individual: Types: Facts:, Individual: Types: Facts: 39.908094, 116.444083 WEDDING scenario Individual: Types: Facts:, <…. ….> “outdoor”, … Individual: Types: Facts:, <…. ….>, “http://www.unitX.com”, … Individual: Types: Facts: “smiling” 24 {srafatir,jain}@ics.uci.edu

25. Upper/Domain Ontology Basic Derivation of E* by A. Gupta, R. Jain. Context of event classes in domain ontology Temporal model (absolute and relative) Spatial model ◦ Coordinate(lat,lng), boundingbox(a pair of coordinates), place-name (e.g. Disney Land), locationType ◦ (Perdurant occurs-at Place),(Place has-boundary BoundingBox), (BoundingBox s-contains Coordinates). Structural model (subevent) ◦ entailment rule 25 {srafatir,jain}@ics.uci.edu

26. Implementation and Results Vacation TripIndian Wedding Organize my photos based on the subevents I participated during event- x Organize my photos based on the subevents I participated during event-y photos of my stay at “blah” hotelphotos of wedding party photos of my visits to famous landmarks photos of groom arriving photos of shoppingphotos of serving dinner photos of having lunch/dinner/breakfast photos of marriage ceremony 26 {srafatir,jain}@ics.uci.edu Sources from Web: ◦ For Trip { Landmark finder, Trip itinerary,location database} ◦ For Wedding { Electronic invitations,face-detector,location database} OWL-API EXIFTOOL Lab tools Personal Archives

27. Ontology Augmentation 27 {srafatir,jain}@ics.uci.edu

28. A Qualitative User Study 28 {srafatir,jain}@ics.uci.edu

29. Implementation and Results 29 {srafatir,jain}@ics.uci.edu Sources from Web: ◦ For Trip { Landmark finder, Trip itinerary,location database} ◦ For Wedding { Electronic invitations,face-detector,location database} OWL-API EXIFTOOL Lab tools Personal Archives