1. Steffen Staab ISWeb – Informationssysteme & Semantic Web Semantic Multimedia Steffen Staab, Univ. Sheffield March 28, 2006

2. Steffen Staab ISWeb – Informationssysteme & Semantic Web My private challenge ….and more than 17,000 other images and mini-movies

3. Steffen Staab ISWeb – Informationssysteme & Semantic Web What I would like to do… vs What I can do Send family recent christmas photos… Send friends pictures that include them… Ask for pictures that depict my children at carnival with a big smile in order to make a presentation about semantic multimedia… Show a friend where we live… Exchange opinions about what are the best shots Record „photo copies“ of signed contracts Query for all architecture images built for X-Media proposal… List to be continued… Store pictures in a folder Query for picture name and date The Semantic Gap

4. Steffen Staab ISWeb – Informationssysteme & Semantic Web Strategies for Narrowing the Semantic Gap Image understanding –Scene classification –People recognition (that, who) –Artifact recognition Context understanding Shared Annotation User Feedback Highly domain- dependent, but so far: little domain knowledge Google Flickr Little explored wrt Semantics (e.g.Santini etal.)

5. Steffen Staab ISWeb – Informationssysteme & Semantic Web Semantic Multimedia For multimedia resources the semantic gap between having the primary data (i.e. audio files, video encoding) and understanding the content is very large, much larger than for text documents. Therefore the promise of the Semantic Web to improve access to such kind of resources is even more tempting than for textual resources. The bottleneck of understanding their semantics is however very large, too. In this talk I will sketch the Koblenz strategy of capturing semantic knowledge for multimedia resources. Our approach is layered into: 1. Understanding image content using ontologies 2. Understanding multimedia using the context in which it appears 3. Sharing image content allowing for shared annotation 4. Inferring semantics from retrieval tasks In particular, I will describe our approach for layers 1 and 3.

6. Steffen Staab ISWeb – Informationssysteme & Semantic Web Simple Annotations - TagFS

7. Steffen Staab ISWeb – Informationssysteme & Semantic Web Motivation Annotation is expensive as a dedicated process  Allow for on-the-fly annotation from all existing applications Semantically Enabled App Semantic Metadata Repository Arbitrary Application Arbitrary Metadata Format

8. Steffen Staab ISWeb – Informationssysteme & Semantic Web Motivation Retrieval strategies cannot be easily anticipated by users  No fixed schema

9. Steffen Staab ISWeb – Informationssysteme & Semantic Web Concept Allow for on-the-fly annotation from all existing applications  Annotation through a filesystem interface No fixed schema  Avoid hierarchical filesystem organisation

10. Steffen Staab ISWeb – Informationssysteme & Semantic Web Use the Filesytem to Link Arbitrary Applications with a Metadata Repository Semantically Enabled App Semantic Metadata Repository Arbitrary Application OS Kernel FAT32 WebDAV Virtual Filesystem ?

11. Steffen Staab ISWeb – Informationssysteme & Semantic Web Directories represent… Queries filesystem paths correspond to queries each directory translates to a parametrized view on the metadata repository views can be nested Example –“/tag beach/depicts steffen“ translates into –depicts(„steffen“, tag(„beach“, /)) –“/” represents metadata repository Metadata Every file is associated with tags given by names of super directories Allowing for linking of directories Multiple hierarchies at creation timeat exploration time

12. Steffen Staab ISWeb – Informationssysteme & Semantic Web Annotated Information Objects listing a directory (= view) returns all information objects corresponding to the view information objects may be files, bookmarks, chapters,... Class handlers implement file system operations (read, write,...) for a class of information objects (files, bookmarks,…) RDFFS maps filesystem operations to operations on a RDF-repository Views and Class Handlers provide tagging for files

13. Steffen Staab ISWeb – Informationssysteme & Semantic Web TagFS Architecture II Semantically Enabled App Semantic Metadata Repository Arbitrary Application OS Kernel FAT32 WebDAV Virtual Filesystem Fuse RDFFs Views Class Handlers Arbitrary Application

14. Steffen Staab ISWeb – Informationssysteme & Semantic Web Usage Examples: Annotate –Take Picture of Steffen and Simon in Sheffield and store it at /tag image/tag sheffield/depicts simon/picture1 –Take Picture of Steffen in Rome and store it at /tag rome/depicts steffen/tag image/picture2 –Record video of Sheffield and store it at /tag video/tag sheffield/video1 –notice that picture1 also depicts Steffen. link picture1 to /depicts Steffen

15. Steffen Staab ISWeb – Informationssysteme & Semantic Web Usage Examples: Retrieve –all documents related to Sheffield: /tag sheffield picture1 video1 –all images depicting Steffen: /tag image/depicts steffen picture1 picture2

16. Steffen Staab ISWeb – Informationssysteme & Semantic Web Shared Annotations: SEA – Semantic Exchange Architecture

17. Steffen Staab ISWeb – Informationssysteme & Semantic Web Use Case: Virtual Organizations Project members need to share, i.e. distribute and retrieve, confidential information among each other Different members have different roles, e.g. manager, researcher, that require different views onto the shared data X-Media ContractsDeliverables X-Media Work Package 1Work Package 2 Lucy@SheffieldSergej@Koblenz

18. Steffen Staab ISWeb – Informationssysteme & Semantic Web Use Case: Image Sharing User X has many images –X wants to share some images publicly, some only with dedicated persons, and some not at all –Due to the amount of images, uploading many images to a central repository is not an option

19. Steffen Staab ISWeb – Informationssysteme & Semantic Web Use Case: Image Sharing Neither X nor X's friends want to pay for a dedicated server or hand over their images to a server managed by a 3 rd party They would like to user their own storage

20. Steffen Staab ISWeb – Informationssysteme & Semantic Web SEA Purpose: Decentralized information sharing, e.g. image sharing Tagging as means for –Personal and collaborative organization of information –Information retrieval –Access control

21. Steffen Staab ISWeb – Informationssysteme & Semantic Web SEA: Architecture RDF store for meta data DHT implementation for efficient distribution of shared information

22. Steffen Staab ISWeb – Informationssysteme & Semantic Web SEA: Features Autonomy for information distribution and sharing Flexible information organization Simple setup and administration of sharing environment Privacy, data security Ad-hoc collaboration

23. Steffen Staab ISWeb – Informationssysteme & Semantic Web Centralized vs. Distributed Sharing with SEA Conventional information sharing characterized by centralization SEA follows a distributed approach

24. Steffen Staab ISWeb – Informationssysteme & Semantic Web Centralized vs. Decentralized (SEA) Centralized –requires dedicated hardware, setup, and administration of a central sharing platform, e.g. document management system –assumes to upload information to a store not controlled by oneself Decentralized (SEA) –hardware, operating already there, administration delegated to the user –requires installations on each peer –information provided by a person is stored on the person's computer

25. Steffen Staab ISWeb – Informationssysteme & Semantic Web desktop integration often not given or requires additional client installation integrates with personal desktop –allows offline utilization –organize once, both for personal use and sharing –changes are immediately effective Centralized vs. Decentralized (SEA)

26. Steffen Staab ISWeb – Informationssysteme & Semantic Web SEA provides... Ad-hoc setup of a sharing environment –only a client installation is required, no additional hardware is needed –Users share their information publicly, or –add each other to their „buddy list“ (list of known users) to share some data only with dedicated buddies

27. Steffen Staab ISWeb – Informationssysteme & Semantic Web Access Control in SEA Access control mechanisms allow to define with whom to share data –based on taggings e.g. everything tagged as „public“ is public e.g. everything tagged as „forSteffen“ is accessible for Steffen –based on rules for access

28. Steffen Staab ISWeb – Informationssysteme & Semantic Web SEA: Data Model Ontological meta model

29. Steffen Staab ISWeb – Informationssysteme & Semantic Web Image understanding using ontologies

30. Steffen Staab ISWeb – Informationssysteme & Semantic Web What is this?

31. Steffen Staab ISWeb – Informationssysteme & Semantic Web Solution Better use context and background knowledge

32. Steffen Staab ISWeb – Informationssysteme & Semantic Web Region-Based Image Labelling 1.Find semantically meaningful regions 2.Label them with concepts 3.Infer higher level annotations from initial labellings 4.Provide user-centred, semantic annotation The overall aim is to improve the access to multimedia content.

33. Steffen Staab ISWeb – Informationssysteme & Semantic Web Initial Labelling Appropriate Content Analysis modules provide initial labelling Scene classification

34. Steffen Staab ISWeb – Informationssysteme & Semantic Web Image Labeling: 1. Initial, region-based Output: Segment Classification Hypothesis set of possible labels for image segments Degree of confidence Scene classification

35. Steffen Staab ISWeb – Informationssysteme & Semantic Web Image Labeling: 1. Initial, region-based Output of Person/Face Detection: Bounding boxes for detected persons/faces Degree of confidence Scene classification

36. Steffen Staab ISWeb – Informationssysteme & Semantic Web Multi-Tier Image Model In order to relate the various initial labellings a multi-tier model is used Annotation tiers contains semantic metadata produced by a content analysis module –Segments and hypothesis sets for the Segment Classification –Bounding Boxes and detected persons/faces Segments of different tiers are related using topological relations Within tiers segments are related using spatial and topological relations

37. Steffen Staab ISWeb – Informationssysteme & Semantic Web Multi-Tier Image Model Segments Label hypotheses Confidence values Bounding boxes Classification of picture …. A1A1 A2A2 B 2 ={l 1,l 2 } A 1 over A 2 A 1 overlaps B 2 …  Spatial & topological information

38. Steffen Staab ISWeb – Informationssysteme & Semantic Web Multimedia Reasoning Aim, now: –integrate available information towards –global, –consistent and –user-oriented annotation 3 tasks: –Consistency Checking –Region Merging –Generation of a higher-level, user-centered annotation Current Focus

39. Steffen Staab ISWeb – Informationssysteme & Semantic Web Consistency Checking Constraint Satisfaction Problem (CSP) Check that label(s) of a region are consistent wrt labels of neighboring regions Ideally: –Leaves one correct label per region More often: –more than one label remains –decision in favor of highest confidence values Process consists of 1.Transformation of multi-tier description into a CSP 2.Application of constraint reasoning to solve the CSP 3.Computing the “best” labeling using the confidence values

40. Steffen Staab ISWeb – Informationssysteme & Semantic Web Constraint Satisfaction Problem (CSP) Definition Consists of set of variables and set of constraints relating several variables Each variable may have values from it’s domain A constraint defines which values can be assigned to a variable depending on the related variables Standard methods exist to solve the CSP Two steps: –Consistency checking, i.e. removal of values from the domain that never satisfy the constraints –Computation of full solutions using search algorithms (i.e. model generation)

41. Steffen Staab ISWeb – Informationssysteme & Semantic Web Constraint Satisfaction Problems Example: Variables: x, y, z Domains: D(x) = {1, 2, 3}, D(y) = {2, 3, 4}, D(z) = {2, 3, 4, 5} Constraints: x >= y, y >= z After consistency checking: D(x) = {2, 3}, D(y) ={2, 3}, D(z) = {2, 3} Concrete Solutions (models): (2, 2, 2), (3, 2, 2), … –Not all possible combinations of domain values are a solution!

42. Steffen Staab ISWeb – Informationssysteme & Semantic Web Image Labelling as a CSP Each segment s is transformed into a variable v s Initial Labellings L s are the domains of the segment variables, D(v s )=L s For each spatial relation type, a constraint sp-rel(v,w) is defined –the spatial constraints define which value combinations are legal for the given relation e.g. left-of(v,w):={(sea,sea),(sky,sky)}, but not (sky,sea) If two segments s,t are related with a spatial relation sp-rel, a corresponding spatial constraint sp-rel(v s,v t ) is instantiated

43. Steffen Staab ISWeb – Informationssysteme & Semantic Web Initial imageSegmentation Mask Example

44. Steffen Staab ISWeb – Informationssysteme & Semantic Web SeaSkySandPerson Region 1 0.05 0.03 0.071.00 Region 20.28 0.42 0.300.00 Region 30.54 0.74 0.320.00 Region 40.79 0.54 0.430.08 Region 50.00 0.80 0.030.09

45. Steffen Staab ISWeb – Informationssysteme & Semantic Web Confidence Values SeaSkySandPerson Region 1 0.05 0.03 0.071.00 Region 20.28 0.42 0.300.00 Region 30.54 0.74 0.320.00 Region 40.79 0.54 0.430.08 Region 50.00 0.80 0.030.09

46. Steffen Staab ISWeb – Informationssysteme & Semantic Web Spatial Relations Confidence Values Sky can not be left of Sea SeaSkySandPerson Region 1 0.05 0.03 0.071.00 Region 20.28 0.42 0.300.00 Region 30.54 0.74 0.320.00 Region 40.79 0.54 0.430.08 Region 50.00 0.80 0.030.09

47. Steffen Staab ISWeb – Informationssysteme & Semantic Web Spatial Relations Confidence Values Sky can not be left of Sea SeaSkySandPerson Region 1 0.05 0.03 0.071.00 Region 20.28 0.42 0.300.00 Region 30.54 0.74 0.320.00 Region 40.79 0.54 0.430.08 Region 50.00 0.80 0.030.09

48. Steffen Staab ISWeb – Informationssysteme & Semantic Web Spatial Relations Confidence Values SeaSkySandPerson Region 1 0.05 0.03 0.071.00 Region 20.28 0.42 0.300.00 Region 30.54 0.74 0.320.00 Region 40.79 0.54 0.430.08 Region 50.00 0.80 0.030.09 Sea can not be above Sky

49. Steffen Staab ISWeb – Informationssysteme & Semantic Web Spatial Relations Confidence Values Sea can not be above Sky SeaSkySandPerson Region 1 0.05 0.03 0.071.00 Region 20.28 0.42 0.300.00 Region 30.54 0.74 0.320.00 Region 40.79 0.54 0.430.08 Region 50.00 0.80 0.030.09

50. Steffen Staab ISWeb – Informationssysteme & Semantic Web Definition of Spatial Constraints Spatial constraints form an integral part of the domain knowledge used for multimedia reasoning Currently they are explicitly defined by a domain expert But: –Seems not feasible for large amounts of concepts and relations –Preferably each constraint should be accompanied by a confidence value, which can hardly be defined by an expert Idea: –Learn constraints from pre-annotated images –Allow for later refinement during run-time by user interaction. –Planned extension of M-OntoMat-Annotizer for this purpose –http://www.acemedia.org/aceMedia/results/software/m-ontomat- annotizer.htmlhttp://www.acemedia.org/aceMedia/results/software/m-ontomat- annotizer.html

51. Steffen Staab ISWeb – Informationssysteme & Semantic Web M-Ontomat-Annotizer

52. Steffen Staab ISWeb – Informationssysteme & Semantic Web Current Status: Use of segment classification Very recently: integration of person/face detection module

53. Steffen Staab ISWeb – Informationssysteme & Semantic Web Initial Evaluation An evaluation framework for region-based image labelling was defined within the aceMedia project. Ground Truth is defined on a grid-basis –a N x N grid is layered on top of each image –each cell is annotated with all depicted concepts For evaluation the segments of the segmentation, or the bounding boxes, are mapped to the respective cells. –For each concept it is counted how often the concept was found correctly, i.e. a correspondence between the segment label and a grid label is found the concept was found in general the concept exists in the GT Based on these values precision and recall for each concept, and the overall process can be defined.

54. Steffen Staab ISWeb – Informationssysteme & Semantic Web Evaluation Results Since the method is based on content analysis modules, we evaluated the improvement reached by applying the constraint reasoning to the segment classification. First, precision, recall and the F-Measure were computed for the segment classification Then, for the CSP method applied to the initial labelling Finally the average improvement was calculated

55. Steffen Staab ISWeb – Informationssysteme & Semantic Web Evaluation Results Concep t Precisio n Rec all F Sky0.770.690.73 Sea0.660.590.62 Sand0.750.940.84 Person0.330.650.44 Total0.690.750.72 Segment Classification Concep t Precisio n Rec all F Sky0.780.910.84 Sea0730.530.62 Sand0.850.970.9 Person0.380.620.47 Total0.760.820.79 Constraint Reasoning Set Up: –Evaluation with ~60 images –A 8x8 grid was used for the ground truth –The segmentation was set up to always produce 8 segments per image Results are promising, showing an 10% increase in average. However, results in the overall performance are needed

56. Steffen Staab ISWeb – Informationssysteme & Semantic Web Next Steps Soft Constraint Reasoning –Fuzzy Constraints to integrate the confidence values into the reasoning –Incremental Constraints to flexibly add constraints during reasoning –both should provide for more robust results and lead to better reduction of the initial label sets Incorporation of a region merging step –Would enable an iterative process and a knowledge-based segmentation Derivation of a higher-level annotation –Currently a simple combination of confidence values is applied the maximum degree for each concept is kept, and each concept is added to the final annotation –later also relations and additional concepts should be inferred

57. Steffen Staab ISWeb – Informationssysteme & Semantic Web Conclusion 1.Narrowing the Semantic Gap requires an Integration of Multiple Techniques 2.Some of the techniques need not be very sophisticated –e.g. tagging 3.Some sophisticated techniques may not range very far –person recognition trained for my family doesn‘t recognize Carsten 4.Different communities need to speak to each other 5.Large chances for the Semantic Web crowd!

58. Steffen Staab ISWeb – Informationssysteme & Semantic Web Bernhard Schüler Thank You! Sergej Sizov Thomas Franz Multimedia Web Services P2P & Complex Systems Simon Schenk S. Mir F. S. Parreiras B. Tausch The wonderful world of ontologies@ISWeb Klaas Dellschaft Olaf Görlitz Rabeeh Ayaz Carsten Saathoff C. Ringelstein Steffen Staab Richard Arndt