1. MediaView -- Towards a “Semantic” Multimedia Database Model
Qing Li
Dept of Computer Science
City University of Hong Kong

2. Outline Motivation & Introduction Modeling Constructs
Logical Implementation
Real-World Applications
Conclusion

3. State-of-the-art Multimedia Systems and Applications
an explosive growth in recent years
demand on managing multimedia using databases
Database techniques for multimedia
data modeling
indexing
query processing
presentation & synchronization

4. “Semantic Gap” semantics-intensive multimedia systems & applications
non-semantic
multimedia data models
Semantic Gap
require
model
raw data, primitive properties (size, format, etc)
semantic meaning of the data

5. Semantic modeling of multimedia -- Why hard?
Context-dependency
Semantics is not a static and intrinsic property
The semantics of an object often depends on:
the application/user who manipulate the object
the role that the object plays
other objects in the same “context”
Example:
Van Gogh’s paintings
flower

6. Why hard? (cont.) Modality-independency
Media objects of different modalities may suggest the similar/related semantic meanings.
Example:
Query:
Results:
Harry Potter has never been the star of a Quidditch team, scoring points while riding a broom far above the ground. He knows no spells, has never helped to hatch a dragon, and has never worn a cloak of invisibility.
image
video
text

7. MediaView – A “Semantic Bridge”
An object-oriented view mechanism that bridges the semantic gap between multimedia systems and databases
Core concept – media view (MV)
a customized context for semantic interpretation of media objects (text docs, images, video, etc)
collectively constitute the conceptual infrastructure of an multimedia system & application

8. Architecture
MediaView
Mechanism

9. Fundamentals of MediaView
Basic concepts – class vs. MV
View operators – basic functions of MV
View algebra – derivations of MV
Comparison – other “dynamic” data models

10. Basic Concepts
Definition 1: Set C as the set of base classes. A base class Ci  C has a unique class name, a type description, and a set of objects associated with it. The type of Ci is referred to as type(Ci), which defines a set of properties as the common interface of all the instances of Ci. The set of properties are referred to as properties(Ci), and each property in it can be a value of a simple type, an instance of a certain class, or a method. The set of objects associated with Ci is defined as extent(Ci)= {o | oCi}.

11. Basic Concepts
Definition 2: A media view MVi is a virtual class that has a unique view name, a type description, and a set of objects associated with it. The type of MVi is referred to as type(MVi), which defines a set of properties properties (MVi) as the common interface of all its instances. Similarly, a property can be a value of a simple type, an instance of a media view, or a method. The set of objects associated with MVi is defined as extent(MVi)= {o | oMVi}.

12. Basic Concepts So, a media view MVi can be represented as a triple:
MVi= <Mi, Pi, Ri,>
Where:
Mi - a set of objects that are included into MVi as its members. Each object o∈Mi belongs to a certain source class, and different members of MVi may belong to different source classes.
Piv - a set of properties (attributes and methods) applied on either MVi itself (Piv) or on all the members (Pim).
Ri - a set of relationships, and each r∈Ri is in the form of <oj, ok, t>, which denotes a relationship of type t between member oj and ok in MVi; Ri itself may exhibit a “graph”.

13. Basic Concepts
Definition 3: A base class Ci is defined as a subclass of another base class Cj if and only if the following two conditions hold: (1) properties(Cj)  properties(Ci), and (2) extent(Ci)  extent(Cj). If Ci is the subclass of Cj, we also say that there is an is-a relationship from Ci to Cj. A base schema (BS) is a directed acylic graph G=(V, E), where V is a finite set of vertices and E is a finite set of edges as a binary relation defined on V×V. Each element in V corresponds to a base class Ci. Each edge in the form of e=<Ci, Cj>E represents an is-a relationship from Ci to Cj (or Ci is a subclass of Cj).

14. Basic Concepts
Definition 4: A media view MVi is a subview of another media view MVj (or there is an is-a relationship from MVi to MVj) if and only if properties(MVj)  properties(MVi) and extent(MVi)  extent(MVj). A view schema (VS) is a directed acyclic graph G={V, E}, where a vertex in V corresponds to a media view MVi, and an edge e=<MVi,MVj>E represents an is-a relationship from MVi to MVj (or MVi is a subview of MVj).

15. Basic Concepts
An example…

16. Basic Concepts
Semantics-based data reorganization via media views

17. Basic Concepts
Definition 5: The semantic graph (SG) is an undirected graph G={V, E}, where V is a finite set of vertices and E is a finite set of edges. Each element ViV corresponds to a multimedia object Oi in the database. E is a ternary relation defined on V×V×N. Each e=<Vi,Vj, n>E represents a semantic link of degree n between object Oi and Oj, where n is the number of media views to which both objects belong. We define n as the correlation factor between Oi and Oj.

18. Basic Concepts
Definition 6: The correlation matrix M=[Mij] is an adjacency matrix of the semantic graph. Specifically, each element Mij contains the correlation factor between Oi and Oj, with all the diagonal elements set to be zero.

19. Basic Concepts
Semantic Graph Model

20. View Operators
A set of operators that take media views and view instances as operands.
Our intension is not to come up with a complete set of operators, but to focus on those that are indispensable in supporting queries and navigation over multimedia objects.

21. View Operators type-level V-overlap
syntax<boolean>:= v-overlap (<media view1, media view2 >)
semantics true, if and only if ( o  O)(oextent(<media view1>) and oextent(<media view2>))
Cross
syntax{<object>}:= cross (<media view1, media view2 >)
semantics{<object>} := {o  O | o  extent(<media view1>) and oextent(<media view2>)}
Sum
syntax{<object>}:= sum (<media view1, meida-view2 >)
semantics{<object>} := {o  O | o  extent(<media view1>) or oextent(<media view2>)}
Subtract
syntax{<object>}:= subtract (<media view1, media view2>)
semantics{<object>}:= {o  O | o  extent(<media view1>) and oextent(<media view2>)}

22. View Operators instance-level Class
syntax<base class> := class(<view instance>)
semantics<view instance> is a instance of <base class>
components
syntax{<object>} := components (<view instance>)
semantics {<object>} := { oO | o is a component (direct or indirect) of <view instance>}
i-overlap
syntax<boolean> := i-overlap (<view instnace1>, <view instance2>)
semantics true, if and only if ( o  O) (o  components (<view instance1>) and o  components(<view instance2>))

23. View Algebra Functions -- derivation of new MVs from existing MVs
Heuristic Enumeration
Blind enumeration
Content-based enumeration
Semantics-based enumeration

24. View Algebra
Definition 7. The n-level correlation matrix M(n) is derived from correlation matrix M by the following formula:
where n is a positive integer and k (0<k<1) is a constant between 0 and 1. Each element [M(n)]ij is defined as the n-level correlation factor between objects Oi and Oj.

25. View Algebra Algebra Operators
select from src-MV where <predicate>
project <property-list> from src-MV
intersect (src-MV1, src-MV2)
union (src-MV1, src-MV2)
difference (src-MV1, src-MV2)

26. Comparison (vs. class) media view object class membership
heterogeneous objects
uniform objects
member acquisition
dynamic inclusion/exclusion of existing objects of other classes
creating new objects
mapping
one object can belong to multiple media views
one object has exactly one class
relationship
inter-member semantic relationship
N/A

27. Comparison (vs. traditional object view)
media view
object view
membership
heterogeneous objects
uniform objects
relationship
inter-member semantic relationship
N/A
member properties
instance-level properties (user-defined)
inherited or derived properties (for view instances)
global properties
MV-level properties (user-defined)

28. Logical Implementation
MediaView Construction
MediaView Customization
MediaView Evolution

29. MediaViews Construction
Work with CBIR systems to acquire the knowledge from queries
Learn from previously performed queries
A multi-system approach to support multi-modality of media objects
Organize the semantics by following WordNet

30. Why WordNet?
Different queries may greatly vary with the liberty of choosing query keywords
We need an approach to organize those knowledge into a logic structure
A simple “context”: a concept in WordNet
Common media views: corresponds to simple contexts
We provide all common media views, based on which users can build complex ones.

31. Navigating the Multimedia Database
Navigating via semantic relationships of WordNet
Semantic Relationship Examples
Synonymy (similar) pipe, tube
Antonymy (opposite) fast, slow
Hyponymy (subordinate) tree, plant
Meronymy (part) chimney, house
Troponomy (manner) march, walk
Entailment drive, ride

32. Navigating the Multimedia Database

33. MediaViews Construction

34. “Multi-dimensional” Semantic Space
“IS-A” relationship in thesaurus
For example, “Season” has a 4-dimension semantic space [“spring”, “summer”, “autumn”, “winter”]

35. Encoding with Probabilistic Tree
A Probabilistic Tree specifies the probability of one media object semantically matching a certain concept in thesaurus.

36. Encoding with Probabilistic Tree
Procedure:
Step i: Following the thesaurus, trace from the target concept C1 to the root concept Root in thesaurus. Assume the path is: <C1, C2 …, Root = Cn>. Start from CC=Cn and initially set P=1.
Step ii: Suppose CC=Ci, and the next concept Ci-1 is one of the k sub-concepts of Ci. If CC is encoded in the Probabilistic Tree of this media object, then let
If not, we let
Step iii: If CC has not reached C1, repeat Step ii. Or, P is the probability of the media object matching concept C1.

37. Evolution through Feedback
A progressive approach
MediaView is accumulated along with the processes of user interactions
Two phases of feedback
System-feedback
User-feedback

38. Evolution through Feedback

39. Evolution through Feedback
Procedure:
Record each feedback performed by users.
For each CBIR system i involved, calculate its accuracy rate of retrieval. That is, simply divide the total number of retrieved results by the number of correct results according to user feedback.
Reset the value of to its accuracy rate respectively.
Wait for next session of user feedback.

40. Fuzzy Logic based Evolution Approach
Due to the uncertainty of the semantics, can not make an absolute assertion that a media object is relevant or irrelevant to a “context”
A media object in a database may be retrieved as a relevant result to a “context” several times: the more times a media object is retrieved, the more confidence it has to be considered as relevant to the context.

41. Fuzzy Logic based Evolution Approach
For a media object e, a context c,
- the accumulation of historial feedback information (from both system and user’s)
- the adjustment of after each feedback session

42. Inverse Propagation of Feedback
The drawback of up-down fashion of calculating the probability
E.g. Whether a media object matches “season” can not leverage from that the media object was a match of “spring”
Solution: propagate the confidence value of a media object being relevant to a concept along the hierarchical structure from bottom-up

43. Inverse Propagation of Feedback
Procedure:
Wait for a feedback session.
For each positive feedback, namely, stating a concept C is relevant to a media object. Following the thesaurus, trace from C to the root concept Root in thesaurus. Assume the path is: <C, C1, C2 …, Root = Cn>.
Append Ci as also positive feedback to that media object, where i=1 to n.

44. MediaView Customization
Two level MediaView Framework

45. MediaView Customization
Dynamically construct complex-context-based media views based on simple ones
An example complex context: “the Grand Hall in City University”
Several user-level operators are devised to support more complex/advanced contexts, besides the basic operators

46. User-level Operators
INHERIT_MV(N: mv-name, NS: set-of-mv-refs, VP: set-of-property-ref, MP: set-of-property-ref): mv-ref
UNION_MV(N: mv-name, NS: set-of-mv-refs): mv-ref
INTERSECTION_MV(N: mv-name, NS: set-of-mv-refs): mv-ref
DIFFERENCE_MV(N1: mv-ref, N2: mv-ref): mv-ref

47. Build a MediaView in Run-time
Example: find out info about "Van Gogh"
Who is "Van Gogh"?
What is his work?
Know more about his whole life.
Know more about his country.
See his famous painting "sunflower"

48. Build a MediaView in Run-time
Who is “Van Gogh”?
INHERIT_MV(“V. Gogh“, {<painter>},name=”Van Gogh” ,);
What is his work?
INTERSECTION_MV(“work”, {<painting>, vg});
Know more about his whole life.
INTERSECTION_MV(“life”, {<biography>, vg});
Know more about his country.
INTERSECTION_MV(“country”, {<country>, vg});
See his famous painting “sunflower”
Set sunflower = INTERSECTION_MV(“sunflower”, {<sunflower>, <painting>}); Set vg_sunflower = INTERSECTION_MV(“vg_sunflower”, {vg_work, sunflower});

49. Authoring Scenario Creates a new media view named after the subject
All multimedia materials used in the document would be put into this MediaView for further reference.
To collect the most relevant materials for authoring, the user performs the MediaView building process.
Import suitable media objects by browsing media views
Reference the manner and style of authoring, to find other media views with similar topics.
Drag & Drop
“learning-from-references”

50. Interface of Our Authoring System

51. System Features A Dynamic Environment
Helps a user select materials from the database to incorporate into the document
Query other similar media views for referencing the manner and/or style of authoring

52. Real-World Applications
A Multimedia Recipe Database
Modeling basis
Personalized (context-aware) manipulation
Cross-media indexing and retrieval system
Novel way of annotating and retrieving media objects
Lead to new indexing strategies

53. A Personalized Recipe Database System
People can not live without foods
Existing recipe websites provide huge amounts of recipes throughout the world
Fail to give support on analyzing and comparing recipes (What are important cooking principles & skills; what makes two dishes’ taste so different, etc.)
Unable to help users find similar recipes in a comprehensive manner (only keyword-based search on recipe names)
Fail to adapt recipes to meet the real-world situation (e.g. due to lack of ingredients or user preference)

54. A Personalized Recipe Database System -- Our Contributions
Propose a recipe model which encompasses static attributes as well as dynamic behaviours (e.g. cooking procedures and constraints)
Present a novel perspective of evaluating the “quality” of a recipe by constructing and analysing its cooking graph (capture both action flows and data/ingredient flows)
Provide a promising way to address the problem of recipe adaptation heuristically (with flexible and feasible solutions)

55. Recipe on the Web

56. Sample Recipe -- The Cooking Procedure of “Triple Cheese Pasta Primavera”
Step
number
Recipe cooking procedure
in steps 1
Dice bell peppers. Slice squash and mushrooms. 2
Cook pasta according to package directions in unsalted water. 3
Meanwhile, in a large skillet melt butter. Add bell peppers; cook and stir occasionally until barely tender, about three minutes. 4
Add squash and mushrooms; cook and stir occasionally until barely tender, about four minutes. 5
Drain pasta; toss with vegetables in skillet. 6
In the saucepot in which the spaghetti was cooked, combine ricotta, mozzarella, milk, Parmesan, Italian seasoning, salt and black pepper. Over a medium-low heat cook and stir cheese mixture just until hot, about 1 minute. 7
Add reserved pasta and vegetables; toss to coat; remove to a serving platter.

57. Parsing the Cooking Procedure of “Triple Cheese Pasta Primavera”
Sample Recipe
Parsing the Cooking Procedure of “Triple Cheese Pasta Primavera”

58. Recipe Model
A recipe R is modeled and represented by a tuple of three elements:
R = <M, RP, SP>
where
(a) M={Mi | i = 1.. m} – a set of ingredients. An ingredient Mi is either a basic ingredient or a set of ingredients:
Mi = <MID, MP>, MID—unique identity, MP—member level properties (and functions) such as the name, quantity and image
An ingredient Mi belongs to one of the three classes: Main, Minor and Seasoning;
(b) RP is a set of recipe-level properties (and functions) applied on R itself, such as the main cooking style, region, nutrition and images of the dish of the recipe;

59. Recipe Model
(c) SP = (V, E, Cons, Ingr) is a labeled directed “Cooking Graph”,
V={vi | i = 1..n} is a set of nodes.
vi—a cooking action
“cooking action constraints”: Cons(vi)—associated constraint conditions that should be satisfied when the action of vi takes place. e.g. conditions on temperature and duration etc.
E is a set of directed edges on V—temporal execution flow of the cooking actions; named “action flows”.
An edge <vi ,vj> —vj should take place after vi.
“cooking transition constraints”: Cons(vi , vj) –the conditions that should be satisfied for the flow to take place.
Ingr(vi) – ingredients that should be added into vi
O(vi) –the output ingredients of vi
These inputs and outputs for the nodes are called “ingredient flows”.

60. The Cooking Graph of “Triple Cheese Pasta Primavera”

61. Basic Properties
Definition 1. (Reachability) A cooking graph is defined as “reachable” if each of its nodes is “reachable”; a node is “reachable” if it is on a directed path from a starting node to the end node.
Definition 2. (Consistency) A cooking graph is defined to be “consistent” if the conditions for each node/edge is consistent (i.e. there exists assignment to variables to make the conditions true).

62. Constraints and Rules
Definition 3. (Constraint) A constraint is a predicate followed by one or more terms, enclosed in parentheses and separated by commas; a term is either a constant, variable or function expression.
Constraints specify all kinds of conditions or restrictions in the recipe model;
Three categories: intra-recipe constraints, inter-recipe constraints and outer-recipe constraints.
Incompatible(Spinach, Tofu) says spinach and tofu are incompatible and should not be cooked together.

63. Constraints and Rules
Definition 4. (Rule) A rule is a logical implication of the form “If Ф Then Ψ” (or, ), where Ф and Ψ are sentences.
Validate the correctness of a recipe through reasoning and recognition process.
Handle complex situations such as to make necessary adjustment or compensation once an improper cooking action occurs.
Describe cooking skills that have been widely accepted and commonly used.
Over_Put(salt) → Add(vinegar|water) says that if too much salt has been put into a dish, then neutralize the salty taste by adding either vinegar or water.

64. Recipe Cooking Graph Mining
Pattern — Some subgraphs occur in one or more cooking graphs and they have certain influence on the cooking effects (e.g. taste, appearance).
Find patterns for a set of recipes
What’s usually done and what’s usually put in the cooking procedure (one action, a series of actions, an ingredients, a set of ingredients, actions combined with ingredients)
Cooking graphs of different recipes may share the same pattern
Distinct subgraphs that determine the cooking effect (e.g. taste) should be identified

65. Sample Patterns

66. Sample Cooking Style Cooking Style Pattern with Dominating Action
Soft Deep-frying
Coating + Passing Oil + deep-fry
Dry Deep-frying
Marinating + Coating + deep-fry
Cooked-frying
Passing Oil/Blanching/Steaming+ stir-fry (+ Thickening)
Slip-frying
(Marinating + Coating) + Passing Oil + stir-fry + Thickening
Soft Stirring
Blanching/Steaming+ stir + Thickening
Braising
Passing Oil/Blanching/Steaming + simmer in sauce (+ Thickening)
Simmering
Blanching + simmer in water/broth
Generally describe how a recipe is cooked in a Pattern Combination or in Graph Abstraction.

67. User Adaptation
Usually a user wants to make a dish that has the same cooking result (e.g. taste, appearance) as the recipe exhibits.
Unfortunately, the user is very likely to get a slightly or even totally different dish as he/she modifies the cooking procedure.
Objective reasons—e.g. lack of some ingredients, Subjective reasons—e.g. wrong cooking actions by carelessness or personal preference.

68. Template Selection and Instantiation
User Adaptation
When the user makes an adaptation, the system will check if the modified cooking graph is feasible.
If not, a set of feasible templates are provided.
The remaining subgraph is replaced by the user selected one.
Property check (Reachability, Consistency)
Template Selection and Instantiation

69. Prototype System Global System vs. User Space

70. Prototype System – Recipe Browser

71. Prototype System – Cooking Pattern Miner

72. Prototype System – Similarity Calculator

73. Summary Proposed a data model to represent a recipe
Advocated cooking graph mining to find frequent used patterns (actions, ingredients)
Attempt to solve recipe adaptation problem by using patterns as templates
Developed a prototype system—RecipeView
Further work include:
discover patterns of cooking graphs
Refine and strengthen the algorithm of recipe adaptation

74. Application Scenario

75. Application Scenario Advantages (vs. traditional retrieval techniques)
Easy-to-compose query
By browsing (to get “seed” objects of arbitrary modalities)
By subject (simply keyword) at various abstraction level
Multi-modal results
a collection of images, text docs, videos, etc
vs. a single type of media
Semantically relevant results
natural outcome of exploring previously learnt knowledge
vs. a set of specifically chosen features

76. Materialized knowledge
Advantages (cont’d)
“Hill-climbing” Effect – retrieval performance grows as more user interactions are conducted
Materialized knowledge
Retrieval process
exploration
encourage
learning
User interactions

77. Conclusion
MediaView – a semantic multimedia database modeling mechanism
to bridge the semantic gap between conventional database and semantics-intensive multimedia applications
A set of user-level operators to accommodate the specialization/generalization relationships among the media views

78. Conclusion
MediaView promises more effective access to the content of media databases
Users could get the right stuff and tailor it to the context of their application easily.
Providing the most relevant content from pre-learnt semantic links between media and context
 high performance database browsing and multimedia authoring tools can enable more comprehensive applications to the user

79. Conclusion
Users could customize specific media view according to their tasks, by using user-level operators
The effectiveness of using MediaView in the experimental problem domains
Multimedia recipe database
Cross-media indexing and retrieval

80. Further Issues
The development and transition of MediaView to a fully-fledged multimedia database system supporting “declarative” queries
Intensive and extensive performance studies
Advanced semantic relations (eg. temporal and spatial ones) can also be incorporated in combining individual media views

81. Email: Qing.Li@cityu.edu.hk
Thank you!
Q & A