2. Symbolic Description and Visual Querying of Image Sequences Using Spatio-Temporal Logic Del Bimbo, E. Vicario and D. Zingoni IEEE Transactions on Knowledge and Data Engineering, 7(4), August 1995.

3. Outline Overview Spatial logic  Region-based formulation  Object-based formulation Temporal logic Sequence Description using STL A System for Image Sequence Retrieval

4. Overview: Spatio-temporal logic (STL) A symbolic representation of spatio-temporal relationships between objects within image sequences. Completely based on logic. Two main components:  Spatial logic for individual scenes  Temporal logic for scene sequences

5. Spatial logic Expresses geometric ordering and relationships between objects  In fact, between the projections of the objects in N-dimensional space. Based on knowing the points (or regions) that are covered by the object.

6. Spatial logic (cont.) A scene is a represented as a triple (V, Obj, F):  V: N-dimensional space Mostly N=2 or 3  Obj: a set of objects in V  F: a mapping from Obj to the powerset of V, i.e. the set of all subsets of V. Associates each object with a set of points over which it stands, i.e. F(p) := { r  V | p stands over r)

7. Spatial logic (cont.) V can be partitioned into a grid of rectangular regions.  A scene will be then represented as a discrete scene.  And F will be then a mapping between objects and the regions they cover.

8. Example I11I11 I12I12 I13I13 I14I14 e1e1 I21I21 I22I22 I23I23 I24I24 e2e2 p2p2 p1p1

9. Spatial logic (cont.) Two possible formulations:  Region-based formulation  Object-based formulation

10. Region-based formulation Express the positioning of a set of objects with respect to a single region in the scene. A spatial assertion has the form: where In addition, there are some shorthand operations:  spatial-eventually (can be +ve or –ve)  spatial-always (can be +ve or –ve)

11. Semantics (S E, J, e n ) |= p - iff the orthogonal projections on axis e n of region g(J) and object p have a nonempty intersection (S E, J, e n ) |=   - iff the spatial assertion (S E, J, e n ) |=  does not hold (S E, J, e n ) |=  1   2 - iff both (S E, J, e n ) |=  1 and (S E, J, e n ) |=  2

12. Semantics (Cont.) (S E, J, e n ) |=  1 unt S+  2 - iff there esists a region g(J’) which is reached from region g(J) moving along the positive direction of axis e n such that assertion  2 holds in region g(J’) and  1 holds in all the regions from g(J) to g(J’) (S E, J, e n ) |=  1 unt S-  2 - iff there esists a region g(J’) which is reached from region g(J) moving along the negetive direction of axis e n such that assertion  2 holds in region g(J’) and  1 holds in all the regions from g(J) to g(J’)

13. Shorthands (S E, J, e n ) |= ◊ S+/-  -  will eventually hold in some of the regions encountered moving from region g(J) along axis e n (S E, J, e n ) |=  S+/-  -  holds in all the regions encountered moving from region g(J) along axis e n

14. Example enen emem p g(K) g(J)

15. Object-based formulation Express the relationships between objects. A spatial assertion has the form: ( read:  holds in q) where q is an object, expresses that (S E, J, e n ) |=  holds in any region g(J) containing q Five different situations occur as follows:

16. (S E, q, x ) |= p - expresses that the projection of q on the x axis is entirely contained in the projection of p q p p p p

17. (S E, q, x ) |= ◊ S+ p - expresses that every point of the projection of q on the x axis has at least one point of the projection of p to its right side p p p p p q

18. (S E, q, x ) |= q unt S+ p - expresses that the projection of q extends until one point is found that belongs to the projection of p q p p p p q

19. (S E, q, x ) |=  p - expresses that the projection of q on the x axis does not intersect with the projection of p p q p

20.  ((S E, q, x ) |=  p ) - expresses that some of the regions of q are aligned with some of the regions of p q p p p p p q p p p p p p

21. Temporal logic In general, used to express the ordering of states or actions in time. A temporal (or state assertion) has the form: where

22. Semantics ( , k) |=  iff the spatial assertion  holds in the kth scene of sequence  ( , k) |=  iff the temporal assertion ( , k) |=  does not hold ( , k) |=  1   2 iff both ( , k) |=  1 and ( , k) |=  2 hold ( , k) |=  1 unt t  2 iff  2 holds in a scene with index k’ > k and  1 holds in all the scenes form k to k’

23. Shorthands Two shorthands can be derived ( , k) |= ◊ t  means that  will hold in some scene subsequent to the kth one ◊ t := true unt t  ( , k) |=  t  means that  holds in all the scene subsequent to the kth one  t  :=  ( true unt t (  ) )

24. Describing Scene Sequences - each node is labeled with the spatial assertion, satisfied in its scene - S1, S2, S3, S4 and S5 are different states on a time line  ( , k) |= (  1 unt t  2 ) holds for ……………  ( , k) |= ◊ t  2 holds for ………………  ( , k) |=  t (  1   2 ) holds for …………….. S 1 S 2 S 3 S 4 S 5  1  1  1  2

25. Sequence Description Using STL 3D scene-based descriptions are used to avoid ambiguity Two different descriptions are possible 1) Observer-centered description 2) Object-centered description

26. Two Different Descriptions Observer-centered description -- images of the same scene taken from different viewpoints -- valid only when the camera is fixed Object-centered description -- how one object sees the rest of the scene -- does not depend on the camera position

27. Example

28. the spatial positions of the house, h as perceived by the car, c along the course of the scene considering object-centered descriptions and referring to the reference system E c associated with the car c   1 := (S Ec, c, x) |= h   2 :=  ((S Ec, c, x) |=  h )   3 := (S Ec, c, y) |= h   4 := (S Ec, c, z) |= h   5 :=  ((S Ec, c, z) |=  h ) The description of the scene using the above assertions can be as follows:

29. A System for Image Sequence Retrieval A. Iconic Querying 1) Visual Querying 2) Automatic Parsing B. Retrieval from Database 1) Sequence Representation (created manually) 2) Sequence Retrieval

30. Retrieval from Database To allow for different level of details in queries, three levels of operations are used.  Level 1: Least details 3 possibilities ( before, overlapping, after )  Level 2: Modest details 6 possibilities  Level 3: Very fine details 13 possibilities ( STL full representational power )

31. Level-1 Operators

32. Level-2 Operators

33. User Interface of the Retrieval System

34. Example 1 (Query Specification)

35. Result of Retrieval Only two frames of the Sequence are shown

36. Example 2 (Query Specification) Definition of motion of one car during the play back of the other

37. Result of Retrieval Two frames of the sequence