1. How to identify complex events in the real world First part : Vision –Image segmentation –Checking for basic events : touching, intersecting, moving, pointing, … Second part : –Defining complex events in terms of basic events –Identifying the complex events

2. Low level vision : overview Properties of individual objects : - xpos - ypos - bounding box - area - color - ratio - movement vector - bounding polygon (*) - #edges / angles (*) - orientation (*) - type {square, rectangle, circle, triangle} (*) Relations between objects : - touching (*) - intersecting (*) - pointing (*)

3. Fig.1 : real world imageFig.2 : bounding boxesFig.3 : bounding polygons 1.Acquisition of real world image. 2.Segmentation with region growing  all basic features 3.Bounding polygon  more advanced features Components of low level vision.

4. Segmentation : region growing Based on Tony’s pseudocode : segmentID = 0 for all coordinates x and y if ([x,y] not visited AND interesting I(x,y)) enqueue [x,y] while queue not empty p  dequeue if interesting p add p to segment[segmentID] enqueue interesting neigbours of p mark p as visited segmentID  segmentID + 1 mark [x,y] as visited

5. Segmentation Step parameter : check every nth pixel in stead of every pixel. A pixel is “interesting” if it’s not very white and has more or less the same color as it’s neighbour. Pixels are marked “visited” by coloring them white, so there is no extra datastructure for these pixels. Individual pixels of a segment are not stored in a datastructure. Only the coordinates of the bounding-box are stored.

6. Matching Every time a frame is grabbed, the new frame is matched against the previous frame. An object in both frames is the same if the bounding box is more or less the same. Handle new / lost objects. Calculate motion vectors for each object. Calculate an average motion vector. If the differences between the average and the individual motion vectors are very small, then the camera is moving.

7. Matching Problems Whenever an object is lost there should be some kind of recovery strategy. –When an object has moved too much, it is lost. It might be useful checking other features (color?) to find the lost object. –When one object splits a second object, the match fails, 2 objects are lost and 3 new objects are found. –When a new object enters, a new object is found. Next frame, it has become bigger, so the match fails, a new object is found, …

8. Finding bounding polygons 1.Take the bounding box from the segmentation algorithm. 2.Traverse each of the sides of the bbox from the left and the right. Mark the endpoints of the segment (first pixel close enough to the average color of the segment) 3.Establish a polygon with the endpoints. Simplify it : remove angles of 180° and merge points too close to each other.

9. Problems with bounding polygons When two objects with the same color are within each others bounding boxes … Possible solution : Store all pixels of a segment. Whenever a bounding point is found, check if it’s really inside the segment.

10. Polygon classification Classification based on properties of edges and angles. (“Every polygon with three edges is a triangle” etc.) First test results :

11. Towards high level vision Expressing more complex events in terms of basic events. (define (pick-up x y) (exists (j) (and (during j (move(hand x))) (during j (contacts(hand x) y)) (during j (attached(hand x) y)) (during j (move y)))))

12. Basic Events : very straightforward definitions ! New / lost objects. Movement : threshold on the movement vector. Touching : 2 objects touch when their bounding polygons touch or are very close to each other. Intersecting : 2 objects intersect when their bounding polygons are intersecting. Support : An object supports another when the removal of the first causes the second to fall. Pointing : An object is pointing at an other object if the first object is more or less oriented towards the second object.

13. Basic events : Support The Support event requires explicit reasoning and is very hard to check. Simplification : All objects on the table are supported by the table. An object on top of another object is supported by the lower one. An object in a robot hand is supported by the robot. An object is in the robot hand when the object touches the hand and is attached to it (= intersection of polygons) When objects always stay on the table, the support event becomes trivial.

14. Building complex events A rule defines how a complex event is assembled from basic events. The Event manager has a memory of rule phases. A rule-phase is an entry of the form (rulename, objectbinding1, objectbinding2, phase). (pick-up, 3,,2) means that object 3 is possibly picking up another unknown object. The phase indicates how many subrules have successfully been evaluated.

15. Evaluating complex events Whenever a “lost object” event occurs, all rulephases with an object-binding to that object, are removed from memory. Every frame we check if we can create new rulephases. Only new or moved objects can cause a new rulephase to be created. When there are no new objects and no object has moved, there simply cannot be new events. Every frame we try to advance all rulephases in memory and extend their object bindings.

16. Evaluating complex events The extension of rulephases may require other events (touching, intersecting, pointing, …) to be checked. Extension : The evaluation of one rule may force the evaluation of another rule : subrules can be rules themselves. Finally, we need to check for rulephases that were evaluated successfully.