1. Agenda

2. Presentation of ImageLab
Digital Library
content-based retrieval
Computer Vision
for robotic automation
Multimedia: video annotation
Medical Imaging
Video analysis
for indoor/outdoor
surveillance
Off-line Video analysis
for telemetry
and forensics
People and vehicle surveillance
Imagelab-Softech
Lab of Computer Vision,
Pattern Recognition and Multimedia
Dipartimento di Ingegneria dell’Informazione
Università di Modena e Reggio Emilia Italy

3. Imagelab: recent projects in surveillance
European
International
Italian & Regional
With Companies
THIS Transport hubs intelligent surveillance EU JLS/CHIPS Project
VIDI-Video: STREP VI FP EU  (VISOR VideosSurveillance Online Repository)
BE SAFE NATO Science for Peace project
Detection of infiltrated objects for security Australian Council
Behave_Lib : Regione Emilia Romagna Tecnopolo Softech
LAICA Regione Emilia Romagna;
FREE_SURF MIUR PRIN Project
Building site surveillance: with Bridge-129 Italia
Stopped Vehicles with Digitek Srl
SmokeWave: with Bridge-129 Italia
Sakbot for Traffic Analysis with Traficon
Mobile surveillance with Sistemi Integrati 2007
Domotica per disabili: posture detection FCRM

4. AD-HOC: Appearance Driven Human tracking with Occlusion Handling

5. Key aspects Based on the SAKBOT system Appearance based tracking
Background estimation and updating
Shadow removal
Appearance based tracking
we aim at recovering a pixel based foreground mask, even during an occlusion
Recovering of missing parts from the background subtraction
Managing split and merge situations
Occlusion detection and classification
Classify the differences as real shape changes or occlusions

6. Example 1 (from ViSOR)
In this simple example you can have a visual output of the tracking system.
In the upper right image we see the visual objects extracted from BS, in the lower image we see appearance models and probability masks. Probability masks are colored from red to blue. Red means low probability, blue high probability.
Finally in the upper left image we how pixels are assigned to different tracks.
How can we do this…

7. Example 2 from PETS 2002
This test video is extracted from the 2002 pets workshop dataset. It’s extremely difficult for some reasons: …
In this video we see that the system can handle difficult situations of multiple people occlusion in outdoor environment. The problem is challenging because of segmentation errors, flattened colours and reflections due to the shop window.

8. Example 3
Now if we take the previous example, and we apply the selective update, we can see that the table occlusion is handled correctly. Regions of the track classified as occluded are shown with dark colors on the probability mask. In those regions, and only on those, model update is not performed. Therefore, even on long-lasting occlusion, both bounding box and centroid of the tracks are correctly estimated.

9. Other experimental results
Imagelab videos (available on ViSOR)
PETS series
The algorithm presented is much less computationally requiring than other popular tracking algorithms. It depends on frame size, number and dimensions of track, but it works around 10 fps on standard hardware.

10. Results on the PETS2006 dataset
Working in real time at 10 fps!

11. Posture classification

12. Distributed surveillance with non overlapping field of view

13. Exploit the knowledge about the scene
To avoid all-to-all matches, the tracking system can exploit the knowledge about the scene
Preferential paths -> Pathnodes
Border line / exit zones
Physical constraints & Forbidden zones NVR
Temporal constraints

14. Tracking with pathnode
A possible path between Camera1 and Camera 4

15. Pathnodes lead particle diffusion

16. Results with PF and pathnodes
Single camera tracking: Multicamera tracking
Recall=90.27% Recall=84.16%
Precision=88.64% Precision=80.00%

17. “VIP: Vision tool for comparing Images of People”
Lantagne & al., Vision Interface 2003
Each extracted silhouette is segmented into significant region using the JSEG algorithm
( Y. Deng ,B.S. Manjunath: “Unsupervised segmentation of color-texture regions in images and video” )
Colour and texture descriptors are calculated for each region
The colour descriptor is a modified version of the descriptor
presented in Y. Deng et al.: “Efficient color representation for
Image retrieval”.
Basically an HSV histogram of the dominant colors.
The texture descriptor is based on D.K.Park et al.: “Efficient
Use of Local Edge Histogram Descriptor”.
Essentially this descriptor characterizes the edge density
inside a region according to different orientations ( 0°, 45°,
90° and 135° )
The similarity between two regions is the weighted sum of
the two descriptor similarities:

18. To compare the regions inside two silhouette, a region matching scheme is used,
involving a modified version of the IRM algorithm presented in J.Z. Wang et al, ”Simplicity:
Semantics-sensitive integrated matching for picture libraries” .
The IRM algorithm is simple and works as follows:
1) The first step is to calculate all of the similarities
between all regions.
2) Similarities are sorted in decreasing order, the
first one is selected, and areas of the
respective pair of regions are compared.
A weight, equal to the smallest percentage area
between the two regions, is assigned to the
similarity measure.
3) Then, the percentage area of the largest region is updated by removing the
percentage area of the smallest region so that it can be matched again.
The smallest region will not be matched anymore with any other region.
4) The process continues in decreasing order for all of the similarities.
In the end the overall similarity between the two region sets is calculated as:

19. ViSOR: Video Surveillance Online Repository

20. The ViSOR video repository

21. Aims of ViSOR
Gather and make freely available a repository of surveillance videos
Store metadata annotations, both manually provided as ground-truth and automatically generated by video surveillance tools and systems
Execute Online performance evaluation and comparison
Create an open forum to exchange, compare and discuss problems and results on video surveillance

22. Different types of annotation
Structural Annotation: video size, authors, keywords,…
Base Annotation: ground-truth, with concepts referred to the whole video. Annotation tool: online!
GT Annotation: ground-truth, with a frame level annotation; concepts can be referred to the whole video, to a frame interval or to a single frame. Annotation tool: Viper-GT (offline)
Automatic Annotation: output of automatic systems shared by ViSOR users.

23. Video corpus set: the 14 categories

24. Outdoor multicamera
Synchronized views

25. Surveillance of entrance door of a building
About 10h!

26. Videos for smoke detection with GT

27. Videos for shadow detection
Already used from many researcher working on shadow detection
Some videos with GT
A. Prati, I. Mikic, M.M. Trivedi, R. Cucchiara, "Detecting Moving Shadows: Algorithms and Evaluation" in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, n. 7, pp , July, 2003

28. Some statistics
We need videos and annotations!

29. SIMULTANEOUS HMM action SEGMENTATION AND Recognition
Action recognition
SIMULTANEOUS HMM action SEGMENTATION AND Recognition

30. Probabilistic Action Classification
Classical approach:
Given a set of training videos containing an atomic action each (manually labelled)
Given a new video with a single action
… find the most likely action
Dataset: "Actions as Space-Time Shapes (ICCV '05)." M. Blank, L. Gorelick, E. Shechtman, M. Irani, R. Basri

31. Classical HMM Framework
Definition of a feature set
For each frame t, computation of the feature set Ot (observations)
Given a set of training observations O={O1…OT} for each action, training of an HMM (k) for each action k
Given a new set of observations O={O1…OT}
Find the model (k) which maximise P(k|O)

32. A sample 17-dim feature set
Computed on the extracted blob after the foreground segmentation and people tracking:

33. From the Rabiner tutorial

34. Online action Recognition
Given a video with a sequence of actions
Which is the current action? Frame by frame action classification (online – Action recognition)
When does an action finish and the next one start? (offline – Action segmentation)
R. Vezzani, M. Piccardi, R. Cucchiara, "An efficient Bayesian framework for on-line action recognition" in press on Proceedings of the IEEE International Conference on Image Processing, Cairo, Egypt, November 7-11, 2009

35. Main problem of this approach
I do not know when the action starts and when it finishes.
Using all the observations, the first action only is recognized
A possible solution: “brute force”. For each action, for each starting frame, for each ending frame, compute the model likelihood and select the maximum. UNFEASIBLE

36. Our approach Subsample of the starting frames (1 each 10)
Adoption of recursive formulas
Computation of the emission probability once for each model (Action)
Current frame as Ending frame
Maximum length of each action
The computational complexity is compliant with real time requirements

37. Different length sequences
Sequences with different starting frame have different length
Unfair comparisons using the traditional HMM schema
The output of each HMM is normalized using the sequence length and a term related to the mean duration of the considered action
This allows to classify the current action and, at the same time, to perform an online action segmentation