1. OpenCV Introduction Hang Xiao Oct 26, 2012

2. History  1999 Jan : lanched by Intel, real time machine vision library for UI, optimized code for intel  2000 Jun : OpenCV alpha 3 。  2000 Dec : OpenCV beta 1 for linux  2006 : the first 1.0 version supports Mac OS  2008 mid : obtain corporate support from Willow Garage  2009 Sep : OpenCV 1.2 （ beta2.0  2009 Oct : Version 2.0 released 。  2010 Dec : OpenCV 2.2 。  2011 Aug : OpenCV 2.3 。  2012 Apr : OpenCV 2.4.

3. Overview  Goals  Develop a universal toolbox for research and development in the field of Computer Vision  Algorithms  More than 350 algorithms, 500 API  Programming language  C/C++, C#, Ch, Python, Ruby, Matlab, and Java (using JavaCV)  OS support  Windows, Android, Maemo, FreeBSD, OpenBSD, iOS, Linux and Mac OS.  Licence  BSDlisence, free for commercial and non-commmercial

4. Overview - Applications  2D and 3D feature toolkits  Egomotion estimation  Facial recognition system  Gesture recognition  Human–computer interaction (HCI)  Mobile robotics  Motion understanding  Object identification  Segmentation and Recognition  Stereopsis Stereo vision: depth perception from 2 cameras  Structure from motion (SFM)Motion tracking

5. Overview - A statistical machine learning library  Boosting (meta-algorithm)  Decision tree learning  Gradient boosting trees  Expectation-maximization algorithm  k-nearest neighbor algorithm  Naive Bayes classifier  Artificial neural networks  Random forest  Support vector machine (SVM)

6. Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

7. Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

8. Image Analysis  Thresholds  Statistics  Pyramids  Morphology  Distance transform  Flood fill  Feature detection  Contours retrieving

9. Image Thresholding  Fixed threshold;  Adaptive threshold;

10. Image Thresholding Examples Source picture Fixed threshold Adaptive threshold

11. Statistics  min, max, mean value, standard deviation over the image  Norms C, L1, L2  Multidimensional histograms  Spatial moments up to order 3 (central, normalized, Hu)

12. Multidimensional Histograms  Histogram operations calculation, normalization, comparison, back project  Histograms types: Dense histograms Signatures (balanced tree)  EMD algorithm The EMD computes the distance between two distributions, which are represented by signatures. The signatures are sets of weighted features that capture the distributions. The features can be of any type and in any number of dimensions, and are defined by the user. The EMD is defined as the minimum amount of work needed to change one signature into the other

13. EMD – a method for the histograms comparison

14. Image Pyramids  Gaussian and Laplacian pyramids  Image segmentation by pyramids

15. Image Pyramids  Gaussian and Laplacian

16. Pyramid-based color segmentation On still pictures And on movies

17. Morphological Operations  Two basic morphology operations using structuring element: erosion dilation  More complex morphology operations: opening : erosion + dilation closing : dilation + erosion morphological gradient : the difference between the dilation and the erosion of an image top hat : the difference between an input image and its opening black hat : the difference between the closing and its input image

18. Morphological Operations Examples  Morphology - applying Min-Max. Filters and its combinations Opening IoB= (I  B)  B Dilatation I  B Erosion I  B Image I Closing IB= (I  B)  BTopHat(I)= I - (I  B)BlackHat(I)= (I  B) - I Grad(I)= (I  B)-(I  B)

19. Distance Transform  Calculate the distance for all non-feature points to the closest feature point  Two-pass algorithm, 3x3 and 5x5 masks, various metrics predefined

20. Flood Filling  Simple  Gradient

21. Feature Detection  Fixed filters (Sobel operator, Laplacian);  Optimal filter kernels with floating point coefficients (first, second derivatives, Laplacian)  Special feature detection (corners)  Canny operator  Hough transform (find lines and line segments)  Gradient runs

22. Canny Edge Detector

23. Hough Transform Detects lines in a binary image Probabilistic Hough TransformProbabilistic Hough Transform Standard Hough TransformStandard Hough Transform

24. Another Sample of the Hough Transform Using Source picture Result

25. Contour Retrieving  The contour representation: Chain code (Freeman code) Polygonal representation Initial Point Chain code for the curve: 34445670007654443 Contour representation

26. Hierarchical representation of contours Image Boundary (W1)(W2)(W3) (B2)(B3)(B4) (W5)(W6)

27. Contours Examples Source Picture (300x600 = 180000 pts total) Retrieved Contours (<1800 pts total) After Approximation (<180 pts total) And it is rather fast: ~70 FPS for 640x480 on complex scenes

28. Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

29. Structural Analysis  Contours processing  Approximation  Hierarchical representation  Shape characteristics  Matching  Geometry  Contour properties  Fitting with primitives  PGH: pair-wise geometrical histogram for the contour.

30. Contour Processing  Approximation: RLE algorithm (chain code) Teh-Chin approximation (polygonal) Douglas-Peucker approximation (polygonal);  Contour moments (central and normalized up to order 3)  Hierarchical representation of contours  Matching of contours

31. Hierarchical Representation of Contours  A contour is represented with a binary tree  Given the binary tree, the contour can be retrieved with arbitrary precision  The binary tree is quasi invariant to translations, rotations and scaling

32. Contours matching  Matching based on hierarchical representation of contours

33. Geometry  Properties of contours: (perimeter, area, convex hull, convexity defects, rectangle of minimum area)  Fitting: (2D line, 3D line, circle, ellipse)  Pair-wise geometrical histogram

34. Pair-wise geometrical histogram (PGH)

35. Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

36. Object Recognition  Eigen objects  Hidden Markov Models

37. Eigen Objects

38. Eigen objects (continued)

39. Hidden Markov Model Definitions - The set of states - The set of measurements - The state at time t - The transition probability matrix - The conditional probability matrix - The starting states distribution

40. Embedded HMM for Face Recognition Model- - Face ROI partition

41. Face recognition using Hidden Markov Models One person – one HMM Stage 1 – Train every HMM Stage 2 – Recognition P i - probability Choose max(P i ) … 1 n i

42. Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

43. Motion Analysis and Object Tracking  Background subtraction  Motion templates  Optical flow  Active contours  Estimators

44. Background Subtraction  Background model (normal distribution)  Background statistics functions: Average Standard deviation Running average

45. Motion Templates  Object silhouette  Motion history images  Motion history gradients  Motion segmentation algorithm silhouetteMHI MHG

46. Motion Segmentation Algorithm  Two-pass algorithm labeling all motion segments

47. Motion Templates Example  Motion templates allow to retrieve the dynamic characteristics of the moving object

48. Optical Flow  Block matching technique  Horn & Schunck technique  Lucas & Kanade technique  Pyramidal LK algorithm  6DOF (6 degree of freedom) algorithm Optical flow equations:

49. Pyramidal Implementation of the optical flow algorithm J imageI image Image Pyramid Representation Iterative Lucas – Kanade Scheme Generic Image (L-1)-th Level L-th Level Location of point u on image u L =u/2 L Spatial gradient matrix Standard Lucas – Kanade scheme for optical flow computation at level L d L Guess for next pyramid level L – 1 Finally, Image pyramid building Optical flow computation

50. 6DOF Algorithm Parametrical optical flow equations:

51. Active Contours  Snake energy:  Internal energy:  External energy:  Two external energy types:

52. Estimators  Kalman filter  ConDensation filter

53. Kalman object tracker

54. Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

55. 3D reconstruction  Camera Calibration  View Morphing  POSIT

56. Camera Calibration  Define intrinsic and extrinsic camera parameters.  Define Distortion parameters

57. Camera Calibration Now, camera calibration can be done by holding checkerboard in front of the camera for a few seconds. And after that you ’ ll get: 3D view of etalon Un-distorted image

58. View Morphing

59. POSIT Algorithm  Perspective projection: Weak-perspective projection:

60. OpenCV Websites  http://opencv.org OpenCV official webpage.  http://opencvlibrary.sourceforge.net/ http://opencvlibrary.sourceforge.net/ OpenCV documentation and FAQs.

61. OpenCV Examples  adaptiveskindetector : detect skin area  fback_c : dense Franeback optical flow  contours : calculate contours on different levels  delaunay : delaunay triangle  find_obj : SURF Detector and Descriptor using either FLANN or brute force matching on planar objects  morphology : open/close, erode/dilate  motempl ： motion templates  mser_sample : Maximal Extremal Region interest point detector  polar_transforms : illustrates Linear-Polar and Log-Polar image transforms  pyramid_segmentation : color pyramid segmentation

62. Thanks !!!