OpenCV Introduction Hang Xiao Oct 26, 2012

History  1999 Jan : lanched by Intel, real time machine vision library for UI, optimized code for intel  2000 Jun : OpenCV alpha 3 。  2000 Dec : OpenCV beta 1 for linux  2006 : the first 1.0 version supports Mac OS  2008 mid : obtain corporate support from Willow Garage  2009 Sep : OpenCV 1.2 （ beta2.0  2009 Oct : Version 2.0 released 。  2010 Dec : OpenCV 2.2 。  2011 Aug : OpenCV 2.3 。  2012 Apr : OpenCV 2.4.

Overview  Goals  Develop a universal toolbox for research and development in the field of Computer Vision  Algorithms  More than 350 algorithms, 500 API  Programming language  C/C++, C#, Ch, Python, Ruby, Matlab, and Java (using JavaCV)  OS support  Windows, Android, Maemo, FreeBSD, OpenBSD, iOS, Linux and Mac OS.  Licence  BSDlisence, free for commercial and non-commmercial

Overview - Applications  2D and 3D feature toolkits  Egomotion estimation  Facial recognition system  Gesture recognition  Human–computer interaction (HCI)  Mobile robotics  Motion understanding  Object identification  Segmentation and Recognition  Stereopsis Stereo vision: depth perception from 2 cameras  Structure from motion (SFM)Motion tracking

Overview - A statistical machine learning library  Boosting (meta-algorithm)  Decision tree learning  Gradient boosting trees  Expectation-maximization algorithm  k-nearest neighbor algorithm  Naive Bayes classifier  Artificial neural networks  Random forest  Support vector machine (SVM)

Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

Image Analysis  Thresholds  Statistics  Pyramids  Morphology  Distance transform  Flood fill  Feature detection  Contours retrieving

Image Thresholding  Fixed threshold;  Adaptive threshold;

Image Thresholding Examples Source picture Fixed threshold Adaptive threshold

Statistics  min, max, mean value, standard deviation over the image  Norms C, L1, L2  Multidimensional histograms  Spatial moments up to order 3 (central, normalized, Hu)

Multidimensional Histograms  Histogram operations calculation, normalization, comparison, back project  Histograms types: Dense histograms Signatures (balanced tree)  EMD algorithm The EMD computes the distance between two distributions, which are represented by signatures. The signatures are sets of weighted features that capture the distributions. The features can be of any type and in any number of dimensions, and are defined by the user. The EMD is defined as the minimum amount of work needed to change one signature into the other

EMD – a method for the histograms comparison

Image Pyramids  Gaussian and Laplacian pyramids  Image segmentation by pyramids

Image Pyramids  Gaussian and Laplacian

Pyramid-based color segmentation On still pictures And on movies

Morphological Operations  Two basic morphology operations using structuring element: erosion dilation  More complex morphology operations: opening : erosion + dilation closing : dilation + erosion morphological gradient : the difference between the dilation and the erosion of an image top hat : the difference between an input image and its opening black hat : the difference between the closing and its input image

Morphological Operations Examples  Morphology - applying Min-Max. Filters and its combinations Opening IoB= (I  B)  B Dilatation I  B Erosion I  B Image I Closing IB= (I  B)  BTopHat(I)= I - (I  B)BlackHat(I)= (I  B) - I Grad(I)= (I  B)-(I  B)

Distance Transform  Calculate the distance for all non-feature points to the closest feature point  Two-pass algorithm, 3x3 and 5x5 masks, various metrics predefined

Flood Filling  Simple  Gradient

Feature Detection  Fixed filters (Sobel operator, Laplacian);  Optimal filter kernels with floating point coefficients (first, second derivatives, Laplacian)  Special feature detection (corners)  Canny operator  Hough transform (find lines and line segments)  Gradient runs

Canny Edge Detector

Hough Transform Detects lines in a binary image Probabilistic Hough TransformProbabilistic Hough Transform Standard Hough TransformStandard Hough Transform

Another Sample of the Hough Transform Using Source picture Result

Contour Retrieving  The contour representation: Chain code (Freeman code) Polygonal representation Initial Point Chain code for the curve: Contour representation

Hierarchical representation of contours Image Boundary (W1)(W2)(W3) (B2)(B3)(B4) (W5)(W6)

Contours Examples Source Picture (300x600 = pts total) Retrieved Contours (<1800 pts total) After Approximation (<180 pts total) And it is rather fast: ~70 FPS for 640x480 on complex scenes

Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

Structural Analysis  Contours processing  Approximation  Hierarchical representation  Shape characteristics  Matching  Geometry  Contour properties  Fitting with primitives  PGH: pair-wise geometrical histogram for the contour.

Contour Processing  Approximation: RLE algorithm (chain code) Teh-Chin approximation (polygonal) Douglas-Peucker approximation (polygonal);  Contour moments (central and normalized up to order 3)  Hierarchical representation of contours  Matching of contours

Hierarchical Representation of Contours  A contour is represented with a binary tree  Given the binary tree, the contour can be retrieved with arbitrary precision  The binary tree is quasi invariant to translations, rotations and scaling

Contours matching  Matching based on hierarchical representation of contours

Geometry  Properties of contours: (perimeter, area, convex hull, convexity defects, rectangle of minimum area)  Fitting: (2D line, 3D line, circle, ellipse)  Pair-wise geometrical histogram

Pair-wise geometrical histogram (PGH)

Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

Object Recognition  Eigen objects  Hidden Markov Models

Eigen Objects

Eigen objects (continued)

Hidden Markov Model Definitions - The set of states - The set of measurements - The state at time t - The transition probability matrix - The conditional probability matrix - The starting states distribution

Embedded HMM for Face Recognition Model- - Face ROI partition

Face recognition using Hidden Markov Models One person – one HMM Stage 1 – Train every HMM Stage 2 – Recognition P i - probability Choose max(P i ) … 1 n i

Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

Motion Analysis and Object Tracking  Background subtraction  Motion templates  Optical flow  Active contours  Estimators

Background Subtraction  Background model (normal distribution)  Background statistics functions: Average Standard deviation Running average

Motion Templates  Object silhouette  Motion history images  Motion history gradients  Motion segmentation algorithm silhouetteMHI MHG

Motion Segmentation Algorithm  Two-pass algorithm labeling all motion segments

Motion Templates Example  Motion templates allow to retrieve the dynamic characteristics of the moving object

Optical Flow  Block matching technique  Horn & Schunck technique  Lucas & Kanade technique  Pyramidal LK algorithm  6DOF (6 degree of freedom) algorithm Optical flow equations:

Pyramidal Implementation of the optical flow algorithm J imageI image Image Pyramid Representation Iterative Lucas – Kanade Scheme Generic Image (L-1)-th Level L-th Level Location of point u on image u L =u/2 L Spatial gradient matrix Standard Lucas – Kanade scheme for optical flow computation at level L d L Guess for next pyramid level L – 1 Finally, Image pyramid building Optical flow computation

6DOF Algorithm Parametrical optical flow equations:

Active Contours  Snake energy:  Internal energy:  External energy:  Two external energy types:

Estimators  Kalman filter  ConDensation filter

Kalman object tracker

Outline  Image Analysis  Structural Analysis  Object Recognition  Motion Analysis and Object Tracking  3D Reconstruction

3D reconstruction  Camera Calibration  View Morphing  POSIT

Camera Calibration  Define intrinsic and extrinsic camera parameters.  Define Distortion parameters

Camera Calibration Now, camera calibration can be done by holding checkerboard in front of the camera for a few seconds. And after that you ’ ll get: 3D view of etalon Un-distorted image

View Morphing

POSIT Algorithm  Perspective projection: Weak-perspective projection:

OpenCV Websites  OpenCV official webpage.  OpenCV documentation and FAQs.

OpenCV Examples  adaptiveskindetector : detect skin area  fback_c : dense Franeback optical flow  contours : calculate contours on different levels  delaunay : delaunay triangle  find_obj : SURF Detector and Descriptor using either FLANN or brute force matching on planar objects  morphology : open/close, erode/dilate  motempl ： motion templates  mser_sample : Maximal Extremal Region interest point detector  polar_transforms : illustrates Linear-Polar and Log-Polar image transforms  pyramid_segmentation : color pyramid segmentation

Thanks !!!