1. Code Development for Processing Video Streams
“NOT reinventing the wheel”
Slides taken from a presentation by Dr. Gary Bradski,
Intel Computer Corp.

2. Progress in Computer Vision
1970’s vision student
First step: Build digitizer and video display hardware
1980’s vision student
First step: Write image class library and interface to video capture card
2000’s vision student
First step: Learn to use standard libraries with standard hardware!

3. Standard Libraries IPL OpenCV VIPeR Toolkit VisSDK
Link from network file system
Download to your local disk
IPL
OpenCV
VIPeR Toolkit
An example program
running under Windows
which can be customized
VisSDK

4. Standard Vision Libraries
Intel IPL (now IPP) – Image processing library
Optimized (MMX) code for filtering, FFT, etc.
Intel OpenCV – Open Computer Vision library
Higher level support for image representation and manipulation
Microsoft Vision SDK – Similar to OpenCV
Includes standard routines for image and video input and output
Otherwise, not used very much here

5. VIPER
This is a starter project in Visual C++ which integrates the standard libraries. It includes
Examples of reading and writing images and video files
Examples of capturing live video and processing it
Some simple rendering and interaction hooks under Windows
Used extensively in Digital Video Special Effects class for several years with great success
It is not elegant and beautiful code.
It is a quick way to implement experiments.

6. First Problem Set
It will give you a chance to use Viper with standard libraries to do some simple processing
Two parts:
Read in an image and compute its histogram
Read in an AVI file and do some simple “tracking”
Instructions for installing VIPER and connecting to standard libraries will be provided.
Tutorial on VIPER, Visual C++, and histograms.
This Friday from 4-5. Location: CCB 155

7. SDKs Galore!
IPL
OpenCV
VIPeR Toolkit
VisSDK

8. Image Analysis Thresholds Statistics Pyramids Morphology
Distance transform
Flood fill
Feature detection
Contours retrieving

9. Image Thresholding
Fixed threshold;
Adaptive threshold;

10. Image Thresholding Examples
Source picture
Fixed threshold
Adaptive threshold

11. Statistics min, max, mean value, standard deviation over the image
Norms C, L1, L2
Multidimensional histograms
Spatial moments up to order 3 (central, normalized, Hu)

12. Image Pyramids Gaussian and Laplacian pyramids
Image segmentation by pyramids
Change the picture to something more clear!

13. Image Pyramids
Gaussian and Laplacian

14. Pyramid-based color segmentation
On still pictures
And on movies

15. Morphological Operations
Two basic morphology operations using structuring element:
erosion
dilation
More complex morphology operations:
opening
closing
morphological gradient
top hat
black hat

16. What is Structuring Element?
The rectangle mask with given position of a principal pixel:

17. Morphological Operations Examples
Morphology - applying Min-Max. Filters and its combinations
Image I
Erosion IB
Dilatation IB
Opening IoB= (IB)B
Closing I•B= (IB)B
Grad(I)= (IB)-(IB)
TopHat(I)= I - (IB)
BlackHat(I)= (IB) - I

18. Distance Transform
Calculate the distance for all non-feature points to the closest feature point
Two-pass algorithm, 3x3 and 5x5 masks, various metrics predefined

19. Flood Filling
Simple
Gradient

20. Feature Detection Fixed filters (Sobel operator, Laplacian);
Optimal filter kernels with floating point coefficients (first, second derivatives, Laplacian)
Special feature detection (corners)
Canny operator
Hough transform (find lines and line segments)
Gradient runs

21. Canny Edge Detector

22. Hough Transform Detects lines in a binary image
Probabilistic Hough Transform
Standard Hough Transform

23. Another Sample of the Hough Transform Using
Source picture
Result

24. Contour Retrieving The contour representation:
Chain code (Freeman code)
Polygonal representation
Initial Point
Chain code for the curve:
Contour representation

25. Hierarchical representation of contours
Image Boundary
(W1)
(W2)
(W3)
(B2)
(B3)
(B4)
Get the english picture!
(W5)
(W6)

26. Contours Examples Source Picture (300x600 = 180000 pts total)
Retrieved Contours
(<1800 pts total)
After Approximation
(<180 pts total)
And it is rather fast: ~70 FPS for 640x480 on complex scenes

27. Optical Flow Block matching technique Horn & Schunck technique
Lucas & Kanade technique
Pyramidal LK algorithm
6DOF (6 degree of freedom) algorithm
Optical flow equations:

28. Pyramidal Implementation of the optical flow algorithm
Image Pyramid Representation
Iterative Lucas – Kanade Scheme
J image
I image
Location of point u on image uL=u/2L
Spatial gradient matrix
Standard Lucas – Kanade scheme for optical flow computation at level L dL
Guess for next pyramid level L – 1
Finally,
Generic Image
(L-1)-th Level
Image pyramid building
L-th Level
Optical flow computation

29. Pyramidal Implementation of the optical flow algorithm

30. 6DOF Algorithm Parametrical optical flow equations:
Reconsider this slide!

31. 3D reconstruction
Camera Calibration
View Morphing
POSIT

32. Camera Calibration Define intrinsic and extrinsic camera parameters.
Define Distortion parameters

33. Camera Calibration
Now, camera calibration can be done by holding checkerboard in front of the camera for a few seconds.
And after that you’ll get:
3D view of etalon
Un-distorted image

34. View Morphing

35. POSIT Algorithm
Perspective projection:
Weak-perspective projection:

36. References
M.Hu.Visual Pattern Recognition by Moment Invariants, IRE Transactions on Information Theory, 8:2, pp , 1962.
B. Jahne. Digital Image Processing. Springer, New York, 1997.
M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active Contour Models, International Journal of Computer Vision, pp , 1988.
J.Matas, C.Galambos, J.Kittler. Progressive Probabilistic Hough Transform. British Machine Vision Conference, 1998.
A. Rosenfeld and E. Johnston. Angle Detection on Digital Curves. IEEE Trans. Computers, 22: , 1973.
Y.Rubner.C.Tomasi,L.J.Guibas.Metrics for Distributions with Applications to Image Databases. Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay, India, January 1998, pp
Y. Rubner. C. Tomasi, L.J. Guibas. The Earth Mover’s Distance as a Metric for Image Retrieval. Technical Report STAN-CS-TN-98-86, Department of Computer Science, Stanford University, September, 1998.
Y.Rubner.C.Tomasi.Texture Metrics. Proceeding of the IEEE International Conference on Systems, Man, and Cybernetics, San-Diego, CA, October 1998, pp

37. References
Gunilla Borgefors. Distance Transformations in Digital Images.Computer Vision, Graphics and Image Processing 34, ,(1986).
G. Bradski and J. Davis. Motion Segmentation and Pose Recognition with Motion History Gradients. IEEE WACV'00, 2000.
P. J. Burt, T. H. Hong, A. Rosenfeld. Segmentation and Estimation of Image Region Properties Through Cooperative Hierarchical Computation. IEEE Tran. On SMC, Vol. 11, N.12, 1981, pp
J.Canny.A Computational Approach to Edge Detection, IEEE Trans. on Pattern Analysis and Machine Intelligence, 8(6), pp (1986).
J. Davis and Bobick. The Representation and Recognition of Action Using Temporal Templates. MIT Media Lab Technical Report 402,1997.
Daniel F. DeMenthon and Larry S. Davis. Model-Based Object Pose in 25 Lines of Code. In Proceedings of ECCV '92, pp , 1992.
Andrew W. Fitzgibbon, R.B.Fisher. A Buyer’s Guide to Conic Fitting.Proc.5 th British Machine Vision Conference, Birmingham, pp , 1995.
Berthold K.P. Horn and Brian G. Schunck. Determining Optical Flow. Artificial Intelligence, 17, pp , 1981.

38. References
J. Serra. Image Analysis and Mathematical Morphology. Academic Press, 1982.
Bernt Schiele and James L. Crowley. Recognition without Correspondence Using Multidimensional Receptive Field Histograms. In International Journal of Computer Vision 36 (1), pp , January 2000.
S. Suzuki, K. Abe. Topological Structural Analysis of Digital Binary Images by Border Following. CVGIP, v.30, n , pp
C.H.Teh, R.T.Chin.On the Detection of Dominant Points on Digital Curves. - IEEE Tr. PAMI, 1989, v.11, No.8, p
Emanuele Trucco, Alessandro Verri. Introductory Techniques for 3-D Computer Vision. Prentice Hall, Inc., 1998.
D. J. Williams and M. Shah. A Fast Algorithm for Active Contours and Curvature Estimation. CVGIP: Image Understanding, Vol. 55, No. 1, pp , Jan.,
A.Y.Yuille, D.S.Cohen, and P.W.Hallinan. Feature Extraction from Faces Using Deformable Templates in CVPR, pp , 1989.
Zhengyou Zhang. Parameter Estimation Techniques: A Tutorial with Application to Conic Fitting, Image and Vision Computing Journal, 1996.