1. 3D Computer Vision and Video Computing Review Midterm Review CSC I6716 Spring 2011 Prof. Zhigang Zhu http://www-cs.engr.ccny.cuny.edu/~zhu/

2. 3D Computer Vision and Video Computing Course Outline n Complete syllabus on the web pages (14 meets) n Rough Outline ( 3D Computer Vision): Part 1. Vision Basics (Total 4) 1. Introduction (1) 2. Image Formation and Processing (1) (hw 1, matlab) 3-4. Features and Feature Extraction (2) ( hw 2) Part 2. 3D Vision (Total 7) 5. Camera Models (1) 6. Camera Calibration (2)(hw 3) 7. Stereo Vision (2) (project assignments) 8. Visual Motion (2) (hw 4) Part 3. Exam and Projects (Total 3) 9. Project topics and exam review/discussion (1) 10. Midterm exam (1) 11. Student Project presentations (1)

3. 3D Computer Vision and Video Computing Course Goals and Questions n What makes (3D) Computer Vision interesting ? l Image Modeling/Analysis/Interpretation n Interpretation is an Artificial Intelligence Problem n Sources of Knowledge in Vision n Levels of Abstraction n Interpretation often goes from 2D images to 3D structures n since we live in a 3D world l Image Rendering/Synthesis/Composition n Image Rendering is a Computer Graphics problem n Rendering is from 3D model to 2D images 2D images 3D world CVCG

4. 3D Computer Vision and Video Computing Related Fields n Image Processing: image to image n Computer Vision: Image to model n Computer Graphics: model to image n Pattern Recognition: image to class l image data mining/ video mining n Artificial Intelligence: machine smarts n Photogrammetry: camera geometry, 3D reconstruction n Medical Imaging: CAT, MRI, 3D reconstruction (2 nd meaning) n Video Coding: encoding/decoding, compression, transmission n Physics: basics n Mathematics: basics n Neuroscience: wetware to concept n Computer Science: programming tools and skills? All three are interrelated! AI Applications basics

5. 3D Computer Vision and Video Computing Applications n Visual Inspection (*) n Robotics (*) n Intelligent Image Tools n Image Compression (MPEG 1/2/4/7) n Document Analysis (OCR) n Image Libraries (DL) n Virtual Environment Construction (*) n Environment (*) n Media and Entertainment n Medicine n Astronomy n Law Enforcement (*) l surveillance, security n Traffic and Transportation (*) n Tele-Conferencing and e-Learning (*)

6. 3D Computer Vision and Video Computing 2. Image Formations n Light and Optics l Pinhole camera model l Perspective projection l Thin lens model l Fundamental equation l Distortion: spherical & chromatic aberration, radial distortion (*option) n Sensing Light n Conversion to Digital Images n Sampling Theorem n Other Sensors: frequency, type, ….

7. 3D Computer Vision and Video Computing 3&4. Feature Extraction n Image Enhancement l Brightness mapping l Contrast stretching/enhancement l Histogram modification l Noise Reduction l ……... n Mathematical Techniques l Convolution l Gaussian Filtering n Edge and Line Detection and Extraction n Region Segmentation n Contour Extraction n Corner Detection

8. 3D Computer Vision and Video Computing Edgels n Define a local edge or edgel to be a rapid change in the image function over a small area l implies that edgels should be detectable over a local neighborhood n Edgels are NOT contours, boundaries, or lines l edgels may lend support to the existence of those structures l these structures are typically constructed from edgels n Edgels have properties l Orientation l Magnitude l Length (typically a unit length)

9. 3D Computer Vision and Video Computing Edge Detection n First order edge detectors (lecture - required) l Mathematics l 1x2, Roberts, Sobel, Prewitt n Canny edge detector (after-class reading) n Second order edge detector (after-class reading) l (Laplacian, LOG / DOG n Hough Transform – detect by voting l Lines l Circles l Other shapes

10. 3D Computer Vision and Video Computing Edge Detection: Typical n Noise Smoothing l Suppress as much noise as possible while retaining ‘true’ edges l In the absence of other information, assume ‘white’ noise with a Gaussian distribution n Edge Enhancement l Design a filter that responds to edges; filter output high are edge pixels and low elsewhere n Edge Localization l Determine which edge pixels should be discarded as noise and which should be retained n thin wide edges to 1-pixel width (nonmaximum suppression) establish minimum value to declare a local maximum from edge filter to be an edge (thresholding)

11. 3D Computer Vision and Video Computing 5. Camera Models n Geometric Projection of a Camera l Pinhole camera model l Perspective projection l Weak-Perspective Projection n Camera Parameters l Intrinsic Parameters: define mapping from 3D to 2D l Extrinsic parameters: define viewpoint and viewing direction n Basic Vector and Matrix Operations, Rotation n Camera Models Revisited l Linear Version of the Projection Transformation Equation n Perspective Camera Model n Weak-Perspective Camera Model n Affine Camera Model n Camera Model for Planes n Summary

12. 3D Computer Vision and Video Computing 6. Camera Calibration n Calibration: Find the intrinsic and extrinsic parameters l Problem and assumptions l Direct parameter estimation approach l Projection matrix approach n Direct Parameter Estimation Approach l Basic equations (from Lecture 5) l Estimating the Image center using vanishing points- Orthocenter Theorem l SVD (Singular Value Decomposition) and Homogeneous System l Focal length, Aspect ratio, and extrinsic parameters l Discussion: Why not do all the parameters together? n Projection Matrix Approach l Estimating the projection matrix M l Computing the camera parameters from M l Discussion n Comparison and Summary

13. 3D Computer Vision and Video Computing 7. Stereo Vision n Problem l Infer 3D structure of a scene from two or more images taken from different viewpoints n Two primary Sub-problems l Correspondence problem (stereo match) -> disparity map n Similarity instead of identity n Occlusion problem: some parts of the scene are visible in one eye only l Reconstruction problem -> 3D n What we need to know about the cameras’ parameters n Often a stereo calibration problems n Lectures on Stereo Vision l Stereo Geometry – Epipolar Geometry (*) l Correspondence Problem (*) – Two classes of approaches l 3D Reconstruction Problems – Three approaches

14. 3D Computer Vision and Video Computing Stereo Vision n Epipolar Geometry l Where to search correspondences l Epipolar plane, epipolar lines and epipoles l Essential matrix and fundamental matrix n Correspondence Problem l Correlation-based approach l Feature-based approach n 3D Reconstruction Problem l Both intrinsic and extrinsic parameters are known l Only intrinsic parameters l No prior knowledge of the cameras (* option)

15. 3D Computer Vision and Video Computing 8. Motion n Problems and Applications l The importance of visual motion l Problem Statement n The Motion Field of Rigid Motion l Basics – Notations and Equations l Three Important Special Cases: Translation, Rotation and Moving Plane l Motion Parallax n Optical Flow l Optical flow equation and the aperture problem l Estimating optical flow l 3D motion & structure from optical flow n Feature-based Approach l Two-frame algorithm l Multi-frame algorithm l Structure from motion – Factorization method (* option) n Advanced Topics l Spatio-Temporal Image and Epipolar Plane Image l Video Mosaicing and Panorama Generation l Motion-based Segmentation and Layered Representation

16. 3D Computer Vision and Video Computing Types of questions n Multiple choices (30) n Short questions, proofs, and simple analysis (70%) n Exam Time: l May 10, 90 minutes (7:30 pm – 9:00 pm)