1. 3D Computer Vision and Video Computing Review Midterm Review CSC I6716 Spring 2003 Zhigang Zhu, NAC 8/203A http://www-cs.engr.ccny.cuny.edu/~zhu/VisionCourse-I6716.html

2. 3D Computer Vision and Video Computing Course Outline n Complete syllabus on the web pages (12-13 lectures) n Rough Outline ( 3D Computer Vision and Video Computing): Part 1. Vision Basics 1. Introduction 2. Sensors 3. Image Formation and Processing ((hw 1, matlab) 4. Features and Feature Extraction (2 lectures, hw 2) Part 2. 3D Vision 5. Camera Models 6. Camera Calibration (hw 3) 7. Stereo Vision (project assignment) 8. Visual Motion (midterm exam) Part 3. Video Computing 9. Video Mosaicing 10. Omnidirectional Stereo 11. Human Tracking 12. Applications (Image-Based Rendering, Video-Coding:MPEG 7, etc.)

3. 3D Computer Vision and Video Computing 1. Introduction: Goals n What is Computer Vision (bigger picture – Part 1)? l Goals l Approaches n What Makes (3D) Computer Vision Interesting (Parts 2 &3) ? l Image Modeling/Analysis/Interpretation n Interpretation is an Artificial Intelligence Problem n Sources of Knowledge in Vision n Levels of Abstraction n Interpretation often goes from 2D images to 3D structures n since we live in a 3D world l Image Rendering/Synthesis/Composition n Image Rendering is a Computer Graphics problem n Rendering is from 3D model to 2D images

4. 3D Computer Vision and Video Computing Related Fields n Image Processing: image to image n Computer Vision: Image to model n Computer Graphics: model to image n Pattern Recognition: image to class l image data mining/ video mining n Artificial Intelligence: machine smarts n Photogrammetry: camera geometry, 3D reconstruction n Medical Imaging: CAT, MRI, 3D reconstruction (2 nd meaning) n Video Coding: encoding/decoding, compression, transmission n Physics: basics n Mathematics: basics n Neuroscience: wetware to concept n Computer Science: programming tools and skills? All three are interrelated! AI Applications basics

5. 3D Computer Vision and Video Computing Applications n Visual Inspection (*) n Robotics (*) n Intelligent Image Tools n Image Compression (MPEG 1/2/4/7) n Document Analysis (OCR) n Image Libraries (DL) n Virtual Environment Construction (*) n Environment (*) n Media and Entertainment n Medicine n Astronomy n Law Enforcement (*) l surveillance, security n Traffic and Transportation (*) n Tele-Conferencing and e-Learning (*)

6. 3D Computer Vision and Video Computing 2. Sensors n Static monocular reflectance data l Film l Video cameras l Digital cameras n Motion sequences (camcorders) n Stereo (2 cameras) n Range data (Range finder) n Non-visual sensory data l infrared (IR) l ultraviolet (UV) l microwaves n Many more

7. 3D Computer Vision and Video Computing The Electromagnetic Spectrum Visible Spectrum 700 nm 400 nm

8. 3D Computer Vision and Video Computing 3. Image Formations n Light and Optics l Pinhole camera model l Perspective projection l Thin lens model l Fundamental equation l Distortion: spherical & chromatic aberration, radial distortion (*option) l Reflection and Illumination: color, lambertian and specular surfaces, Phong, BDRF (*option) n Sensing Light n Conversion to Digital Images n Sampling Theorem n Other Sensors: frequency, type, ….

9. 3D Computer Vision and Video Computing 4. Feature Extraction n Image Enhancement l Brightness mapping l Contrast stretching/enhancement l Histogram modification l Noise Reduction l ……... n Mathematical Techniques l Convolution l Gaussian Filtering n Edge and Line Detection and Extraction n Region Segmentation n Contour Extraction n Corner Detection

10. 3D Computer Vision and Video Computing Edgels n Define a local edge or edgel to be a rapid change in the image function over a small area l implies that edgels should be detectable over a local neighborhood n Edgels are NOT contours, boundaries, or lines l edgels may lend support to the existence of those structures l these structures are typically constructed from edgels n Edgels have properties l Orientation l Magnitude l Length (typically a unit length)

11. 3D Computer Vision and Video Computing Edge Detection n First order edge detectors (lecture - required) l Mathematics l 1x2, Roberts, Sobel, Prewitt n Canny edge detector (after-class reading) n Second order edge detector (after-class reading) l (Laplacian, LOG / DOG n Hough Transform – detect by voting l Lines l Circles l Other shapes

12. 3D Computer Vision and Video Computing Edge Detection: Typical n Noise Smoothing l Suppress as much noise as possible while retaining ‘true’ edges l In the absence of other information, assume ‘white’ noise with a Gaussian distribution n Edge Enhancement l Design a filter that responds to edges; filter output high are edge pixels and low elsewhere n Edge Localization l Determine which edge pixels should be discarded as noise and which should be retained n thin wide edges to 1-pixel width (nonmaximum suppression) establish minimum value to declare a local maximum from edge filter to be an edge (thresholding)

13. 3D Computer Vision and Video Computing Part 2. 3D Vision n Closely Related Disciplines l Image processing – image to mage l Pattern recognition – image to classes l Photogrammetry – obtaining accurate measurements from images n What is 3-D ( three dimensional) Vision? l Motivation: making computers see (the 3D world as humans do) l Computer Vision: 2D images to 3D structure l Applications : robotics / VR /Image-based rendering/ 3D video n Lectures on 3-D Vision Fundamentals (Part 2) l Camera Geometric Model (2 – this class- topic 5) l Camera Calibration (2 – topic 6) l Stereo (2 – topic 7) l Motion (2 –topic 8)

14. 3D Computer Vision and Video Computing 5. Camera Models n Geometric Projection of a Camera l Pinhole camera model l Perspective projection l Weak-Perspective Projection n Camera Parameters l Intrinsic Parameters: define mapping from 3D to 2D l Extrinsic parameters: define viewpoint and viewing direction n Basic Vector and Matrix Operations, Rotation n Camera Models Revisited l Linear Version of the Projection Transformation Equation n Perspective Camera Model n Weak-Perspective Camera Model n Affine Camera Model n Camera Model for Planes n Summary

15. 3D Computer Vision and Video Computing 6. Camera Calibration n Calibration: Find the intrinsic and extrinsic parameters l Problem and assumptions l Direct parameter estimation approach l Projection matrix approach n Direct Parameter Estimation Approach l Basic equations (from Lecture 5) l Estimating the Image center using vanishing points- Orthocenter Theorem l SVD (Singular Value Decomposition) and Homogeneous System l Focal length, Aspect ratio, and extrinsic parameters l Discussion: Why not do all the parameters together? n Projection Matrix Approach (…after-class reading) l Estimating the projection matrix M l Computing the camera parameters from M l Discussion n Comparison and Summary l Any difference?

16. 3D Computer Vision and Video Computing 7. Stereo Vision n Problem l Infer 3D structure of a scene from two or more images taken from different viewpoints n Two primary Sub-problems l Correspondence problem (stereo match) -> disparity map n Similarity instead of identity n Occlusion problem: some parts of the scene are visible in one eye only l Reconstruction problem -> 3D n What we need to know about the cameras’ parameters n Often a stereo calibration problems n Lectures on Stereo Vision l Stereo Geometry – Epipolar Geometry (*) l Correspondence Problem (*) – Two classes of approaches l 3D Reconstruction Problems – Three approaches

17. 3D Computer Vision and Video Computing Stereo Vision n Epipolar Geometry l Where to search correspondences l Epipolar plane, epipolar lines and epipoles l Essential matrix and fundamental matrix n Correspondence Problem l Correlation-based approach l Feature-based approach n 3D Reconstruction Problem l Both intrinsic and extrinsic parameters are known l Only intrinsic parameters l No prior knowledge of the cameras (* option)

18. 3D Computer Vision and Video Computing 8. Motion n Problems and Applications (Topic 8 Motion I) l The importance of visual motion l Problem Statement n The Motion Field of Rigid Motion (Topic 8 Motion I) l Basics – Notations and Equations l Three Important Special Cases: Translation, Rotation and Moving Plane l Motion Parallax n Optical Flow (Topic 8 Motion II) l Optical flow equation and the aperture problem l Estimating optical flow l 3D motion & structure from optical flow n Feature-based Approach (Topic 8 Motion II) l Two-frame algorithm l Multi-frame algorithm l Structure from motion – Factorization method (* option) n Advanced Topics (Topic 8 Motion II; Part 3 – not yet covered!) l Spatio-Temporal Image and Epipolar Plane Image l Video Mosaicing and Panorama Generation l Motion-based Segmentation and Layered Representation

19. 3D Computer Vision and Video Computing Types of questions n Multiple choices (50%) n Short questions, proofs, and simple analysis (50%) n Exam Time: April 8 th, 2 hours (1:50 pm – 3:50 pm) n After Exam ; Project AllocationsProject Allocations l About 2-3 students form a team l One single project, but separate reports l Send me email on your choices of projects / and teams