Download presentation

Presentation is loading. Please wait.

1
CSCE 641 Computer Graphics: Image-based Modeling Jinxiang Chai

2
Image-based modeling Estimating 3D structure Estimating motion, e.g., camera motion Estimating lighting Estimating surface model

3
Traditional modeling and rendering User input Texture map survey data Geometry Reflectance Light source Camera model Images modeling rendering For photorealism: - Modeling is hard - Rendering is slow

4
Can we model and render this? What do we want to do for this model?

5
Image based modeling and rendering Images user input range scans ModelImages Image-based modeling Image-based rendering

6
Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

7
Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

8
Spectrum of IBMR Images user input range scans Model Images Image based modeling Image- based rendering Geometry+ Images Geometry+ Materials Images + Depth Light field Panoroma Kinematics Dynamics Etc. Camera + geometry

9
Stereo reconstruction Given two or more images of the same scene or object, compute a representation of its shape How can we estimate camera parameters? knowncameraviewpoints

10
How can we estimate the camera parameters?

11
Camera calibration Augmented pin-hole camera - focal point, orientation - focal length, aspect ratio, center, lens distortion Known 3D Classical calibration - 3D 2D - correspondence Camera calibration online resources

12
Camera and calibration target

13
Classical camera calibration Known 3D coordinates and 2D coordinates - known 3D points on calibration targets - find corresponding 2D points in image using feature detection algorithm

14
Camera parameters u0u0 v0v0 100 -s y 0 sxsx а u v 1 Perspective proj. View trans. Viewport proj. Known 3D coords and 2D coords

15
Camera parameters u0u0 v0v0 100 -s y 0 sxsx а u v 1 Perspective proj. View trans. Viewport proj. Known 3D coords and 2D coords Intrinsic camera parameters (5 parameters) extrinsic camera parameters (6 parameters)

16
Camera matrix Fold intrinsic calibration matrix K and extrinsic pose parameters (R,t) together into a camera matrix M = K [R | t ] (put 1 in lower r.h. corner for 11 d.o.f.)

17
Camera matrix calibration Directly estimate 11 unknowns in the M matrix using known 3D points (X i,Y i,Z i ) and measured feature positions (u i,v i )

18
Camera matrix calibration Linear regression: Bring denominator over, solve set of (over-determined) linear equations. How?

19
Camera matrix calibration Linear regression: Bring denominator over, solve set of (over-determined) linear equations. How? Least squares (pseudo-inverse) - 11 unknowns (up to scale) - 2 equations per point (homogeneous coordinates) - 6 points are sufficient

20
Nonlinear camera calibration Perspective projection:

21
Nonlinear camera calibration Perspective projection: KRTP

22
Nonlinear camera calibration Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: KRTP

23
Nonlinear camera calibration Perspective projection: 2D coordinates are just a nonlinear function of its 3D coordinates and camera parameters: KRTP

24
Multiple calibration images Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N This can be formulated as a nonlinear optimization problem:

25
Multiple calibration images Find camera parameters which satisfy the constraints from M images, N points: for j=1,…,M for i=1,…,N This can be formulated as a nonlinear optimization problem: Solve the optimization using nonlinear optimization techniques: - Gauss-newton - Levenberg-MarquardtLevenberg-Marquardt

26
Nonlinear approach Advantages: can solve for more than one camera pose at a time fewer degrees of freedom than linear approach Standard technique in photogrammetry, computer vision, computer graphics - [Tsai 87] also estimates lens distortions (freeware @ CMU) http://www.cs.cmu.edu/afs/cs/project/cil/ftp/html/v-source.html Disadvantages: more complex update rules need a good initialization (recover K [R | t] from M)

27
Camera Calibration Public calibration toolbox - http://research.microsoft.com/en-us/um/people/zhang/Calib/http://research.microsoft.com/en-us/um/people/zhang/Calib/ - http://www.vision.caltech.edu/bouguetj/calib_doc/http://www.vision.caltech.edu/bouguetj/calib_doc/

28
How can we estimate the camera parameters?

29
Application: camera calibration for sports video [Farin et. Al] imagesCourt model

30
Stereo matching Given two or more images of the same scene or object as well as their camera parameters, how to compute a representation of its shape? What are some possible representations for shapes? depth maps volumetric models 3D surface models planar (or offset) layers

31
Outline Stereo matching - Traditional stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving

32
Stereo matching 11.1, 11.2,.11.3,11.5 in Sezliski book D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1/2/3):7- 42, April-June 2002.A taxonomy and evaluation of dense two-frame stereo correspondence algorithms Readings

33
Stereo scene point optical center image plane

34
Stereo Basic Principle: Triangulation Gives reconstruction as intersection of two rays Requires >calibration >point correspondence

35
Stereo correspondence Determine Pixel Correspondence Pairs of points that correspond to same scene point Epipolar Constraint Reduces correspondence problem to 1D search along conjugate epipolar lines Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html epipolar line epipolar plane

36
Stereo image rectification

37
reproject image planes onto a common plane parallel to the line between optical centers pixel motion is horizontal after this transformation two homographies (3x3 transform), one for each input image reprojection C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.Computing Rectifying Homographies for Stereo Vision

38
Rectification Original image pairs Rectified image pairs

39
Stereo matching algorithms Match Pixels in Conjugate Epipolar Lines Assume brightness constancy This is a tough problem Numerous approaches >A good survey and evaluation: http://www.middlebury.edu/stereo/ http://www.middlebury.edu/stereo/

40
Your basic stereo algorithm For each epipolar line For each pixel in the left image compare with every pixel on same epipolar line in right image pick pixel with minimum matching cost Improvement: match windows This should look familiar.. (normalized cross correlation or SSD) Can use Lukas-Kanade or discrete search (latter more common)

41
Window size Smaller window + - Larger window + - W = 3W = 20 Effect of window size

42
More constraints? We can enforce more constraints to reduce matching ambiguity - smoothness constraints: computed disparity at a pixel should be consistent with neighbors in a surrounding window. - uniqueness constraints: the matching needs to be bijective - ordering constraints: e.g., computed disparity at a pixel should not be larger than the disparity of its right neighbor pixel by more than one pixel.

43
Stereo results Ground truthScene Data from University of Tsukuba Similar results on other images without ground truth

44
Results with window search Window-based matching (best window size) Ground truth

45
Better methods exist... A better method Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Vision, September 1999. Ground truth

46
More recent development High-Quality Single-Shot Capture of Facial Geometry [siggraph 2010, project website]project website - capture high-fidelity facial geometry from multiple cameras - pairwise stereo reconstruction between neighboring cameras - hallucinate facial details

47
More recent development High Resolution Passive Facial Performance Capture [siggraph 2010, project website]project website - capture dynamic facial geometry from multiple video cameras - spatial stereo reconstruction for every frame - building temporal correspondences across the entire sequence

48
Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth

49
Camera calibration errors Poor image resolution Occlusions Violations of brightness constancy (specular reflections) Large motions Low-contrast image regions Stereo reconstruction pipeline Steps Calibrate cameras Rectify images Compute disparity Estimate depth What will cause errors?

50
Outline Stereo matching - Traditional stereo - Active stereo Volumetric stereo - Visual hull - Voxel coloring - Space carving

51
Active stereo with structured light Project “structured” light patterns onto the object simplifies the correspondence problem camera 2 camera 1 projector camera 1 projector Li Zhang’s one-shot stereo

52
Active stereo with structured light

53
Laser scanning Optical triangulation Project a single stripe of laser light Scan it across the surface of the object This is a very precise version of structured light scanning Digital Michelangelo Project http://graphics.stanford.edu/projects/mich/

54
Laser scanned models The Digital Michelangelo Project, Levoy et al.

55
Laser scanned models The Digital Michelangelo Project, Levoy et al.

56
RGBD Sensors 2008 2010 2013 Lower size/cost with better accuracy

57
Kinect Sensor

Similar presentations

© 2019 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google