Presentation is loading. Please wait.

Presentation is loading. Please wait.

Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Seales, Henry Fuchs University of North Carolina at Chapel.

Similar presentations


Presentation on theme: "Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Seales, Henry Fuchs University of North Carolina at Chapel."— Presentation transcript:

1 Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Seales, Henry Fuchs University of North Carolina at Chapel Hill Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Seales, Henry Fuchs University of North Carolina at Chapel Hill Multi-Projector Displays using Camera-based Registration

2 Multi-Projector Panoramic Displays Traditional Display Setups

3 Multi-Projector Panoramic Displays Traditional Display Setups Precise Geometry, Well-defined overlaps

4 Generalized Panoramic Display Flexible Display Setups Casually aligned projectors and screens

5 Problem : Traditionally rigid designTraditionally rigid design –Painstaking construction  Room must be modified  Projector alignment is tedious –High maintenance cost –Constant adjustments  Projectors become “unaligned” over time

6 Problem : Instant Panoramic Display Generalized display environmentGeneralized display environment Setup display surface approximatelySetup display surface approximately Align projectors casuallyAlign projectors casually Render with what you haveRender with what you have

7 Multi-Projector Goal Seamless imagerySeamless imagery Immersive 3D images, moving userImmersive 3D images, moving user Casually setup projectors and surfacesCasually setup projectors and surfaces Low maintenanceLow maintenance Auto-CalibrationAuto-Calibration Perceptually single logical projectorPerceptually single logical projector

8 Goal : Seamless Display (a) Projector Registration –Geometric alignment (b) Intensity blending –Smooth transition Projection 1 Projection 2

9 Goal : Seamless Display (a) Projector Registration –Geometric alignment (b) Intensity blending –Smooth transition

10 Goal : Seamless Display (a) Projector Registration –Geometric alignment (b) Intensity blending –Smooth transition

11 Goal : Seamless Display (a) Projector Registration –Geometric alignment (b) Intensity blending –Smooth transition (a) Projector Registration –Geometric alignment (b) Intensity blending –Smooth transition

12 Goal : Seamless Display (a) Projector Registration –Geometric alignment (b) Intensity blending –Smooth transition

13 Previous approaches Multi-projector setups Sweet spotSweet spot –Static user Moving UserMoving User –Head-tracked user

14 Previous approaches: Sweet spot (static user) Multiple projectorsMultiple projectors –(well-defined overlaps) –Flight sims, –Panoram, Trimensions –[Raskar98] Single wide-field-of-view projectorSingle wide-field-of-view projector –OmniMax –Alternate Realities User

15 Previous approaches: Moving user CAVECAVE –Rigid setup –Well-defined geometry –Large support structure Projector User

16 Previous approaches: Moving user Office of the Future [Raskar98]Office of the Future [Raskar98] –Relaxed construction constraints –Assumes precise display geometry recovery –Demonstrated for a single projector

17 Related work: Image Mosaicing QuickTime VRQuickTime VR Pair-wise mosaicing of photographsPair-wise mosaicing of photographs Generalized mosaicingGeneralized mosaicing

18 Goal for this paper Seamless imageSeamless image Immersive 3D images, moving userImmersive 3D images, moving user Arbitrary projectors and surfacesArbitrary projectors and surfaces Eliminate maintenanceEliminate maintenance –Self-calibration Perceptually single logical projectorPerceptually single logical projector

19 Basic Approach: Panoramic Display Position projectors approximatelyPosition projectors approximately Use video cameras to recoverUse video cameras to recover –display geometry –projector configuration Track userTrack user Render images of 3D scenesRender images of 3D scenes Warp and blend projected images to achieve registrationWarp and blend projected images to achieve registration

20 Outline Single projectorSingle projector –Calibration, Display recovery, Rendering Multiple projectorMultiple projector –Registration, Blending New techniquesNew techniques –Post rendering warp, mesh unification

21 Video

22 Outline Single projectorSingle projector –Rendering, Calibration, Display recovery Multiple projectorMultiple projector –Registration, Blending New techniquesNew techniques –Post rendering warp, mesh unification

23 User Projector 3D Display Surface Model (1) Desired Image (2) Render Model 2-pass Rendering

24 User Projector (1) Desired Image (2) Render Model 2-pass Rendering Project this “warped” image, the viewer will see it correctly 3D Display Surface

25 Single projector display 2-pass unknowns Arbitrary display surfaceArbitrary display surface Arbitrary projector locationArbitrary projector location –Need to determine these for 2 pass rendering Display Surface ? Projector ? User Stereo camera pair - How?

26 Single projector display Camera calibration Display Surface? Projector ? Stereo camera pair 1. Calibrate stereo pair

27 Single projector display Display surface estimation Display Surface? Projector ? Stereo camera pair 1. Calibrate stereo pair 2. Determine display surface

28 Single projector display Projector estimation Display Surface Projector ? Stereo camera pair 1. Calibrate stereo pair 2. Determine display surface 3. Determine projector

29 Single projector display Projector estimation Display Surface Projector Stereo camera pair 1. Calibrate stereo pair 2. Determine display surface 3. Determine projector

30 Display surface is recovered by cameras (via projector)Display surface is recovered by cameras (via projector) –Surface mesh is created from the 3D points –Allows us to calibrate the projector Approximate speedApproximate speed –720 surface point samples (~1 minute) –14,000 surfaces point samples (~15 minutes) Single projector display Display Surface Estimation

31 Single projector display Coordinate Frame Display surface and projector are in the coordinate frame of the “Calibration Pattern”Display surface and projector are in the coordinate frame of the “Calibration Pattern” x y z

32 Multiple Projectors Panoramic display environments A single stereo-pair cannot see entire displayA single stereo-pair cannot see entire display –Must use several stereo-pairs –Each in its own coordinate frame –Must register data into common coordinate frame AdvantageAdvantage –We control the light –Makes registration easier

33 Multiple Projectors Surface mesh registration Display Surface

34 Display surfaces are in different coordinate frames Multiple Projectors Display surfaces are in different coordinate frames Display Surfaces x y z x y z x y z

35 x y z x y z x y z Must register into common coordinate frame Multiple Projectors Must register into common coordinate frame

36 x y z Display Surfaces Surface meshes are registered into the same coordinate frame Multiple Projectors Surface meshes are registered into the same coordinate frame

37 x y z Display Surfaces Re-calibrate the projectors so they are registered into a common coordinate frame Re-calibrate projectors so they are registered Multiple Projectors Re-calibrate projectors so they are registered

38 x y z Display Surfaces With display surfaces and projectors registered we can achieve geometric correctness... Re-calibrate projectors so they are registered Multiple Projectors Re-calibrate projectors so they are registered

39 Multiple Projectors Intensity Blending Need to attenuate intensities in the overlapped region

40 Intensity Blending Use camera to find overlaps Camera observes overlapped region

41 Intensity Blending Assigning intensity weights (alpha) Assign intensity weights based on distance between projector boundaries Camera image plane

42 Intensity Blending Assigning intensity weights (alpha) Assign intensity weights based on distance between projector boundaries Camera image plane

43 Intensity Blending: Alpha-masks Projector alpha-masks

44 Achieving our goal Seamless display Display surface and projector registrationDisplay surface and projector registration –Geometrically alignment imagery Intensity blendingIntensity blending –Smooth intensity transition between projectors

45 Registration Issues Errors in estimationErrors in estimation –user location –3D display surface –projector parameters

46 Registration Issues Errors in estimationErrors in estimation –user location –3D display surface –projector parameters Single projector, errors not visibleSingle projector, errors not visible Multi-projector overlapMulti-projector overlap –Geometric mis-registrations –Gaps, Discontinuities

47 Sources of errors Head-tracker for user locationHead-tracker for user location Display surface recoveryDisplay surface recovery –Camera Calibration  Radial Distortion  Pattern Imperfection –Feature detection Projector calibrationProjector calibration –Relies on display surface recovery

48 Re-projection error M m m' P P’ P m m' Projector framebuffer

49 Sources of re-projection error Minimization residualMinimization residual Projector not pin-hole deviceProjector not pin-hole device Inaccuracies in estimated 3D pts on display surfaceInaccuracies in estimated 3D pts on display surface

50 Effect of errors User locationUser location Display surface meshesDisplay surface meshes  difficult to register  incorrect surface representation Projector parametersProjector parameters  re-projection error  incorrect projection matrix

51 Geometric Error Compensation ObjectiveObjective  Neighboring projectors use same display surface representation  Each projection matrix maps 3D screen pts to 2D pixels that illuminate them Overcoming inaccuraciesOvercoming inaccuracies  Geometric continuity everywhere for display  Fidelity of projector model

52 Outline Single projectorSingle projector –Calibration, Display recovery, Rendering Multiple projectorMultiple projector –Registration, Blending New techniquesNew techniques –Post rendering warp, mesh unification

53 Geometric Error Compensation ObjectiveObjective  Neighboring projectors use same display surface representation Surface Mesh Unification Each projection matrix maps 3D screen pts to 2D pixels that illuminate them Each projection matrix maps 3D screen pts to 2D pixels that illuminate them Post Rendering Warp

54 Surface mesh unification Merge multiple recovered meshesMerge multiple recovered meshes Create a single representation of display surfacesCreate a single representation of display surfaces Rigid transformation creates near alignmentRigid transformation creates near alignment Errors lead to discontinuitiesErrors lead to discontinuities Use smooth 3D transitionUse smooth 3D transition

55 Surface mesh unification x y z x y z P1 P2 Actual surface

56 Surface mesh unification

57

58

59 Surface Unification New surface mesh

60 Surface Unification New surface mesh Original surface Both projectors use same (incorrect) surface geometry

61 Geometric Error Compensation ObjectiveObjective 4 Neighboring projectors use same display surface representation Surface Mesh Unification Ü Each projection matrix maps 3D screen pts to 2D pixels that illuminate them Post Rendering Warp

62 Modified re-projection error M M’ P1 P2 Actual Pt Estimated Pt

63 Modified re-projection error M M’ P1 P2’ P2 P1’

64 Modified re-projection error M M’ P1 P2’ P2 P1’ m1m1 m1m1 m2m2 m2m2

65 Modified re-projection error M M’ m1m1 m 1 ’’ m2m2 m 2 ’’ m1m1 m 1 ’’ m2m2 m 2 ’’ P1 P2’ P2 P1’

66 Effect of reprojection error Physically m1 and m2 create image at MPhysically m1 and m2 create image at M In estimated model, m1’’ and m2’’ correspondIn estimated model, m1’’ and m2’’ correspond Features of a virtual object gets rendered at m1’’ and m2’’Features of a virtual object gets rendered at m1’’ and m2’’ But, m1 and m2 are not likely to correspondBut, m1 and m2 are not likely to correspond

67 Effect of Re-projection error M M’ m1m1 m 1 ’’ m2m2 m 2 ’’ User V V M

68 Eliminating re-projection error We haveWe have  m1’’ = P 1 ’ M’  m2’’ = P 2 ’ M’ m1m1 m 1 ’’ m2m2 m 2 ’’ M V

69 Eliminating re-projection error We haveWe have  m1’’ = P 1 ’ M’  m2’’ = P 2 ’ M’ We need a nonlinear projective functionWe need a nonlinear projective function  m1 =   (M’)  m2 =   (M’) m1m1 m 1 ’’ m2m2 m 2 ’’ M V

70 Eliminating re-projection error We haveWe have  m1’’ = P 1 ’ M’  m2’’ = P 2 ’ M’ We need a nonlinear projective functionWe need a nonlinear projective function  m1 =   (M’)  m2 =   (M’) Cannot be achieved using single projection matrixCannot be achieved using single projection matrix Lack of hardware supportLack of hardware support m1m1 m 1 ’’ m2m2 m 2 ’’ M V

71 Post Rendering warp Goal:  m1 =  1 (M’) m2 =  2 (M’) (a) Render using estimated projection matrix  m1’’ = P 1 ’ M’ m2’’ = P 2 ’ M’ (b) Warp resultant image  m1’’  m1 m2’’  m2

72 Post Rendering warp Goal:  m1 =  1 (M’) m2 =  2 (M’) (a) Render using estimated projection matrix  m1’’ = P 1 ’ M’ m2’’ = P 2 ’ M’ (b) Warp resultant image  m1’’  m1 m2’’  m2 Expensive to specify per-pixel warpExpensive to specify per-pixel warp  Use piecewise planar approximation

73 Post Rendering warp (using texture mapping) Define grid of pixels in projector framebuffer (m)Define grid of pixels in projector framebuffer (m) Project and record 3D surfaces pts (M)Project and record 3D surfaces pts (M) After surface unification (M’),After surface unification (M’),  find reprojected points (m’’) Tesselate the framebufferTesselate the framebuffer Load rendered image in texture memoryLoad rendered image in texture memory Render vertices m with texture coordinates as m’Render vertices m with texture coordinates as m’

74 Post Rendering warp (using texture mapping) Project pixel grid Record 3D pts Find reprojected pixels

75 Post Rendering warp (using texture mapping) Compute required post-rendering warp

76 Summary of Techniques Basic Approach At each projectorAt each projector –calibrate camera pair –project dots and find 3D display surface –(find surface illuminated by neighboring projectors) –find projector parameters Find intensity blending functionsFind intensity blending functions Register surface meshes (near alignment)Register surface meshes (near alignment)

77 Summary of Techniques Compensating the errors Unify meshes using smooth 3D transitionUnify meshes using smooth 3D transition  Use ‘camera overlap’ as transition region Re-compute all best-fit projector matricesRe-compute all best-fit projector matrices Find projective functionFind projective function  piecewise planar approximation using multiple triangles Specify all rendering tasks using a display listSpecify all rendering tasks using a display list

78 Summary of Techniques At run time, at each projectorAt run time, at each projector  Read current user location  Render virtual object using two-pass rendering  Post-rendering 2D warp  Intensity blending

79 Restricted cases Static User (Sweet spot)Static User (Sweet spot) –VSMM98 Planar display surfacePlanar display surface –Tech report Available at project web pageAvailable at project web page

80 Features Task of each projector is independentTask of each projector is independent 2-pass rendering, post-rendering warp is pre-defined2-pass rendering, post-rendering warp is pre-defined  Can be specified using display list Traditional hardwareTraditional hardware

81 5 Single lens 1024x768 projectors5 Single lens 1024x768 projectors Multiple 640x480 camerasMultiple 640x480 cameras SGI InfiniteReality2 for each projectorSGI InfiniteReality2 for each projector Matlab for calibration, OpenGL for renderingMatlab for calibration, OpenGL for rendering Implementation

82 Issues Resampling artifacts due to limited resolutionResampling artifacts due to limited resolution Color matchingColor matching Viewer-dependent parametersViewer-dependent parameters Limited depth-of-fieldLimited depth-of-field

83 Acknowledgements Office of the Future groupOffice of the Future group NSF STCNSF STC NTII (Advanced Network Systems)NTII (Advanced Network Systems) IntelIntel

84 Summary Multiple projectors to create panoramic imageryMultiple projectors to create panoramic imagery Geometric registration is major challengeGeometric registration is major challenge Video cameras to create 3D representationVideo cameras to create 3D representation Generalized display configurationGeneralized display configuration


Download ppt "Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, Brent Seales, Henry Fuchs University of North Carolina at Chapel."

Similar presentations


Ads by Google