1. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill1 Painting on Movable Objects Deepak Bandyopadhyay, UNC-CH Ramesh Raskar, MERL Henry Fuchs, UNC-CH

2. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill2 Dynamic Shader Lamps IntroductionIntroduction Related WorkRelated Work ImplementationImplementation ResultsResults

3. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill3 Motivation For some applications, it is easier to work with a physical object directly, rather than using traditional displays or see-through AR HMDs. For some applications, it is easier to work with a physical object directly, rather than using traditional displays or see-through AR HMDs. »Reasons: Ergonomics, Passive Haptics »Example application: virtual 3D painting We show how to implement such systems.

4. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill4 Overview Projector1 Projector2 Objects Tracker PC

5. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill5 Video

6. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill6 Applications Testbed for AR applicationsTestbed for AR applications –projectors on movable objects AR Interaction mediumAR Interaction medium –Art –Teleimmersion –Architecture »annotation »visualization –Others, see paper

7. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill7 Dynamic Shader Lamps IntroductionIntroduction Related WorkRelated Work ImplementationImplementation ResultsResults

8. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill8 Shader Lamps Basic Idea Raskar et al 1, 2001 1 Ramesh Raskar, Greg Welch, Kok-Lim Low and Deepak Bandyopadhyay. “Shader Lamps : Animating Real Objects with Image-based Illumination”, Eurographics Rendering Workshop (EGRW 2001), London.

9. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill9 Shader Lamps Basic Idea Rearranging terms in optical path

10. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill10 Radiance Adjustment Desired Virtual k(x) cos(  p ) d(x) 2 L’(x,  )= I p (x,  p ) L(x,  ) =  F(x, ,  i ) L i (x,  i ) d  i Projected Real d ( x ) 2 k ( x ) cos(  p ) I p (x,  p ) = k (x)>0 L ( x,  )

11. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill11 Image based Illumination Basic IdeaBasic Idea »Render images and project on objects »Multiple projectors »View and object dependent color

12. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill12 Related Work : 3D Painting Hanrahan & Haeberli, 1990 2Hanrahan & Haeberli, 1990 2 –Painting on 3D models –Screen & mouse interface Agrawala & Beers, 1995 3Agrawala & Beers, 1995 3 –Painting Scanned Surfaces »Object is static »Tracked pointer moved on its surface »Separate display 2 Pat Hanrahan and Paul Haeberli, “Direct WYSIWYG Painting and Texturing on 3D Shapes”. SIGGRAPH’90. 3 Maneesh Agrawala, Andrew C. Beers and Marc Levoy, “3D Painting on Scanned Surfaces”, Symposium on Interactive 3D Graphics (I3D’95).

13. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill13 Related Work : 3D Painting Gregory, Ehmann & Lin, 1999 4Gregory, Ehmann & Lin, 1999 4 –inTouch haptic painting »Modeling, painting, force feedback »No real object; purely virtual Keefe, Feliz et al, 2001 5Keefe, Feliz et al, 2001 5 –CavePainting »Physical interface objects »Display is purely virtual 4 Arthur Gregory, Stephen Ehmann and Ming C. Lin, “inTouch : Interactive Multiresolution Modeling and 3D Painting with a Haptic Interface”, Virtual Reality (VR’99). 5 Daniel F. Keefe, Daniel A. Feliz, Tomer Moscovitch, David Laidlaw and Joseph LaViola, Jr., “CavePainting : a Fully Immersive 3D Artistic Medium and Interactive Experience”, I3D 2001.

14. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill14 Dynamic Shader Lamps IntroductionIntroduction Related WorkRelated Work ImplementationImplementation ResultsResults

15. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill15 Projector Calibration Calibrate each projector (5 minutes)Calibrate each projector (5 minutes) –Match 3D points in world space to 2D pixels in projector space. –Solve for 3x4 projection matrix »ignore radial distortion »minimize least-square error (standard CV method) »split into intrinsic (projection) and extrinsic matrix Render 3D points in the world with good registrationRender 3D points in the world with good registration 2D cross hair

16. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill16 Model Generation/Authoring Need: vertices, faces, normals, texture coordinates and color/material scheme per object to be paintedNeed: vertices, faces, normals, texture coordinates and color/material scheme per object to be painted –Acquire points using magnetic stylus –Faces, textures by hand / surface reconstruction –Texture coordinates non-repeating, fit geometry –Future: automate with modeling tools.

17. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill17 Rendering Tracked Object Algorithm each-frame: Tracker gives 6DOF reading of sensors. For view i (projector i) do Load projection and model view matrix i For object j do Apply world to sensor j matrix Draw object in sensor coordinates Interact with object

18. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill18 3D painting interface Objects hand-held or set on table.Objects hand-held or set on table. Tracked stylus with spherical tipTracked stylus with spherical tip –facilitates contact painting Projected touch palette, modeled as a static object with behavior:Projected touch palette, modeled as a static object with behavior: –choose contact, spray or texture paint –choose brush color

19. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill19 Painting objects Determine object intersections (point-polygon)Determine object intersections (point-polygon) –Find set of polygons to modify Color blending using brush function 6Color blending using brush function 6 –Contact paint uses a function on surface of object B = 1-(r/R) 2, r<R –Spray paint uses a similar brush function in 3D –Color (or texture) blending: C final = C init.(1-B) + C brush.B r R 6 From Arthur Gregory’s inTouch system

20. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill20 Painting objects Triangle rasterization w/ texture map modification 6Triangle rasterization w/ texture map modification 6 –Scan 2D and 3D simultaneously. qq pp uu vv concurrent (x,y,z) Polygon in 3D (u,v) 2D texture map Update color Look up brush fn 6 From Arthur Gregory’s inTouch system

21. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill21 Dynamic Shader Lamps IntroductionIntroduction Related WorkRelated Work ImplementationImplementation ResultsResults

22. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill22 Performance / numbers ? 60fps rendering 3 objects, 500+ polygons60fps rendering 3 objects, 500+ polygons –~100 texture maps, mostly 128x128 –Running on SGI InfiniteReality2 or 450MHz PC with multi-monitor graphics card Not optimized for larger data setsNot optimized for larger data sets 100-110ms latency (high).100-110ms latency (high). –Roughly estimated »1cm of distance lag when moved at 10cm/s –Later, precisely measured

23. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill23 Issues and Future Work Tracker LimitationsTracker Limitations –Range (Polhemus Fastrak, 50-75 cm) »use wide-area tracker –Technology (ferrous objects distort magnetic field) Display limitationsDisplay limitations –Dynamic Intensity Blending –Limited dynamic range of projector –Occlusion, overlap, shadows Scaling it upScaling it up –Room sized environments, multiple projectors

24. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill24 Conclusions Dynamic Shader LampsDynamic Shader Lamps –A set of techniques for interactively changing appearance attributes of neutrally colored physical objects, static or moving Prototype implementationPrototype implementation –Painting on multiple real objects –display devices - projectors »3D display surfaces (real objects) –3D input devices - tracker & stylus Extensions - how to generalize/scale.Extensions - how to generalize/scale.

25. 5/1/2000Deepak Bandyopadhyay / UNC Chapel Hill25 Acknowledgements ISAR reviewersISAR reviewers Miriam Mintzer Fuchs, age 10Miriam Mintzer Fuchs, age 10 People at UNC (paper, video & demo)People at UNC (paper, video & demo) –Herman Towles, Greg Welch, Anselmo Lastra, Russ Taylor, Ming Lin, Scott Russell –Andrei State, Jim Mahaney, Kurtis Keller, John Thomas, David Harrison, Arthur Gregory –Kok-Lim Low, Wei-Chao Chen, Ben Lok, Kenny Hoff, Dave McAllister, Miguel Otaduy, Gentaro Hirota STC Tracked Shader Lamps, fundingSTC Tracked Shader Lamps, funding

26. Dynamic Shader Lamps Deepak Bandyopadhyay, Ramesh Raskar, Henry Fuchs http://www.cs.unc.edu/~debug/papers/DSLpaint/ Painting on Movable Objects Images