Differential Instant Radiosity for Mixed Reality Martin Knecht, Christoph Traxler, Oliver Mattausch, Werner Purgathofer, Michael Wimmer Institute of Computer Graphics and Algorithms Vienna University of Technology

Motivation 2 Martin Knecht

Motivation 3 Martin Knecht

Problem statement Virtual objects should seamlessly blend into the real scene Provide a plausible illusion Simulation of light interaction between real and virtual objects is necessary Solution should run at interactive to real-time frame rates 4 Martin Knecht

Related Work Image Based Lighting Ritschel and Grosch used a HDR camera to capture environment Hierarchical Importance Sampling, Clarberg Real-time Global Illumination Instant Radiosity introduced by Keller Imperfect Shadow Maps from Ritschel et. al Merging Real and Virtual Scenes Differential Rendering introduced by Fournier and extended by Debevec Differential Photon Mapping from Grosch 5 Martin Knecht

Assumptions Geometry of real scene known BRDFs of real surfaces are known Surrounding illumination is distant 6 Martin Knecht

Differential Instant Radiosity for MR Idea: Combine Differential Rendering (DR) with Instant Radiosity (IR) Why?  DR adds virtual objects to real scene  IR fast method to compute GI solution in real-time But:Two solutions are needed for DR!  not really How?Keep track on the light paths 7 Martin Knecht

GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 8 Martin Knecht

GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 9 Martin Knecht

GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 10 Martin Knecht

GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 11 Martin Knecht

GI - Instant Radiosity Use Virtual Point Lights (VPLs) to approximate global illumination 12 Martin Knecht

Differential Rendering Method to add virtual objects into real scene images Compute two global illumination solutions and add only difference to real scene image +Minimizes visual error from wrong BRDF approximations -We need to calculate two GI solutions in real-time 13 Martin Knecht

Differential Rendering Difference Buffer Grey means no difference GI(Real + Virtual) GI(Real) 14 Martin Knecht

Differential Rendering Masked Video Image Difference Buffer Final Image 15 Martin Knecht

How to get GI(Real+Virtual) and GI(Real)? Keep track on the light paths L:Lightsource D:Diffuse light bounce S:Specular light bounce E: Eye Heckberts classification Courtesy of Paul Guerrero 16 Martin Knecht

How to get GI(Real+Virtual) and GI(Real)? Keep track on the light paths L:Lightsource D:Diffuse light bounce S:Specular light bounce E: Eye Direct illumination: LDE Heckberts classification Courtesy of Paul Guerrero 17 Martin Knecht

How to get GI(Real+Virtual) and GI(Real)? Keep track on the light paths L:Lightsource D:Diffuse light bounce S:Specular light bounce E: Eye Direct illumination: LDE 1st bounce diffuse indirect illumination: LDDE Heckberts classification Courtesy of Paul Guerrero 18 Martin Knecht

Differential Instant Radiosity for MR Calculate shading result only once Decide on path information to which GI solution the result belongs L [x] D [x] E L [x] D [x] D [x] E [x] can be „real“ or „virtual“ All „real“ then add to GI(Real+Virtual) and GI(Real) One [x] „virtual“, only add to GI(Real+Virtual) 19 Martin Knecht

L R D R D R E path Differential Instant Radiosity for MR 20 Martin Knecht GI(Real + Virtual) GI(Real)

L R D R D R E path Added to both Buffers Differential Instant Radiosity for MR GI(Real + Virtual) GI(Real) 21 Martin Knecht

L R D R D V E path Differential Instant Radiosity for MR 22 Martin Knecht GI(Real + Virtual) GI(Real)

L R D R D V E path Add only to the GI(Real+Virtual) Buffer Differential Instant Radiosity for MR GI(Real + Virtual) GI(Real) 23 Martin Knecht

Differential Instant Radiosity for MR Take blocking geometry into account 24 Martin Knecht

L R D R D R E path Differential Instant Radiosity for MR 25 Martin Knecht GI(Real + Virtual) GI(Real)

L R D R D R E path Virtual red wall blocks light! Differential Instant Radiosity for MR GI(Real + Virtual) GI(Real) 26 Martin Knecht

Limitations 27 Martin Knecht

Limitations Double shadowing Incosistent color bleeding 28 Martin Knecht

Results Test device Core2 Quad CPU Q9550 at 2.8GHz 8GB RAM NVIDIA Geforce GTX 285 with 1GB VRAM Windows 7 64-Bit C# with SlimDX to access DirectX Martin Knecht

Results 30 Martin Knecht

Results 31 Martin Knecht

Results 32 Martin Knecht

Results 33 Martin Knecht

Our method uses instant radiosity to simulate mutual influence between objects Direct Illumination Indirect Illumination Combination of Differential Rendering and Instant Radiosity Lightsources, objects, camera can be dynamic Cannot take all paths into account Works reasonable fast (20 – 30 fps) Conclusion 34 Martin Knecht

Consider all light paths Add camera artifacts Camera-based, real-time reconstruction of geometry Real-time BRDF estimation Future Work 35 Martin Knecht

Thank you for your attention! 36 Martin Knecht