Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography Paul Debevec, University of California at Berkley

Motivation Visual Effects Visual Effects –Adding synthetic actors/props Architectural simulation Architectural simulation –Adding buildings to landscapes

Related Work Previous methods used to insert synthetic objects into real scenes Previous methods used to insert synthetic objects into real scenes –Drawing –Environment mapping –Approximate geometric model and lighting Global illumination algorithms Global illumination algorithms Recovery of HDR radiance maps from photographs Recovery of HDR radiance maps from photographs

Related Work High Dynamic Range refers to the range of brightness values present in a scene High Dynamic Range refers to the range of brightness values present in a scene Eyes, photos, displays, all capture a limited, non-linear range Eyes, photos, displays, all capture a limited, non-linear range Everyday brightness range (about :1) 1000:1100:1 Source: Microsoft Developers Network

Related Work  Why is this a problem?  Limited brightness range results in lost scene detail from saturation and/or underexposure  OK for traditional image use Photos from OpenEXR by LucasFilm, LTD.

Related Work What about image-based lighting? What about image-based lighting? –Photo pixel values are not representative of relative Radiance in the scene Images by Paul Debevec, 98

Related Work Solution: Take multiple photos with varying shutter speeds and ‘combine’ somehow to create accurate relative radiance maps Solution: Take multiple photos with varying shutter speeds and ‘combine’ somehow to create accurate relative radiance maps Images by Paul Debevec, 98

Related Work Is HDR really important? Is HDR really important?

Method Divide the scene into three parts Divide the scene into three parts

Method Distant Scene Distant Scene –Light-based model –Required to provide correct incident illumination to local/synthetic objects from desired viewpoint –Assumed that no light from the model will affect the distant scene –Any level of geometric detail wanted

Method Local Scene Local Scene –Material-based model (has BRDF) –Must approximate the real scene local to the position of the synthetic objects –‘Local’ means photometrically local –‘Approximate’ means geometrically AND photometrically representative of the real scene

Method Synthetic Objects Synthetic Objects –Complete materials-based models of elements that are to be inserted into the scene –Geometric and lighting information –Any shape and material supported by the global illumination method that you plan to use

Method Compositing using a light probe Compositing using a light probe –One method to create the distant scene from a real location –Uses a metal sphere and a camera to gather light from a point

Method Compositing using a light probe Compositing using a light probe –This light is mapped onto the distant scene geometry and used to calculate the lighting –Lighting is calculated with global illumination

Method Results Results

Method Problem with the current definition of the local scene Problem with the current definition of the local scene –We said that it must approximate the material of the real scene –It can be really hard to get the BRDF/material of a real object! –Can we modify the local scene so it doesn’t require this?

Method Local Scene 2 (Differential Rendering) Local Scene 2 (Differential Rendering) –Approximate the material of the scene to within reason (not exact) –Render with and without your synthetic objects and calculate the difference –Add that difference back into the original image

Method Difference between local scene render with and without synthetic objects Difference between local scene render with and without synthetic objects - =

Results

Conclusion This paper presented a framework for rendering synthetic objects into real world environments with real lighting This paper presented a framework for rendering synthetic objects into real world environments with real lighting –High Dynamic Range radiance maps –Global Illumination –Distant and local scenes, synthetic objects –Differential rendering

Future Work Method to generate boundary between distant and local scene Method to generate boundary between distant and local scene More efficient global illumination More efficient global illumination

Comments Pros Pros –Open framework –Good results Cons Cons –Hard to decide what local scene should be! –Local scene can be very complicated