Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister

Motivation Complex natural scenes are difficult to acquire Acquisition needs to be easy and robust Image-based lighting offers high realism We would like to relight image-based models at any scale (from small objects to cities)

Motivation Image-based Relighting –no scene geometry – just images –no assumptions about scene reflectance properties

Previous Work Forward Approaches –Georghiades 99, Debevec 2000, Malzbender 01, Masselus 02, Peers 03 Inverse Approaches –Zongker 99, Chuang 00, Wexler 02 Pre-computed Light Transport –Sloan 02, Ng 03

Reflectance Field 8D function: [Debevec 2000] (θ r, φ r ) (u r,v r ) (θ i, φ i ) (u i,v i )

Reflectance (Weighting) Function Assumes incident illumination originates at infinity x,y are image space coordinates θiθi φiφi

Light Transport Model A light flow in the scene can be modeled as a multiple-input / multiple-output linear system: Scene light transport matrix T Incident Light L Observed Image B Unroll to a vector

Light Transport Model Solve independently for each output pixel multiple-input / single-output linear system : Scene light transport vector T i Incident Light L Observed Pixel b i

Representation Approximate T i as a sum of 2D rectangular kernels R k,i, each with weight w k,i. θiθi φiφi

Inverse Estimation Given input images L j we record observed pixel values b ij : Given matrix L and vector b i the goal is to estimate T i –Positions and sizes of the rectangular kernels R k,i –Weights w k, i

Estimating Kernel Weights Assume that we know sizes and positions of the kernels R k,i and would like to compute their weights Efficient solution using quadratic programming

Estimating Kernel Positions & Sizes Hierarchical kd-tree subdivision of the kernels input image domain At each level choose subdivision that reduces error the most Kernels are non-overlapping

Kernel Subdivisions specular refractive subsurface scattering glossy hard shadow Subdivisions

Spatial Correction The kernels search strategy does not always work Solution: For each output pixel: –try kernel positions and sizes of the neighboring output pixels –try shifted versions of the current kernels –solve for new weights –keep new kernels if the error decreases

Integration with Incident Illumination Is very efficient For each output pixel i The incident illumination is stored as a summed- area table to evaluate

Data Acquisition We have built two acquisition systems –Indoor scenes / small objects –Outdoor scenes (city)

Acquisition System I

Example Input Images

Results Refractive and specular elements Prediction Actual

Results – New Illumination

Results - White Vertical Bar Prediction Actual

Results Estimate Actual Diffuse elements, shadows

Results - White Vertical Bar

Results Estimate Actual Subsurface Scattering

Results - White Vertical Bar

Results Glossy elements and interreflections Estimate Actual

Results - White Vertical Bar

Results One shifted version of the same image used as input illumination

Acquisition System II Two Synchronized Cameras Camera #1 Camera #2

Example Observed Images

Results – Relighting The City White vertical bar

Lessons Inverse approaches benefit from good kernel search strategies & more computation power Inverse approaches are more efficient than forward approaches Challenges: –Scene needs to be static –Varied set of input illumination –Illumination is not at infinity

Conclusions Advantages of our algorithm: –Natural Illumination Input –All-frequency Robustness –Compact Representation –Progressive Refinement –Fast Evaluation –Simplicity

Future Work New acquisition systems –object and camera are fixed w.r.t. each other and they rotate in a single, natural environment Combining representations from different viewpoints and proxy geometry Coarse-to-fine estimation in the observed image space –start with low resolution observed images & search exhaustively for the best kernels –propagate the kernels to higher resolution images

Acknowledgements Jan Kautz Barb Cutler Jennifer Roderick Pfister EGSR Reviewers