Interactive Virtual Relighting and Remodelling of Real Scenes C. Loscos 1, MC. Frasson 1,2,G. Drettakis 1, B. Walter 1, X. Granier 1, P. Poulin 2 (1) iMAGIS* - GRAVIR/IMAG - INRIA Rhône-Alpes * iMAGIS is a joint project of CNRS/INRIA/UJF/INPG * iMAGIS is a joint project of CNRS/INRIA/UJF/INPG (2) Département d ’informatique et de recherche opérationnelle, Université de Montréal Université de Montréal

MAGIS i Motivation Interior design Geometric modificationChanges in lighting

MAGIS i Motivation ü Goal: interactive system simple capture process simple capture process interactive ( ~ 1 sec. per frame) interactive ( ~ 1 sec. per frame) modification of lighting modification of lighting modification of geometry modification of geometry

MAGIS i Motivation ü We have to: create a simple model of the real scene create a simple model of the real scene  geometry  approximate reflectance represent real global illumination represent real global illumination develop interactive methods for modifications develop interactive methods for modifications ü Goal is to be convincing, not highly accurate

MAGIS i Previous Work ü Geometric reconstruction vision methods [Faugeras et al. 97,...] (Realise) vision methods [Faugeras et al. 97,...] (Realise) constraint-based systems [Debevec et al. 96, Poulin et al. 98] constraint-based systems [Debevec et al. 96, Poulin et al. 98] software: Photomodeler, etc. software: Photomodeler, etc. ü Reflectance recovery e.g., [Sato et al. 97, Ward92, Debevec98, Yu et al. 98, etc]. e.g., [Sato et al. 97, Ward92, Debevec98, Yu et al. 98, etc].

MAGIS i Previous Work ü Real-time direct shadows real point light source [State et al. 96] real point light source [State et al. 96] ü Common global illumination non-interactive non-interactive  [Nakamae et al. 86, Fournier et al. 93, Jancène et al. 95, Debevec 98, Yu et al. 98, Yu et al. 99] interactive interactive  [Drettakis et al. 97, Loscos et al. 98]

MAGIS i Algorithm Overview ü Input ü Pre-process ü Interactive modification

MAGIS i Algorithm Overview - Assumptions ü Single viewpoint ü Diffuse assumption ü Lighting: direct lighting: ray casting direct lighting: ray casting indirect lighting: hierarchical radiosity indirect lighting: hierarchical radiosity radiosity = reflectance x ( direct light + indirect light )

MAGIS i Simple Input Process ü Geometric reconstruction several (4-5) images from different viewpoints several (4-5) images from different viewpoints geometric modelling using “Rekon” [Poulin et al. 98] geometric modelling using “Rekon” [Poulin et al. 98] ü Reflectance reconstruction several (5-7) images from a single viewpoint several (5-7) images from a single viewpoint different lighting conditions: single light source at different positions different lighting conditions: single light source at different positions “radiance images” “radiance images”

MAGIS i Input ü Radiance images from single viewpoint combining multiple images reduces artefacts of estimation combining multiple images reduces artefacts of estimation different lighting conditions

MAGIS i Pre-process ü Computation of approximate diffuse reflectance pixel by pixel compute individual reflectance images compute individual reflectance images merge reflectance images using confidence values merge reflectance images using confidence values ü Initialise lighting system data structure data structure hierarchical radiosity system hierarchical radiosity system

MAGIS i Reflectance Computation ü For each radiance image photographreflectance reflectance = radiosity / ( direct light + indirect light )

MAGIS i Confidence Images ü Estimate confidence confidence ~ quality of reflectance estimate confidence ~ quality of reflectance estimate create a confidence image per light source position create a confidence image per light source position ü Begin with confidence = Visibility low in shadow regions low in shadow regions ü Filtering process to remove unwanted effects low for outliers (specular effects, light tripod) low for outliers (specular effects, light tripod)

MAGIS i Merged Reflectance Computation x x avg. reflectanceconfidence merged reflectance

MAGIS i Interactive Modification: Shadow Reprojection ü Direct illumination: pixel by pixel ü Indirect illumination: optimised radiosity solution a pixel Reflectance Indirect lighting Direct lighting

MAGIS i Shadow Re-projection photograph simulated

MAGIS i Add/move/remove object (virtual or real) ü Visible surface changes: pixel by pixel local update project bounding box of dynamic object project bounding box of dynamic object  localise directly affected pixels original object insertion

MAGIS i Add/move/remove object (virtual or real) ü Direct lighting updates: shaft structure localisation of visibility changes (shadows) localisation of visibility changes (shadows) accelerate visibility computation (blocker lists) accelerate visibility computation (blocker lists) original object insertion

MAGIS i Add/move/remove object (virtual or real) ü Indirect illumination computed by a radiosity solution (optimised by the shaft structure) ü Example: moving object Position 1Position 2

MAGIS i Real Object Removal

MAGIS i Removing Real Objects ü Use of the reflectance image (lighting effects removed) to generate new textures reflectance images

MAGIS i Light Source Modification ü Insertion of a virtual light source computation for every pixel computation for every pixel  new form-factors  new visibility ü Indirect illumination: radiosity solution

MAGIS i Lighting Modification Insertion of virtual light Original virtual lighting

MAGIS i Video

MAGIS i Conclusion ü Input data simple to acquire data simple to acquire ü Pre-process data structures optimised for fast updates data structures optimised for fast updates ü Interactive modification add and move virtual objects add and move virtual objects remove real objects remove real objects relighting relighting

MAGIS i Future Work ü Improve reflectance computation use of high dynamic range images (instead of RGB) use of high dynamic range images (instead of RGB) better control of indirect illumination better control of indirect illumination ü Allow motion of real objects ü Faster: parallel computation

MAGIS i Future Work ü Remove restrictions diffuse reflectance [Yu et al. 99] diffuse reflectance [Yu et al. 99] fixed view-point fixed view-point