1. Sebastian Enrique Columbia University senrique@cs.columbia.edu Relighting Framework COMS 6160 – Real-Time High Quality Rendering Nov 3 rd, 2004

2. COMS 6160 Relighting Framework Sebastian Enrique - Columbia University - senrique@cs.columbia.edu Nov 3 rd, 2004 2 About Relighting Why it is useful? Photorealistic real-time rendering of complex scenes with complex illumination is an open problem. One IBR approach is to capture or render a set of original images from a scene, and then relight it in real time to produce the same scene but with novel illumination! IMAGES += LIGHTS RELIGHTED SCENE What is Relighting? “Given a set of different illuminated images from a scene, relighting is the process of producing new images of the same scene with new lighting conditions, composing in some way the original data.”

3. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 3 About Relighting (cont.) What is one of the most challenging parts of it? High quantity of images is needed to produce good results, the challenge is to find/use a good compression technique in order to relight fast and using as less memory as possible. IMAGES ? COMPRESSED IMAGES COMS 6160Nov 3 rd, 2004 Relighting Framework

4. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 4 Some Previous Work Doorsey et al. used a progressive radiosity method where they pre-rendered synthetic scenes to simulate lighting conditions –superimposing single light source images- in opera stages in 1991. Many (Hallinan ‘94, Epstein ‘95, etc.) have pre-acquired real images changing the lighting direction. Debevec compressed the pre-acquired images in JPG and processed in the compressed domain in 2000. Sloan uses low-frequency spherical harmonics on geometry in 2002. In 2003 Sloan uses clustered or VQPCA on spherical harmonic coefficients. Ng compressed data using wavelets in 2003. Sloan (Local Deformable PRT) and Ramamoorthi (Triple Product Wavelets) are the most recent related techniques, 2004. COMS 6160Nov 3 rd, 2004 Relighting Framework

5. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 5 LSD Relighting Approach The Lighting Sensitive Display (LSD) was developed by Shree Nayar, Peter Belhumeur, and Terry Boult. Basically, it is a monitor with an attached camera that captures the lighting conditions of the environment. The monitor shows an scene, which changes (is relighted) as the illumination in the environment changes. They adopted an image-based approach, using a large set of images. COMS 6160Nov 3 rd, 2004 Relighting Framework

6. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 6 LSD Relighting Approach (cont.) They have developed a novel algorithm (using two stages of Principal Component Analysis or PCA) that compresses that large set of images and allows the relighting in real-time with complex lighting conditions. They have reached compression ratios of 476:1 for colored images. The algorithm simultaneously exploits correlations over the lighting domain as well as coherences over the spatial domain of the image. COMS 6160Nov 3 rd, 2004 Relighting Framework

7. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 7 LSD Relighting Approach (cont.) We based our Relighting Framework on the LSD approach. In the following slides we will explain: How the input images for the LSD algorithm should be taken. First compression stage of the algorithm. Second compression stage of the algorithm. Real-Time rendering. Then, we will get on: Extending the LSD approach with the use of cubemaps. Problems found. Future directions. Discussion and Videos COMS 6160Nov 3 rd, 2004 Relighting Framework

8. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 8 Setting Up Images Fixed Viewpoint Grid of Positions for Distant Light Sources, it’s in the front face of an imaginary cube. Distant Light Source For Single Image Scene to Render / Capture To take the images, a grid of light source positions is generated in the front face of an imaginary cube (a plane parallel to the image plane). For each position on the grid, place the distant light with maximum intensity and generate (render or capture) an image. COMS 6160Nov 3 rd, 2004 Relighting Framework

9. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 9 First Stage SVD The first SVD (Singular Value Decomposition) exploits the fact that locally (within small regions of the images) the variation due to changes in illumination can be approximated by a small number of bases. The image is divided into m square blocks, each containing p pixels, and bases are computed for each. Each image I i is an image of the scene illuminated by a single distant point light source. IMAGES Image divided in m square blocks Each block has p pixels Image I i COMS 6160Nov 3 rd, 2004 Relighting Framework

10. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 10 First Stage SVD (cont.) I i j denote the jth block of the ith image. For each block in the scene a low-dimensional approximation is computed as follows: I j is a p x n matrix representing a collection of image blocks. The I column of I j is formed by p pixels of the jth block. Using SVD, a rank b approximation to I j is found as: E j is a p x b column-orthogonal matrix, called block bases. S j is a b x b diagonal matrix. C j is an n x b column-orthogonal matrix. The singular values from S j are absorbed into C jT, getting a b x n matrix, L j, called block lighting coefficients. Each I j represents the pixels for the same block on all images COMS 6160Nov 3 rd, 2004 Relighting Framework

11. Ij Ij SjSj C jT p x n n x n = xx diagonal matrix (singular values) Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 11 First Stage SVD (cont.) Ij Ij C jT p x n b x bb x n  xx EjEj p x b SjSj Applying Rank b: SVD: EjEj p x p Ij Ij LjLj p x n b x n  x EjEj p x b Abosorving S j values into C iT : COMS 6160Nov 3 rd, 2004 Relighting Framework

12. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 12 First Stage SVD (cont.) Then, stacking all of the m L j block matrices, we get the lighting coefficient matrix L. The image bases m E j are also stacked in the matrix E. The collection of submatrices within E and L contain all the information needed to approximate the collection of images corresponding to the n lighting directions. L (m * b) x n E (m * p) x b COMS 6160Nov 3 rd, 2004 Relighting Framework

13. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 13 First Stage SVD (cont.) Image: 640x480; n=64x64=4096; m=40x30=1200; p=16x16=256; b=10 COMS 6160Nov 3 rd, 2004 Relighting Framework

14. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 14 Second Stage SVD The fact that there is much coherence in image blocks is exploited using a second SVD. It is applied to the light coefficient matrix L: U is an (m x b) x q column-orthogonal matrix, called lighting coefficient bases. V is a q x n matrix, called compressed coefficient matrix. Rank q denotes the number of linear bases kept to approximate L. L V (m * b) x n q x n  x U (m * b) x q COMS 6160Nov 3 rd, 2004 Relighting Framework

15. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 15 Second Stage SVD (cont.) Image: 640x480; n=64x64=4096; m=40x30=1200; p=16x16=256; b=10; q=200 COMS 6160Nov 3 rd, 2004 Relighting Framework

16. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 16 Examples Relighted Image SVD First Stage Rank 3 Original Image Bad Rank Election (showing only red color component) Note block discontinuities! COMS 6160Nov 3 rd, 2004 Relighting Framework

17. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 17 Examples (cont.) Relighted Image SVD First Stage: Rank 10 SVD Second Stage: Rank 20 Original Image (in fact, two combined original images) Higher Ranks Only small differences in brightness are noticeable, but there are no visible block discontinuities. COMS 6160Nov 3 rd, 2004 Relighting Framework

18. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 18 Real-Time Rendering Real-Time illumination is represented by the illumination field vector s. Each element s i corresponds to a point light, and its value represents the intensity that that point light is contributing with to the scene. To render the relighted scene in real-time, preprocessed matrices E, U, and V should be used: 1.Compute a compressed coefficient vector as the product (q elements) 2.Compute a lighting coefficient vector I as (m*b elements) 3.To render the j block, a subvector I j from I must be multiplied with the corresponding stacked E j matrix. This must be done each frame for the m blocks. (b elements) (p elements) COMS 6160Nov 3 rd, 2004 Relighting Framework

19. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 19 Real-Time Rendering (cont.) COMS 6160Nov 3 rd, 2004 Relighting Framework

20. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 20 Demo: Point Lights / David Image: 480x640; n=6x16x16=1536; m=32x32=1024; p=16x16=256; b=10; q=20 COMS 6160Nov 3 rd, 2004 Relighting Framework

21. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 21 Extending LSD: Cube Lighting In place of using only the front face of the cube to capture images with different lighting condition, use all of the 6 faces. The parameters that change are the quantity of input images and the length of the s light field vector (both now 6 * n). In this way, the scene can be relighted with illumination coming for all around. Grid of Positions for Distant Light Sources in the Full Cube Distant Light Source For Single Image Fixed Viewpoint COMS 6160Nov 3 rd, 2004 Relighting Framework

22. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 22 High-Dynamic Range Cubic Environment Map Having the relighting scheme for the full cube, we can add cubic environment maps to relight the scene. To have more accuracy, High-Dynamic Range (HDR) cubic environment maps are used (floating point values in place of 0..255). The CubicMap could be rotated, and each element in the s light field vector is contributed by the value of the corresponding texel, weighted by the solid angle. HDR(i) is the corresponding texel value; N is the corresponding cube face normal; R is the vector from the origin to the center of the element i in the cube grid; m is the face cube grid resolution (squared is how many images per face exist); each cube face has a size of 2 x 2. COMS 6160Nov 3 rd, 2004 Relighting Framework

23. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 23 Demo: CubeMap & Point Lights / Nicole Image: 512x512; n=6x16x16=1536; m=32x32=1024; p=16x16=256; b=10; q=20 COMS 6160Nov 3 rd, 2004 Relighting Framework

24. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 24 Framework Summary PREPROCESS (SVD 1 & 2) IMAGES E b = […] U b = […] V b = […] MATRICES E b = […] U b = […] V b = […] MATRICES REAL-TIME RELIGHTING FRAMEWORK RELIGHTED SCENE HDR Generate Scene-Matrices For Images Set Of Each Scene to Relight Relight in Real-Time the Given Scene-Matrices with HDR CubeMap and Point Lights COMS 6160Nov 3 rd, 2004 Relighting Framework

25. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 25 Problems Found Preprocess takes TOO MUCH TIME. Preprocess uses TOO MUCH MEMORY. Few things could be done in current graphics hardware (HDR cubemap processing specially). COMS 6160Nov 3 rd, 2004 Relighting Framework

26. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 26 Future Directions Optimize preprocessing stages (distributed computations?) Use error metrics to automatically select adequate ranks. Extend user interface to allow the relighting using lower ranks than that given in the input matrices. Allow viewpoint changes. COMS 6160Nov 3 rd, 2004 Relighting Framework

27. Sebastian Enrique - Columbia University - senrique@cs.columbia.edu 27 End of Talk Ready to hear… Comments Suggestions Discussions Questions More VIDEOS to show while chatting… COMS 6160Nov 3 rd, 2004 Relighting Framework