Light Field Mapping: Hardware-Accelerated Visualization of Surface Light Fields

What is a Surface Light Field? 4-D Function – f (r, s, Θ, Φ) 4-D Function – f (r, s, Θ, Φ) Defines radiance of every point on surface of an object in every viewing direction Defines radiance of every point on surface of an object in every viewing direction (r,s) – Describe surface location (r,s) – Describe surface location (Θ, Φ) – Describe viewing location (Θ, Φ) – Describe viewing location In practice, almost always discrete In practice, almost always discrete

Proposed Approach f (r, s, Θ, Φ) ≈ ∑ g k (r,s) h k (Θ, Φ) (eq 1) f (r, s, Θ, Φ) ≈ ∑ g k (r,s) h k (Θ, Φ) (eq 1)

Surface Light Field Approximation Approximation algorithms assume data given as 4-D grid Approximation algorithms assume data given as 4-D grid f (r p, s p, Θ q, Φ q ) f (r p, s p, Θ q, Φ q ) p = 1, …, M – discrete values of surface location p = 1, …, M – discrete values of surface location q = 1, …, N – discrete values of viewing angles q = 1, …, N – discrete values of viewing angles

Surface Light Field Approximation f (r p, s p, Θ q, Φ q ) ≈ ∑ g k (r p,s p ) h k (Θ q, Φ q ) (eq 2) f (r p, s p, Θ q, Φ q ) ≈ ∑ g k (r p,s p ) h k (Θ q, Φ q ) (eq 2) Only practical if # of terms is small Only practical if # of terms is small Difficult to find good approximation to complete SLF using few summation terms Difficult to find good approximation to complete SLF using few summation terms

Surface Light Field Approximation Instead, surface of object partitioned into smaller units Instead, surface of object partitioned into smaller units By decomposing SLF of each unit, close approximation of original data obtained By decomposing SLF of each unit, close approximation of original data obtained Allows for efficient storage and fast rendering Allows for efficient storage and fast rendering

Using Singular Value Decomposition Use SVD to factor SLF Use SVD to factor SLF This method more robust, optimal This method more robust, optimal To apply, 4D SLF must be rearranged into matrix To apply, 4D SLF must be rearranged into matrix

Approximation Through SVD F P = USV T F P = USV T U – square matrix (u k ) U – square matrix (u k ) V – square matrix (v k ) V – square matrix (v k ) S – diagonal matrix (σ k ) S – diagonal matrix (σ k ) USV T = ∑σ k u k v k T (eq 4) USV T = ∑σ k u k v k T (eq 4)

Triangle-Centered Approximation Partition light field function into individual triangles Partition light field function into individual triangles f (r, s, Θ, Φ) = ∑ ∏ ∆i (r,s) f (r, s, Θ, Φ) (eq 5) f (r, s, Θ, Φ) = ∑ ∏ ∆i (r,s) f (r, s, Θ, Φ) (eq 5) Θ – azimuth angle Θ – azimuth angle Φ – elevation angle Φ – elevation angle This is the triangle light field This is the triangle light field When rendered, produces visible discontinuities at triangle edges When rendered, produces visible discontinuities at triangle edges

Vertex-Centered Approximation To eliminate discontinuities, partition SLF around every vertex To eliminate discontinuities, partition SLF around every vertex f (r, s, Θ, Φ) = ∑ Λ vj (r,s) f (r, s, Θ, Φ) (eq 8) f (r, s, Θ, Φ) = ∑ Λ vj (r,s) f (r, s, Θ, Φ) (eq 8) In this method, each triangle shares light field maps with neighboring triangles In this method, each triangle shares light field maps with neighboring triangles

Representation of Light Field Maps 2D texture representation of surface map as G k (s,t) 2D texture representation of surface map as G k (s,t) 2D texture representation of view map as H k (x,y) 2D texture representation of view map as H k (x,y) Texture coordinate computation Texture coordinate computation X=(dx + 1) /2 (eq 13) X=(dx + 1) /2 (eq 13) Y=(dy + 1) /2 Y=(dy + 1) /2

Rendering Algorithm Triangle-centered & vertex centered approaches differ only in how each individual approximation term is evaluated Triangle-centered & vertex centered approaches differ only in how each individual approximation term is evaluated Surface map coordinates for both do not need to be recomputed Surface map coordinates for both do not need to be recomputed View map coordinates recomputed every time view changes View map coordinates recomputed every time view changes

Rendering Algorithm Now evaluate kth approximation term Now evaluate kth approximation term Triangle-centered – multiply pixel-by-pixel image projections of 2 texture fragments Triangle-centered – multiply pixel-by-pixel image projections of 2 texture fragments Vertex-centered – multiply pixel-by-pixel 3 pairs of light field maps from each vertex and add together Vertex-centered – multiply pixel-by-pixel 3 pairs of light field maps from each vertex and add together

Hardware Accelerated Implementation Pixel-by-pixel approach (modulation)– multiplication of surface map fragment by view map fragment Pixel-by-pixel approach (modulation)– multiplication of surface map fragment by view map fragment Multitexturing hardware support allows for effective modulation of two texture fragments in one rendering pass Multitexturing hardware support allows for effective modulation of two texture fragments in one rendering pass K-term approximation K-term approximation K rendering passes for triangle-centered K rendering passes for triangle-centered 3K rendering passes for vertex-centered 3K rendering passes for vertex-centered

Data Acquisition First images are captured under fixed lighting conditions ( images) First images are captured under fixed lighting conditions ( images) Object geometry is computed through structured lighting system consisting of projector and camera(10-20 scans) Object geometry is computed through structured lighting system consisting of projector and camera(10-20 scans) Scans are registered together in same reference frame used for image registration Scans are registered together in same reference frame used for image registration Resulting points fed into mesh editing software Resulting points fed into mesh editing software Finally, mesh is projected onto camera image Finally, mesh is projected onto camera image