1. 1 Online Construction of Surface Light Fields By Greg Coombe, Chad Hantak, Anselmo Lastra, and Radek Grzeszczuk

2. 2 ** Image-Based Modeling Benefits: Photorealistic content Modeling by acquisition Difficulties: Sampling issues Lack of feedback

3. 3 Motivation Casual Capture Pick up a video camera, wave it around the model, and capture reflectance model

4. 4 Surface Light Field A surface light field is a representation of the appearance of a model with known geometry and static lighting: viewing direction surface position (u,v) (θ,Φ)(θ,Φ)

5. 5 Our system Incremental system for online capture, construction, and rendering of surface light fields Each image is processed as it is captured Interactive feedback Guide user to undersampled regions Data-driven heuristic Structured method for handling missing data

6. 6 Previous Work Regular parameterizations light field data [Levoy96, Gortler96] Sparse, scattered data [Debevec96, Debevec98, Buehler01] BRDF capture systems [Dana99, Lafortune97, Marschner99, Debevec01, Gardner03] Function fitting Lafortune BRDF [McAllister02], Torrance-Sparrow BRDF [Sato97], Clusters of BRDFs [Lensh01], Homomorphic Factorization [McCool?], Bi-quadratic polynomials [Malzbender01] Online Methods Fixed viewpoint, progressive refinement [Matusik04] Adaptive Meshing of light field [Schirmacher99] Streaming non-linear optimization [Hillesland04] Data Mining [Brand02, Roweis97]

7. 7 SLFs using PCA Represent 4D function as a matrix Use Principal Component Analysis [Noshino01, Chen02] For each surface patch, break the 4D function into two 2D functions Store the principal components in texture maps rank

8. 8 SLFs using PCA Problems: Requires that the entire data set be available at once. Difficult to locate undersampled regions Requires recomputation when new images are added SVD *

9. 9 Online SVD Online SVD [Brand02] is a incremental PCA Update output matrices one sample at a time Advantages: Never store the entire data matrix at once Stream images Online SVD *...

10. 10 Online SVD Algorithm 6. The rotations are computed by re-diagonalizing this matrix 3. Compute the orthogonal component p 2. Project new image samples onto eigenspace j 4.If ||p|| is less than a threshold, then we just compute the rotations and update the eigenspace. U’ = UR u V’ = VR v 5.Otherwise, the current rank r is insufficient, so we append a new eigenvector. U’ = [U; m]R u V’ = VR v 1. Existing rank-r PCA

11. 11 Error Reference PCAOnline SVD Rank

12. 12 Imputation Missing data in red Imputed Matrix factorization approach requires fully resampled data matrices Missing data is a common problem Occlusion, meshing errors Imputation is the process of filling in these holes with “reasonable” guesses Better than zeros or mean Use the current Online SVD approximation to generate these missing values In practice, need about 5-10 initial images, and 50%+ coverage Further details in [Brand03]

13. 13 Automatic Feedback Aid the user in capturing surface light fields Direct attention towards undersampled areas Use a data-driven quality heuristic E = sqrt( e0^2 + e1^2) e0 – variation over hemisphere e1 – variation over surface

14. 14 System Real Time Pose Estimation Resampling*Online SVDVisibility* Quality Heuristic Video Camera Rendering Capture PCRender PC * From Intel’s OpenLF system

15. 15 Results Captured imageSLF before incorporating new image SLF after incorporating new image

16. 16 Results

17. 17 Timing

18. 18 Conclusion We present a method with Incremental construction Data-driven quality heuristic Online SVD is well-suited for streaming model Combine distribution and solution Requires only one pass over data

19. 19 Future Work We are still a long ways from “Casual Capture” Geometry must be known a priori Fixed illumination conditions Implementation issues Real-time pose estimation errors

20. 20

21. 21 Online SVD Problem: Multiple small rotations can accumulate error Brand proposes splitting output matrices to avoid accumulating error Advantages: Due to small working set, most data in cache (fast)

22. 22 Higher-Dimensional Problems We would like to extend Online SVD to higher-order factorizations Moveable light source Introduces another dimension of data Requires tensor product expansion View mapsSurface mapsLight maps

23. 23 2. Convergence Averaged over twenty random vertices

24. 24 3. Sensitivity

25. 25 Imputation Incomplete sample: (-1, ??) Existing SVD approximation Impute using SVD