1. Real-Time 3D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Ilya Korsunsky Princeton University Stanford University Hunter College

2. 3D Scanning

3. Possible Research Goals Low noiseLow noise Guaranteed high accuracyGuaranteed high accuracy High speedHigh speed Low costLow cost Automatic operationAutomatic operation No holesNo holes

4. 3D Model Acquisition Pipeline 3D Scanner

5. 3D Model Acquisition Pipeline 3D Scanner View Planning

6. 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment View Planning

7. 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMerging View Planning

8. 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning

9. 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay

10. 3D Model Acquisition Difficulties Much (often most) time spent on “last 20%”Much (often most) time spent on “last 20%” – Merging, Displaying, View Planning Pipeline not optimized for hole-fillingPipeline not optimized for hole-filling Not sufficient just to speed up scanner – must design pipeline for fast feedbackNot sufficient just to speed up scanner – must design pipeline for fast feedback

11. Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Human

12. Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMerging Done?Done? View Planning DisplayDisplay Challenge: Real Time

13. Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part I: Structured-Light Triangulation

14. Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part II: Fast ICP

15. Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part III: Voxel Grid

16. Triangulation Project laser stripe onto objectProject laser stripe onto object Object Laser CameraCamera

17. CameraCamera Triangulation Depth from ray-plane triangulationDepth from ray-plane triangulation Laser (x,y) Object

18. Triangulation Faster acquisition: project multiple stripesFaster acquisition: project multiple stripes Correspondence problem: which stripe is which?Correspondence problem: which stripe is which?

19. Continuum of Triangulation Methods Slow, robust Fast, fragile Multi-stripeMulti-frame Single-frame Single-stripe

20. Time-Coded Light Patterns Assign each stripe a unique illumination code over time [Posdamer 82]Assign each stripe a unique illumination code over time [Posdamer 82] Space Time

21. Illumination history = (WB),(BW),(WB) CodeCode Codes for Moving Scenes Assign time codes to stripe boundariesAssign time codes to stripe boundaries Perform frame-to-frame tracking of corresponding boundariesPerform frame-to-frame tracking of corresponding boundaries – Propagate illumination history [Hall-Holt & Rusinkiewicz, ICCV 2001] [Hall-Holt & Rusinkiewicz, ICCV 2001]

22. Designing a Code Want many “features” to track: lots of black/white edges at each frameWant many “features” to track: lots of black/white edges at each frame Try to minimize ghosts – WW or BB “boundaries” that can’t be seen directlyTry to minimize ghosts – WW or BB “boundaries” that can’t be seen directly

23. Designing a Code 00111110 10110110 01001001 0001000111001100 00001101 10100111 10000101 1111111100100010 [Hall-Holt & Rusinkiewicz, ICCV 2001]

24. Implementation Pipeline:Pipeline: DLP projector illuminates scene @ 60 Hz.DLP projector illuminates scene @ 60 Hz. Synchronized NTSC camera captures videoSynchronized NTSC camera captures video Pipeline returns range images @ 60 Hz.Pipeline returns range images @ 60 Hz. ProjectCodeProjectCodeCaptureImagesCaptureImagesFindBoundariesFindBoundariesMatchBoundariesMatchBoundariesDecodeDecodeComputeRangeComputeRange

25. Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part II: Fast ICP

26. Aligning 3D Data This range scanner can be used for any moving objectsThis range scanner can be used for any moving objects For rigid objects, range images can be aligned to each other as object movesFor rigid objects, range images can be aligned to each other as object moves

27. ICP in the Real-Time Pipeline Potential problem with ICP: local minimaPotential problem with ICP: local minima – In this pipeline, scans close together – Very likely to converge to correct (global) minimum Basic ICP algorithm too slow (~ seconds)Basic ICP algorithm too slow (~ seconds) – Point-to-plane minimization – Projection-based matching – With these tweaks, running time ~ milliseconds [Rusinkiewicz & Levoy, 3DIM 2001]

28. Anchor Scans If object moves too quickly or is out of range, ICP will failIf object moves too quickly or is out of range, ICP will fail Global driftGlobal drift Instead of aligning to last scan, keep set of scans to realign toInstead of aligning to last scan, keep set of scans to realign to Large: have enough dataLarge: have enough data Distinct: don’t store too manyDistinct: don’t store too many Allows user to restart scanningAllows user to restart scanning

29. Recapturing Alignment

30. Real-Time 3D Model Acquisition Pipeline 3D Scanner AlignmentAlignment MergingMergingDone?Done? View Planning DisplayDisplay Part III: Voxel Grid

31. Merging and Rendering Goal: visualize the model well enough to be able to see holesGoal: visualize the model well enough to be able to see holes Cannot display all the scanned data – accumulates linearly with timeCannot display all the scanned data – accumulates linearly with time Standard high-quality merging methods: processing time ~ 1 minute per scanStandard high-quality merging methods: processing time ~ 1 minute per scan Use fast voxel based merging techniqueUse fast voxel based merging technique

32. Merging and Rendering

35. +

36. Point rendering, using accumulated normals for lightingPoint rendering, using accumulated normals for lighting

37. Problems with Merging Misaligned Scans (error in ICP)Misaligned Scans (error in ICP) – Moderate misalignment Voxels rendered on top of one anotherVoxels rendered on top of one another Noisy but OKNoisy but OK

38. Problems with Merging Misaligned Scans (error in ICP)Misaligned Scans (error in ICP) – Outliers Give wrong modelGive wrong model Eliminate thin and long triangles and associated pointsEliminate thin and long triangles and associated points Can be aggressive eliminating b/c of large amt dataCan be aggressive eliminating b/c of large amt data Also rejects low quality dataAlso rejects low quality data

39. Example: Photograph 18 cm.

40. Result

41. Postprocessing Real-time displayReal-time display – Quality/speed tradeoff – Goal: let user evaluate coverage, fill holes Offline postprocessing for high-quality modelsOffline postprocessing for high-quality models – Global registration – High-quality merging (e.g., using VRIP [Curless 96])

42. Example: Photograph 18 cm.

43. Postprocessed Model

44. Summary 3D model acquisition pipeline optimized for obtaining complete, hole-free models3D model acquisition pipeline optimized for obtaining complete, hole-free models Use human’s time most efficientlyUse human’s time most efficiently Pieces of pipeline selected for real-time use:Pieces of pipeline selected for real-time use: – Structured-light scanner for moving objects – Fast ICP variant – Simple grid-based merging, point rendering

45. Limitations Prototype noisier than commercial systemsPrototype noisier than commercial systems – Could be made equivalent with careful engineering – Ultimate limitations on quality: focus, texture Scan-to-scan ICP not perfect  alignment driftScan-to-scan ICP not perfect  alignment drift – Due to noise, miscalibration, degenerate geometry – Reduced, but not eliminated, by “anchor scans” – Possibly combine ICP with separate trackers