1. Real-time Structured Light Depth Extraction Kurtis Keller & Jeremy Ackerman Department of Computer Science University of North Carolina Chapel Hill, North Carolina, USA EI2000

2. Divisions of talk  Application and reason for research  Comparisons of various, relevant depth extraction systems (structured light, etc..)  First design with large digital projector  Second design with real time custom projector  Improvements...

3. High speed structured light depth extraction for Augmented reality surgical procedures

4. Application: Medical l Virtual / Augmented Reality view from single incision. l This information can be rendered with textures to surgeon’s viewpoint. l Must be real-time, small, able to image shiny/curved surfaces.

5. Comparison of various, relevant Depth Extraction Systems l Stereo laparoscopes l Traditional viewing of images l Automated depth extraction poor l Structured light depth Extraction l Too slow with standard projector and camera l Can be used on curved surfaces l Laser scan l Too slow for real time use l Miniaturization

6. Surface Texture and Z depth scanning methods for real-time medical use

7. Structured light (Binary encoded) Start with coarse projected texture, then increase number of bars for finer resolution. 1 2 3

8. First System

9. First Design: Structured Light with large digital projector

10. First System v Not Real time v Utilized off the shelf (modified) components l TI micromirror projector Custom lens to flex scope to laparoscope Custom lens to flex scope to laparoscope v 3D scans in lab and animal cadaver tests v Bulky

11. First System ( in operation) v Close to Real time (60Hz frame imaging, 5Hz updates) v Custom components l Projector Lens Custom lens to flex scope to laparoscope Custom lens to flex scope to laparoscope v Off the shelf componets l TI Micromirror projector l Laparoscope Pulnics progressive scan camera Pulnics progressive scan camera

12. Second System (real time)

13. Second System v Almost Real time (~10 Hz) v Utilized custom components l Displaytech ferro-reflective LCD engine in custom projector l Custom lens to flex scope to laparoscope v Fits in with standard surgical environment l Custom projector snap onto standard medical laparoscope l Light source 2 meters away from patient by fiber light Projector about the size of an orange, fitting to surgeons hand and light weight Projector about the size of an orange, fitting to surgeons hand and light weight l All exposed surfaces surgical steel and autoclaveable.

14. High-speed Projector v Miniature size (50mm Dia x 75mm long) v Custom components l Displaytech 640x480 180Hz. engine l Custom lens to laparoscope

15. High-speed Miniature Projector Attached to Laparoscope Attached to Laparoscope Exploded view Exploded view

16. v Real time v High-speed components l Custom, 180Hz projector l 180+ Hz 512x512 pixel camera l High speed graphics processor

17. Hardware Improvements l Color additive scanning by synchronized color wheel l Single tube Stereo Laparoscope l Reduction of Specular rejection problems l Increased light level bit depth l Image processing/rejection Color Wheel

18. FUTURE RESEARCH in HIGH SPEED STRUCTURED LIGHT l Obtain Ferro reflective (or other) displays which are not hampered by crystal migration l If possible, purchase graphics processor and high speed camera from same vendor l Full projection imaging is being performed, not micro internals that we emphasized. l Virtual Teleconferencing l Inspection l Surveillance