1. Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture Matthew Flagg, Jay Summet, James M. Rehg GVU Center College of Computing Georgia Institute of Technology

2. Matthew Flagg © 2005 2 2 Ubiquitous Interactive Displays Every flat surface can be an interactive display

3. Matthew Flagg © 2005 3 3 VRP: Shadow Elimination Single Projector Case

4. Matthew Flagg © 2005 4 4 Half power shadows Shadow Elimination Double Projector Case Passive VRP

5. Matthew Flagg © 2005 5 5 Shadow Elimination Boosting projector outputs Proportional feedback law

6. Matthew Flagg © 2005 6 6 Occluder Light Suppression Detecting occluded pixels

7. Matthew Flagg © 2005 7 7 Detecting occluded pixels Occluder Light Suppression

8. Matthew Flagg © 2005 8 8 Detecting occluded pixels Occluder Light Suppression

9. Matthew Flagg © 2005 9 9 Detecting occluded pixels Occluder Light Suppression Nonlinear feedback law

10. Matthew Flagg © 2005 10 Virtual Rear Projection Show ICCV’03 demo video

11. Matthew Flagg © 2005 11 2 Challenges for VRP  High image quality  Seams between display regions projected by different projectors  Photometric Uniformity  Fast Compensation  Avoid perception of shadows caused by system lag  Image processing required to ensure high image quality

12. Matthew Flagg © 2005 12  Camera view of screen must be unobstructed  Requires reference image capture before occlusion  Cannot be co-located with projector  Shadows still perceptible  Shadow detection image processing performed on CPU Limitations With Previous Work

13. Matthew Flagg © 2005 13  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 New Approach

14. Matthew Flagg © 2005 14 New Approach  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 IR backlit camera image

15. Matthew Flagg © 2005 15  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 New Approach Turn off occluded pixels

16. Matthew Flagg © 2005 16 New Approach  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 Occluder Light Suppression

17. Matthew Flagg © 2005 17  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Shadow Eliminator  Blinding Light Suppressor  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 New Approach Turn on occluded pixels with second projector

18. Matthew Flagg © 2005 18 New Approach  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 Shadow Elimination

19. Matthew Flagg © 2005 19 New Approach  Detect Occluders, Not Shadows  Co-locate projector with camera  Active IR imaging  Based on work by Tan and Pausch CHI’02  Projector Roles:  Blinding Light Suppressor  Shadow Eliminator  Image Processing on GPU  Pixel Shaders  Render-To-Texture with DirectX9.0 Shadow Elimination and Occluder Light Suppression

20. Matthew Flagg © 2005 20 Fast Compensation: GPU-Centric Approach

21. Matthew Flagg © 2005 21 Fast Compensation: GPU-Centric Approach 1. Warping, background subtraction 2. Median filtering and dilation for inter-frame tolerance 3. Gaussian blur for blending 4. Compositing and warping

22. Matthew Flagg © 2005 22 Pixel Shader Pipeline camera texture background texture render texture 1 render texture 2 back buffer display image (A)(B)(C)

23. Matthew Flagg © 2005 23 Addressing Image Quality LAM for left projector LAM for right projector  Luminance Attenuation Maps (LAMs)  Simple feedback-based approach to accommodate non-linearities of projector- camera  Seam Blending seam – no blending seam – with blending

24. Matthew Flagg © 2005 24 Virtual Rear Projection Results Play Video

25. Matthew Flagg © 2005 25 Virtual Rear Projection Results System ComponentLatency Camera Capture to PC-Memory9.09ms PC-Memory to GPU-Memory1.70ms Pixel Shaders2.14ms Projector40.27ms Total Latency53.20ms  Image processing speed increased from 15Hz to 110Hz (camera capture rate), placing limit on the projector (85Hz refresh rate)  Projector latency accounts for 76% of total system latency!  With occluder movement tolerance of 5cm, shadows are imperceptible up to 94 cm/sec (fast walking)

26. Matthew Flagg © 2005 26 Conclusion  Presented new approach to VRP  Occluder Detection in IR spectrum  All processing moved to GPU  2 System Challenges Met  Display Image Quality  Shadow Perception Avoidance  Shadows eliminated fast enough to accommodate walking

27. Matthew Flagg © 2005 27 Future Work  Explore hardware solutions  Recent results show an LCD projector having ½ the latency of a DLP and LCOS  User Study  VRP currently used in Collaborative Design Lab in School of Aerospace Engineering  Replicate laboratory evaluation of passive VRP with new active VRP system  Improve Image Quality  Better seam blending