1. A Distributed Cooperative Framework for Continuous Multi- Projector Pose Estimation IEEE VR 2009 - March 16, 2009 Tyler Johnson, Greg Welch, Henry Fuchs, Eric La Force, and Herman Towles Department of Computer Science University of North Carolina at Chapel Hill

2. A Distributed Cooperative Framework for Continuous Calibration2 Funding ONR: Behavior Analysis and Synthesis for Intelligent Training (BASE-IT), Dr. Roy Stripling, Program Manager ONR: Virtual Technologies and Environments Program (VIRTE), CDR Dylan Schmorrow, Program Manager ONR-NAVAIR: Deployable Intelligent Projection Systems for Training, SBIR contract with Renaissance Sciences Corporation IARPA: Mockup Future Analyst Workspace (A-Desk), Dr. Jeff Morrison, Program Manager NSF: Integrated Projector-Camera Modules for the Capture and Creation of Wide-Area Immersive Experiences, CRI:IAD grant

3. A Distributed Cooperative Framework for Continuous Calibration3 Adaptive Multi-Projector Displays An Intelligent Projector Unit (IPU)

4. A Distributed Cooperative Framework for Continuous Calibration4 Challenges Geometric Compensating for display surface shape Co-registration of projection images Photometric Intensity blending in image overlaps Matching colors between projectors No Compensation Compensation

5. A Distributed Cooperative Framework for Continuous Calibration5 Geometric Calibration Calibrate Projectors Estimate Display Surface Before Display Use Project Structured Light During Display Use Geometric Image Correction Render Imagery Estimate Display Surface Calibrate Projectors Concurrently Continuous Calibration

6. A Distributed Cooperative Framework for Continuous Calibration6 Continuous Calibration A Calibrated Two-Projector Display Projectors Moved or Bumped A Recalibrated Two-Projector Display

7. A Distributed Cooperative Framework for Continuous Calibration7 Cooperative Calibration We propose a distributed, Kalman filter-based approach to continuous calibration where intelligent projector units interact as peers to cooperatively estimate the poses (orientation & position) of all projectors during actual display use

8. A Distributed Cooperative Framework for Continuous Calibration8 Related Work Continuous Calibration Active (Calibration Patterns) Imperceptible Structured Light [Cotting04,05] Passive (Application Imagery) Continuous Display Surface Estimation [Yang&Welch01] Single Projector Pose Estimation[Johnson&Fuchs07] Multi-Projector Pose Estimation [Zhou08] Hybrid Automatic switch from passive to active [Zollmann06] Distributed Upfront Calibration [Bhasker06]

9. A Distributed Cooperative Framework for Continuous Calibration9 Contributions Our Kalman filter-based distributed cooperative framework provides Continuous pose estimation for multiple projectors Compatible with both active and passive feature collection All projectors may move simultaneously Temporal filtering Fault tolerance & scalability

10. A Distributed Cooperative Framework for Continuous Calibration10 Cooperative Calibration Peer-to-Peer based, Single Program, Multiple Data Each IPU considers itself to be the “local” IPU Other IPUs are considered “remote” IPUs Each IPU is responsible for calculating its own pose using local and remote correspondences Assumptions The internal calibration of each IPU is fixed and known The geometry of the display surface is static and known Projectors remain mostly stationary, however they may drift over time or occasionally be moved by the user

11. A Distributed Cooperative Framework for Continuous Calibration11 Local Correspondences Measured for each IPU between its primary and secondary cameras Provides an estimate of pose

12. A Distributed Cooperative Framework for Continuous Calibration12 Remote Correspondences Measured between the primary camera of the local IPU and a remote IPU Remote IPU acts as a reference in estimating pose of local IPU

13. A Distributed Cooperative Framework for Continuous Calibration13 Collection of Measurements Display Surface

14. A Distributed Cooperative Framework for Continuous Calibration14 Pose of Kalman Filter Display Surface Model Pose of Intrinsics of Measurements in Predicted Measurements in Measurements in Measurement Function Estimate, using

15. A Distributed Cooperative Framework for Continuous Calibration15 Kalman Filter Error Covariance State Pose of

16. A Distributed Cooperative Framework for Continuous Calibration16 Filter Update Predict IPUs remain stationary Add additional uncertainty Time Update Measurement Update Correct state based on residual Measurement Jacobian Measurement Noise (t)

17. A Distributed Cooperative Framework for Continuous Calibration17 Distributed Operation Each IPU has local access to Its own intrinsic calibration & pose Its own camera images Display surface model Kalman filter update requires remote access to Intrinsic calibration & poses of remote IPUs Error & process noise covariance of remote IPUs Images from primary cameras of remote IPUs captured at time

18. A Distributed Cooperative Framework for Continuous Calibration18 Distributed Operation Request/response mechanism for exchanging camera images, pose information etc

19. A Distributed Cooperative Framework for Continuous Calibration19 Results x y z Before Movement During Movement After Movement Ψ θ φ mm rad 1240 1160 -95 -120 2640 2560 2.85 2.7 -0.3 -0.7 0.35 0.15

20. A Distributed Cooperative Framework for Continuous Calibration20 Video Distributed Cooperative Pose Estimation in a Two-IPU display P P

21. A Distributed Cooperative Framework for Continuous Calibration21 Discussion Observability & Drift Surface geometry may not fully constrain pose Possible to “anchor” solution in unobservable directions Cooperative Estimation Not required for a working system Ensures imagery is registered between projectors, especially when pose may be unobservable P P

22. A Distributed Cooperative Framework for Continuous Calibration22 Future Work Continuous calibration of display surface Dynamic projector refocusing Dynamic photometric blending Improve scalability

23. A Distributed Cooperative Framework for Continuous Calibration23 Future Applications

24. A Distributed Cooperative Framework for Continuous Calibration24 In Conclusion