1. Mixed Reality University of Central Florida Media Convergence Laboratory @ IST School of Electrical Engineering & Computer Science Digital Media Department

2. Media Convergence Laboratory Using mixed reality to solve real problems

3. The faces of MR Physical Reality (PR) – real world Virtual Reality (VR) – purely synthetic Augmented Reality (AR) – virtual assets registered in real world Augmented Virtuality (AV) – real (people, props) layered in virtual space

4. Mixed Reality Properties Blends the real and the synthetic into a single landscape & experience Addresses multiple senses Requires proper registration of real and virtual, relative to each other Typically requires complex behaviors of virtual characters Enables buy-in through imagination

5. Limitations along the continuum PR is constrained by physical space VR limits person-to-person expression and context of PR AR often limits escape from real world AV allows rich layering In general, we want to move smoothly along the MR continuum

6. Visual capture Optical Video HMDs Light capture 3d laser mocap

7. Visual rendering Dome Screens Flat World Demo Dome MR Windows HMDs

8. Audio capture / rendering Surround Hydrophones Holophone Mixing Designing Delivery in constrained settings

9. Pipeline: Audio for VR/MR Planning & Prescripting Capturing Synthesizing, Mixing, and Mastering Designing Sound & Integrating Delivering

10. Special effects - Colorkinetics SmartJack3 (USB to DMX) - Colorkinetics JuiceBox2 / iColor MR Lights - Gilderfluke MP3-50/40 - 4 Channel Dimmer Packs - Pneumatic / Smoke System - Sound Transducers (”Bass Shakers”) - Colorkinetics SmartJack3 (USB to DMX) - Colorkinetics JuiceBox2 / iColor MR Lights - Gilderfluke MP3-50/40 - 4 Channel Dimmer Packs - Pneumatic / Smoke System - Sound Transducers (”Bass Shakers”)

11. Tracking Technologies Magnetic Optical Acoustical Inertial Vision (often with markers/features) Hybrid (hardware and soft/hardware)

12. What we currently use Wide field of view: 51 degree in horizontal direction, 37 degree in vertical direction-high resolution: VGA (640 × 480) weight: 286 g standard camera I/O: NTSC Coastar (Co-Optical Axis for AR) InterTrax2 Wireless Tracker IS 900 Wide Area Tracker

13. Registration / Illumination Virtual and real must be properly placed relative to each other Inter-occlusion must be properly managed Mutual shadowing must occur, including shadows from real caused by virtual light The effects of ambient light (real and virtual) must be rendered

14. Story Virtual characters must have appropriate behaviors, reactive and proactive Virtual characters often need to mimic those of real characters Capture, replay must be provided for entertainment, after-action review, etc.

15. Putting it together

16. Content is king Create experiences that have long-term impact Experiential movie trailers Situational awareness training Creative collaboration Cognitive and physical rehabilitation Teacher screening and training Free-choice learning … Imagination is the most important dimension

17. Siggraph ‘03 – experiential trailer

18. Situational awareness MR MINI-MOUT

20. Peer Collaboration

21. Along the Continuum Start in PR (look at current plant) Move to AR (add new equipment and new windows) Individual jumps out to VR to privately review designs Move to AV as all are surrounded by new design, but still see each other

22. What We Must Support Jointly visualized “what if?” scenarios Face-to-face interactions Alternative and often opposite POVs Personal and group creation Enablers Tangible and tactile components Constructive distractions (sandbox) Shared display (1 st & 3 rd person views) Shared interfaces Scalability, interoperability & portability

23. Cognitive rehabilitation Patient and therapist in context of patient’s kitchen but in safety of clinic patient at home reconfigurable kitchen as seen thru HMD movement for after action review real items to use tethered for now

24. Pre vs. Post Milk Spoon Cereal Bowls Make Cereal Bowls Cereal Spoon Make Cereal Pre-testing HomePost-testing Home

25. Stuttering

26. Teacher screening and training Cost to hire a teacher is $15,000 Cost to remove can be much higher Turnover in inner city is 50% in 3 years Many last less than a year Children are unintended guinea pigs Want to select those who can succeed Want to help those who want to succeed

27. Front room – early prototype

28. Back room -- puppeteers

29. Interaction with a Virtual Class In front of class At desk of student Personal contact AI for all but student being directly addressed Vision-based tracking

30. Free-choice learning

32. Water’s journey Coming to a science center near you Initial focus is on the Everglades But, water is central to all ecologies Move clock backwards or forwards Real blends to virtual User-chosen policies affect what is experienced

33. Forest fire visualizations Fire spread model from Farsite Tree rendering from SpeedTree Fire and smoke visualization is ours Used in environmental economics & Everglades museum projects

34. Some basic research Blending of real and virtual in changing illumination Overcoming HMD camera limitations Real-time video chroma keying in presence of noise and inconsistent lighting LOD management (forests) Neoroevolution for virtual character behaviors Painting in HDR

35. Lab people characteristics Original director from theater and theme parks Lab manager from museum industry Chief programmer from Lego Content delivery people from digital media, children’s literature, audio, theater, museum, … Content providers from all over campus PhD students have interests in graphics, neuroevolution, vision, ecology, software engineering, simulation & modeling Undergraduates are mainly from CS and DM

36. Charles E. Hughes Professor & Associate Director School of Electrical Engineering & Computer Science Professor Digital Media Director, Media Convergence Lab E-Mail: ceh@cs.ucf.edu Home Page: http://www.cs.ucf.edu/~ceh/ Graphics Lab: http://graphics.cs.ucf.edu Media Convergence Lab: http://mcl.ucf.edu Contact information

37. J. Michael Moshell Professor Digital Media Professor School of Electrical Engineering & Computer Science Co-founder, Media Convergence Lab Home Page: http://www.cs.ucf.edu/~jmmoshell/ Media Convergence Lab: http://mcl.ucf.edu Contact information