A Framework for Perceptual Studies in Photorealistic Augmented Reality Martin Knecht 1, Andreas Dünser 2, Christoph Traxler 1, Michael Wimmer 1 and Raphael Grasset 3 2 HIT Lab NZ University of Canterbury 3 HIT Lab NZ / ICG University of Canterbury / Graz University of Technology 1 Institute of Computer Graphics and Algorithms Vienna University of Technology

Martin Knecht 1 Introduction Motivation Perceptual Issues Framework Pilot Study Limitations Future Work Conclusion

Martin Knecht 2 Motivation Is this a photograph or computer generated? What visual factors affect your decision? Rademacher P., et al. (2001) Courtesy of Rademacher et al. (2001)

Martin Knecht 3 Motivation People are very good in judging if an image looks photo-realistic or not  But process behind it not fully understood! In augmented reality (AR) virtual objects should be rendered in a photorealistic way Goal: A framework that allows us to study photorealistic rendering techniques at real-time frame rates

Martin Knecht 4 Perceptual Issues Kruijff E., et al. (2010) distinguish between two types of problems: Technological limitations Perceptual nature Proposed framework focuses on perceptual issues

Martin Knecht 5 Perceptual Pipeline Perceptual pipeline used to classify issues Kruijff E., et al. (2010) Environment Capturing Augmentation Display Device User AR Framework

Martin Knecht 6 Capturing / Augmentation Stages Capturing: Covers the process of converting the optical image to a digital one Kruijff E., et al. (2010) Image resolution, lens distortion, exposure, color correctness,… Augmentation: Deals with adding the virtual objects into the augmented scene Kruijff E., et al. (2010) Registration errors, occlusion, rendering,…

Martin Knecht 7 Framework Our contribution: Framework to study photorealistic rendering techniques Interactive tasks with global illumination Three different rendering modes Fast prototyping of experiments (XML) A pilot study to test framework

Martin Knecht 8 Framework Differential instant radiosity is used to render the scenes Knecht M., et al. (2010) Rendering modes can be changed during run- time Mode A: shadows / indirect illumination Mode B: shadows / indirect illumination Mode C: shadows / indirect illumination

Martin Knecht 9 Rendering Mode A Different rendering modes are supported No shadows No indirect illumination

Martin Knecht 10 Rendering Mode B Different rendering modes are supported With shadows No indirect illumination

Martin Knecht 11 Rendering Mode C Different rendering modes are supported With shadows With indirect illumination

Martin Knecht 12 Pilot Study Pilot study to evaluate our framework Twenty-one participants (15 male, 6 female) Age between 19 – 59 5 tasks

Task 1: Estimate distance between real and virtual cubes Martin Knecht 13 Pilot Study

Task 2: Place real cube at position of virtual one Martin Knecht 14 Pilot Study

Task 3: Place virtual cube at position of real one using computer keyboard Martin Knecht 15 Pilot Study

Task 4: Grab and lift real cube Martin Knecht 16 Pilot Study

Task 5: Grab and lift virtual cube - real cube was used for tactile feedback Martin Knecht 17 Pilot Study

Martin Knecht 18 Pilot Study Duration: 30 to 60 minutes Interview followed afterwards Measured Distance error for tasks 1, 2, 3 Time for tasks 2, 3, 4, 5 Analyzed data using non-parametric Friedman tests Result: No significant effect of rendering technique found

Martin Knecht 19 Pilot Study

Martin Knecht 20 Pilot Study

Martin Knecht 21 Pilot Study Distance estimation perceived easier by 6 participants in left/right and height direction than in depth Although no significant effect was found

Martin Knecht 22 Pilot Study Participants made heavy use of occlusion cue to place cubes in Task 2 & 3 – (19 out of 20)

Martin Knecht 23 Pilot Study 7 participants preferred manipulation with a computer keyboard (Task 2 vs. Task 3).

Martin Knecht 24 Limitations No support for stereo rendering No distance information of environment Only video see-through HMDs are supported Tone-mapping still needs manual fine tuning No support for mobile devices

Martin Knecht 25 Future Work Enhance rendering framework Support for stereo rendering Add camera artifacts New studies with new tasks Tasks without occlusion information Use chin rest to reduce movement Design new tasks

Martin Knecht 26 Conclusion We proposed a research test-bed for perceptual studies – still work in progress Different rendering modes allow new kind of experiments We showed the results of a pilot study using our framework

Thank you for your attention! What features would you like to have in such a framework? 27 Martin Knecht