School of Computer Science and Software Engineering Design Issues in Human Visual Perception Experiments on Region Warping Monash University Yang-Wai Chow Ronald Pose Matthew Regan

Overview Background  Address Recalculation Pipeline  Priority Rendering  Large object segmentation  Scene tearing artefacts  Region Warping Related Work  Image/Video compression  Level-of-Detail (LOD) management  Visual attention models Design Issues Conclusions

Background  Address Recalculation Pipeline  Priority Rendering  Large object segmentation  Scene tearing artefacts  Region Warping

Latency is a major factor that plagues the designing of immersive Head Mounted Display (HMD) virtual reality systems End-to-end latency is defined as the time between a user’s actions and when those actions are reflected by the display The Address Recalculation Pipeline (ARP) was designed to reduce the end-to-end latency due to user head rotations for immersive Head Mounted Display (HMD) virtual reality systems The Address Recalculation Pipeline (ARP) Average latency to head rotations WITHOUT pipeline Average latency to head rotations WITH pipeline Head Tracking Image Creation Buffer Swap Image Valid Head Tracking Image Valid

Priority Rendering allows different section of the scene that surrounds the user’s head to be rendered onto separate display memories and therefore can updated at different update rates Different section of the scene can be rendered onto separate display memories before being combined to form an image of the whole scene Priority Rendering

Segmentation of large virtual world object in conjunction with Priority Rendering can further reduce the overall rendering load Fractal terrain example – A fractal terrain typically consists of thousands of polygons. If the terrain were to be segmented for priority rendering, different sections of the fractal terrain could be updated at different update rates Large object segmentation

However the implementation of object segmentation in conjunction with priority rendering introduces a potential scene tearing problem Scene tearing artefacts Scene tearing artefacts will completely destroy the illusion of reality, and therefore have to be addressed before object segmentation can be used effectively with priority rendering

Region Warping was designed to hide the scene tearing artefacts resulting from object segmentation with Priority Rendering Region Warping Region Warping essentially involves the perturbation of object vertices in order to hide the tearing artefacts

An example showing the scene tearing effect Scene tearing

The exact same frame, this time with Region Warping Region Warping results

Initial Region Warping experiments and analysis do not portray the human perception or response to the warping Some initial experiments on region warping have been conducted (Please refer to our paper entitled “Region Warping in a Virtual Reality System with Priority Rendering” – IADIS Applied Computing 2005) However the analysis that was performed were mathematical analysis of the distortions and did not portray the human visual perception to region warping Therefore further experimentation involving human subjects are necessary Initial region warping experiments and analysis

Related Work  Image/Video compression  Level-of-Detail (LOD) management  Visual attention models

A number of other computer graphics techniques have also been developed that take advantage of the limitations in the psychophysical aspects of the human visual perception The human visual system has a finite perception to detail A variety of computer graphics techniques attempt to reduce the computational effort and time required in the rendering of scenes or generating of images without greatly affecting a user’s perception of the displayed image quality Examples of human visual perception exploits can be found in lossy image/video compression techniques JPEG and MPEG coding are lossy compression techniques that attempt to decrease the amount of data in the representation of an image/video Related work

Level of Detail (LOD) management used in computer graphics attempts to decrease the amount of detail in a scene without the user noticing the difference Level of Detail (LOD) management have been used by degrading the peripheral detail in a HMD virtual reality system’s display, with no adverse effects to the user task performance LOD methods also include the use of object meshes with varying degrees of complexity A number of criteria for switching between varying LODs have been proposed and implemented Simpler object meshes can be switched into the display without the user noticing any alterations in the scene Level of Detail (LOD)

Others have experimented with visual attention models to selectively concentrate computational effort on important sections of a scene A user will typically focus on prominent or conspicuous objects in a scene and will pay less attention to the details contained in the rest of the environment This is a fundamental feature of the human visual system known as inattention blindness Some researchers have conducted visual perception experiments based on visual attention models in order to predict where the user will be looking in a scene and focus rendering effort on those sections of the scene Visual attention models

Design Issues  Stereoscopy versus Monoscopy  Distance from the user  Head tracked displays  Scene complexity  Virtual environment characteristics  Scene familiarity  Ability to interact with surrounding environment

As part of the human visual system, stereo vision is one of the main factors that contribute to a user’s perception of realism within a virtual world The human visual system is based on stereo vision A human perceives information such as depth based on the disparity between left and right eyes It is therefore conceivable that a user might perceive differently the effects of the distortions caused by region warping in a virtual world viewed through a stereoscopic display or a monoscopic display Stereoscopy versus monoscopy

A human perceives less detail in objects that are located further away compared to closer objects Objects that are further from the users will appear to be smaller from the user’s viewpoint Human binocular disparity/stereo perception fails after a distance of more than 30 meters It is therefore possible that distortions that are concentrated at a distance away from the user will be less noticeable Distance from the user

Upon user head movements or rotations, based on feedback from the motoric senses the human brain will expect the scene to change in certain ways The human eye is less sensitive to details that move rapidly over the retina Perception to distortion might vary between head tracked virtual reality systems and non-head tracked systems Head tracked displays

The human visual system depends on the retina to cope with the wealth of information contained within the scene The user visual attention will typically be focus or drawn to certain interesting areas of the scene High levels of detail contained within the scene will distract and possibly overwhelm the user’s perception Scene complexity

The makeup of the virtual world might also potentially affect a user’s perception to the region warping distortions Distortions will appear less obvious on scenes containing random nature objects compared to scenes containing rigid structured architectures The reason for this is because highly structured scenes contain straight lines can might noticeably bend as a result of the warping Virtual environment characteristics

The human brain might unconsciously correct perceived errors when presented with a familiar scene The human visual system is governed by real world experiences, as the brain interprets images presented to the retina based on these experiences The human brain might overlook errors contained in familiar scene Alternatively the converse might apply, in that errors might appear more prominent when presented with a familiar scene Scene familiarity

A user’s focus of attention will be divided with allowed to interact with the surrounding environment Interactability with the surrounding environment adds to the user’s feeling or sense of presence in the virtual world When a user is immersed in a virtual world, his/her mindset and focus of attention will be drawn towards trying to manipulate objects or to navigate through the virtual world Ability to interact with surrounding environment

Region warping attempts to take advantage of the limitations in the human visual perception, to reduce the overall rendering load of a ARP virtual reality system with priority rendering However while region warping appears to works well on computer monitors and perform well by analytical measures, the real test can only be with real human perception Therefore before this technique can be used effectively, it is important to characterize and understand the level of distortions that might be perceivable to the user This paper has outlined our work and future approaches Conclusions

Questions or Suggestions?