1. Visual Perception in Realistic Image Synthesis Ann McNamara

2. Outline Introduction Modeling important characteristics of the human visual system (HVS) Perception based rendering Image quality metrics Tone reproduction operators Summary

3. Realism Architecture Stage lighting Entertainment Safety systems Archaeology

4. Human Visual System Physical structure well established Perceptual behaviour is a complex process

5. Modeling Important Characteristics of the Human Visual System

6. Visual Acuity How well we can see fine detail Adaptation level Rods and cones Cones

7. Number of grating that fall on one degree of the retina Dependent on distance Spatial Frequency

8. Spatial mechanisms (channels) which are used to represent the visual information at various scales and orientations as it is believed that primary visual cortex does.

9. Contrast Sensitivity Function Contrast sensitivity function which specifies the detection threshold for a stimulus as a function of its spatial frequencies.

10. Campbell-Robson contrast sensitivity chart Contrast Sensitivity

11. Masking Visual masking affecting the detection threshold of a stimulus as a function of the interfering background stimulus which is closely coupled in space and time.

12. Masking

13. Colour Appearance

14. Perceptually Based Rendering

15. ] [Mitchell 1987] Low Sampling Densities Non-uniform sampling is less conspicuous Optimise using how the eye perceives noise as a function of contrast and colour Raytracing -> Point Samples-> Aliasing

16. Uniform Non-Uniform Adaptive Sampling Schemes ] [Mitchell 1987]

17. Low Sampling Densities Contrast Colour R 0.4 G 0.3 B 0.6

18. ] [Mitchell 1987] Low Sampling Densities

19. Frequency Based Raytracing ] [Bolin &Meyer 1992] Synthesise directly into frequency domain Simple vision model to control Where to cast rays How to spawn rays

20. Frequency Based Raytracing ] [Bolin &Meyer 1992] Vision model Contrast sensitivity Spatial frequency Masking

21. Frequency Based Raytracing ] [Bolin &Meyer 1992] Specific luminance difference at low intensity more important than same luminance difference at high intensity Colour spatial frequency variations given fewer samples Decrease rays spawned in high frequency regions

22. Limited Color Acuity ] [Meyer & Liu1998] Colour Abberation Limited sampling of receptor Spatial acuity of opponent channels

23. ] [Meyer & Liu1998]

24. Application How much computation is enough? How much reduction is too much? An objective metric of image quality which takes into account basic characteristics of the HVS could help to answer these questions without human assistance.

25. Questions of Appearance Preservation The Concern Is Not Whether Images Are the Same Rather the Concern Is Whether Images Appear the Same

26. Perceptually Based Adaptive Sampling Algorithm ] [Bolin &Meyer 1998] Image quality model embedded into image synthesis Use statistical information about spatial frequency to determine where to estimate values where samples were yet to be taken

27. Perceptually Based Adaptive Sampling Algorithm ] [Bolin &Meyer 1998] JND’s VDM

28.  = 200s  = 400s  = 800s  = 1600s Deterministic radiosity Monte Carlo radiosity Convergence Evaluation ] [ Myszkowski 1997]

29.  vs. reference 0.5  vs.  ] [ Myszkowski 1997] Termination Criterion

30. Physical Based Perceptual Metric ] [Ramasubramanian et al1999] Threshold model defines a physical error metric Handles luminance-dependent and spatially dependent processing independently Allowing pre-computation of spatially- dependent component

31. Physical Based Perceptual Metric ] [Ramasubramanian et al1999]

32. Image Quality Metrics

33. Image Quality Compare and validate lighting simulations Use comparisons to guide rendering more efficiently Compute less without altering perception Pixel by pixel comparison might be > 0, human might not see any difference

34. RMSE 9.5 RMSE 5.2 Pixel by Pixel Comparison Prikryl, 1999

35. Visible Differences Predictor VDP Image 2 Image 1 Psychometric Function Probability Summation Visualisation of Differences Amplitude Nonlinear. Amplitude Nonlinear. Contrast Sensitivity Function Contrast Sensitivity Function + Cortex Transform Cortex Transform Masking Function Masking Function Unidirectional or Mutual Masking [ Daly ‘93, Myszkowski ‘98]

36. VDP: Results StandardComparison Pixel differences: Standard - Comparison Pixel differences The VDP response: probability of perceiving the differences VDP response

37. Daly’s VDP: Features Predicts local differences between images Takes into account important visual characteristics: Amplitude compression Advanced CSF model Masking Uses the cortex transform, which is a pyramid-style, invertible & computationally efficient image representation Daly, 1993

38. Visible Discrimination Model Map of Just Noticeable Differences Point sample function to model optics Resample the image according to foveal eccentricity Band pass response Contrast pyramid steerable filters Lubin, 1997

39. Visible Discrimination Model Both images subjected to Identical processing Distance measure Difference in responses for Each channel and summing Them to obtain a JND Map of the two images Lubin, 1997

40. An Experimental Evaluation of Computer Graphics Imagery Comparing image to real-world scene An approach to image synthesis consisting of A physical module A perceptual module ] [Meyer et al, 1986]

41. An Experimental Evaluation of Computer Graphics Imagery ] [Meyer et al, 1986] DifferenceSimulatedMeasured

42. An Experimental Evaluation of Computer Graphics Imagery ] [Meyer et al, 1986]

43. An Experimental Evaluation of Computer Graphics Imagery ] [Meyer et al, 1986]

44. Image Quality Metrics ] [Rushmeier et al, 1995] Components of perceptually based metrics adapted from image compression Gervais et al 1984 Mannos et al 1974 Daly 1993

45. Image Quality Metrics ] [Rushmeier et al, 1995] Daly tested very well Real Room Simulated Model of Room

46. Visual Psychophysics Determine the relationship between the physical world and human’s subjective experience of that world Measure the response (“psycho”) to a known stimulus (“physics”)

47. Why Lightness ? ] [Gilchrist 1977 ]

48. ] [McNamara et al 1998, 2000] A Psychophysical Investigation Painted 5-sided cube Objects painted with different grey paints Complex illumination, with secondary reflections

49. Graphic Reconstructions ] [McNamara et al 1998, 2000]

50. Experiment Rendered Real Scene ] [McNamara et al 1998, 2000 ]

51. Results ] [McNamara et al 1998, 2000]

52. Tone Reproduction Operators

53. ~10 5 cd/m 2 ~10 -5 cd/m 2 Tone Reproduction ~100 cd/m 2 ~1 cd/m 2 Same Visual Response ?

54. Tone Reproduction for Realistic Images Mapping between radiances computed and light energy emitted from CRT Psychophysical model of brightness perception Observer model Display model Tumblin & Rushmeier, 1993

55. Tone Reproduction

56. Tone Reproduction for Realistic Images http://graphics.cs.uni-sb.de/~slusallek/Doc/html/node14.html LowMediumHigh Tumblin & Rushmeier, 1993

57. A Contrast-based Scalefactor for Luminance Display http://graphics.cs.uni-sb.de/~slusallek/Doc/html/node14.html Linear transform L d = mL W Matching contrast between real and image Ward, 1994

58. A Contrast-based Scalefactor for Luminance Display http://graphics.cs.uni-sb.de/~slusallek/Doc/html/node14.html Min-MaxWard Ward, 1994

59. A Model of Visual Adaptation for Realistic Image Synthesis Threshold visibility Changes in colour appearance Visual acuity Temporal Sensitivity Ferwerda et al, 1996

60. A Model of Visual Adaptation for Realistic Image Synthesis Ferwerda et al, 1996

61. Spatially Nonuniform Scaling for High Contrast Images Incorrect to apply the same mapping to each pixel Spatial position Chiu et al, 1993

62. Quantization Techniques for Visualization of High Dynamic Range Pictures Similar to Chiu et al Rational rather than logarithmic Accounts for the non-linearities of both the display device and human perception The biggest advantages is speed Schlick, 1994

63. A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes Preserve visibility of objects Histogram - adjusted to minimise the visible contrast distortions Also includes glare, colour sensitivity, and acuity Larson et al, 1997

64. A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes Larson et al, 1997

65. Perceptually Driven Radiosity ] [Gibson & Hubbold. 1997] Steer computation to areas in need of most refinement A-priori estimate adaptation luminance Tone-mapping to transform luminance to display Distance between two colors in uniform colour space = numerical measure of perceived difference

66. Perceptually Driven Radiosity ] [Gibson & Hubbold. 1997] Stop patch refinement once the difference between successive levels becomes perceptually unnoticeable Determine the perceived importance of any shadow Optimise the mesh for faster interactive display and minimise storage

67. Standard shadow testing (19.33 hours) Perceptually-driven shadow testing (3.10 hours) ] [Gibson & Hubbold. 1997 ] Shadow Testing

68. Output mesh Optimised mesh

69. ] [ Hedley et al. 1997 ] Discontinuity Meshing Throw out discontinuities that are deemed visually unimportant Tone mapping Compare colour differences along the discontinuity line

70. Culled discontinuities Original scene

71. Summary Applications of visual perception in computer graphics Efficient software Image quality evaluations Tone reproduction operators Knowledge of HVS can be used to greatly benefit the synthesis of realistic images at various stages of production

72. Conclusion Great deal of potential Perceptually accurate as well as physically correct Allow high level of confidence in computer imagery allowing us to demonstrate to the world that our images are faithful representations !

73. Thank You Ann McNamara Ann.McNamara@tcd.ie http://www.cs.tcd.ie/ann.mcnamara

74. Extra…

75. Spatial and Orientation Mechanisms The following filter banks are commonly used: Gabor functions (Marcelja80), Steerable pyramid transform (Simoncelli92), Discrete Cosine Transform (DCT), Difference of Gaussians (Laplacian) pyramids (Burt83,Wilson91), Cortex transform (Watson87, Daly93).

76. Cortex Transform: Organization of the Filter Bank

77. Orientation Bands Cortex Transform: Orientation Bands Input image

78. Spatiovelocity CSF Contrast sensitivity data for traveling gratings of various spatial frequencies were derived in Kelly’s psychophysical experiments (1960). Daly (1998) extended Kelly’s model to account for target tracking by the eye movements. log visual sensitivity log velocity [deg/sec] log spatial frequency [cycles/deg] Temporal frequency [Hz]

79. Visual Masking Masking is strongest between stimuli located in the same perceptual channel, and many vision models are limited to this intra-channel masking. The following threshold elevation model is commonly used: