1. Applied Perception in Graphics Erik Reinhard University of Utah reinhard@cs.utah.edu

2. Computer Graphics Produce computer generated imagery that cannot be distinguished from real scenesProduce computer generated imagery that cannot be distinguished from real scenes Do this in real-timeDo this in real-time

3. Trends in Computer Graphics Greater realismGreater realism –Scene complexity –Lighting simulations Faster renderingFaster rendering –Faster hardware –Better algorithms Together: still too slow and unrealisticTogether: still too slow and unrealistic

4. Algorithm design Largely opportunisticLargely opportunistic Computer graphics is a maturing fieldComputer graphics is a maturing field Hence, a more directed approach is neededHence, a more directed approach is needed

5. Long Term Strategy Understand the differences between natural and computer generated scenesUnderstand the differences between natural and computer generated scenes Understand the Human Visual System and how it perceives imagesUnderstand the Human Visual System and how it perceives images Apply this knowledge to motivate graphics algorithmsApply this knowledge to motivate graphics algorithms

6. This Presentation (1) Reinhard et. al., “Color Transfer between Images”, IEEE CG&A, sept. 2001.

7. This Presentation(2) Reinhard et. al., “Photographic Tone Reproduction for Digital Images, SIGGRAPH 2002.

8. Introduction The Human Visual System is evolved to look at natural images NaturalRandom

9. Human Visual System

10. Retina

11. Color Processing Rod and Cone pigments

12. Color Processing Cone output is logarithmic Color opponent space

13. Image Statistics Ruderman’s work on color statistics:Ruderman’s work on color statistics: –Principal Components Analysis (PCA) on colors of natural image ensembles –Axes have meaning: color opponents (luminance, red-green and yellow-blue)

14. Color Processing Summary Human Visual System expects images with natural characteristics (not just color)Human Visual System expects images with natural characteristics (not just color) Color opponent space has decorrelated axesColor opponent space has decorrelated axes Color space is logarithmic (compact and symmetrical data representation)Color space is logarithmic (compact and symmetrical data representation) Independent processing along each axis should be possible  ApplicationIndependent processing along each axis should be possible  Application

15. Color Transfer Make one image look like anotherMake one image look like another For both images:For both images: –Transfer to new color space –Compute mean and standard deviation along each color axis Shift and scale target image to have same statistics as the source imageShift and scale target image to have same statistics as the source image

16. L Color Space Convert RGB triplets to LMS cone space Convert RGB triplets to LMS cone space Take logarithm Take logarithm Rotate axes Rotate axes

17. Why not use RGB space? Input imagesOutput images RGB L

18. Color Transfer Example

21. Color Processing Summary Changing the statistics along each axis independently allows one image to resemble a second imageChanging the statistics along each axis independently allows one image to resemble a second image If the composition of the images is very unequal, an approach using small swatches may be used succesfullyIf the composition of the images is very unequal, an approach using small swatches may be used succesfully

22. Tone Reproduction

24. Global vs. Local GlobalGlobal –Scale each pixel according to a fixed curve –Key issue: shape of curve LocalLocal –Scale each pixel by a local average –Key issue: size of local neighborhood

25. Global Operators TumblinWard Ferwerda

26. Global Operators TumblinWard Ferwerda

27. Local Operator Pattanaik

28. Spatial Processing Light reaches the retina and is detected by rods and conesLight reaches the retina and is detected by rods and cones The number of rods and cones is much larger than the number of nerves leaving the eyeThe number of rods and cones is much larger than the number of nerves leaving the eye Hence, data reduction occurs in the retinaHence, data reduction occurs in the retina

29. Spatial Processing Certain aspects of natural images are more important than othersCertain aspects of natural images are more important than others For example, contrast edges need to be detected with accuracy, whereas slow gradients do not need to be perceived at high resolutionFor example, contrast edges need to be detected with accuracy, whereas slow gradients do not need to be perceived at high resolution

30. Spatial Processing Circularly symmetric receptive fieldsCircularly symmetric receptive fields Centre-surround mechanismsCentre-surround mechanisms –Laplacian of Gaussian –Difference of Gaussians –Blommaert Scale space modelScale space model

31. Scale Space (Histogram Equalized Images)

32. Tone Reproduction Idea Modify existing global operator to be a local operator, e.g. Greg Ward’sModify existing global operator to be a local operator, e.g. Greg Ward’s Use spatial processing to determine a local adaptation level for each pixelUse spatial processing to determine a local adaptation level for each pixel

33. Blommaert Brightness Model Gaussian filter Center/surround Neural response Brightness

34. Brightness

35. Scale Selection Alternatives Mean value Thresholded How large should a local neighborhood be?

36. Mean Value

37. Thresholded

38. Tone-mapping Local adaptation Greg Ward’s tone- mapping with local adaptation

39. Results Good results, but something odd about scale selection:Good results, but something odd about scale selection: For most pixels, a large scale was selectedFor most pixels, a large scale was selected Implication: a simpler algorithm should be possibleImplication: a simpler algorithm should be possible

40. Simplify Algorithm Greg Ward’s tone- mapping with local adaptation Simplify Fix overall lightness of image

41. Global Operator Results WardOur method

42. Global Operator Results WardOur method

43. Global  Local Global operator Local operator

44. Local Operator Results Global Local

45. Local Operator Results GlobalLocalPattanaik

46. Summary Knowledge of the Human Visual System can help solve engineering problemsKnowledge of the Human Visual System can help solve engineering problems Color and spatial processing investigatedColor and spatial processing investigated Direct applications shownDirect applications shown

47. Ongoing Research Natural Image StatisticsNatural Image Statistics Applications:Applications: –Reconstruction filters –Perlin noise –Fractal terrains

48. Ongoing Research Impoverished environments

49. Future Work This presentation

50. Acknowledgments Thanks to my colaborators: Peter Shirley, Jim Ferwerda, Mike Stark, Mikhael Ashikhmin, Bruce Gooch, Tom TrosciankoThanks to my colaborators: Peter Shirley, Jim Ferwerda, Mike Stark, Mikhael Ashikhmin, Bruce Gooch, Tom Troscianko This work sponsored by NSF grants 97- 96136, 97-31859, 98-18344, 99-78099 and by the DOE AVTC/VIEWSThis work sponsored by NSF grants 97- 96136, 97-31859, 98-18344, 99-78099 and by the DOE AVTC/VIEWS