1. Computational Colour Vision Stephen Westland Centre for Colour Design Technology University of Leeds s.westland@leeds.ac.uk June 2005 http://www.colourtech.org Oxford Brookes University

2. Computational Colour Vision Introduce some basic concepts - the physical basis of colour Computational approaches to how colour vision works Phenomenology of colour perception (the problem) Computational and psychophysical studies of transparency perception

3. The Physical Basis of Colour C( ) = E( )P( ) The colour signal C( ) is the product at each wavelength of the power in the light source and the reflectance of the object E( ) P( ) E( )P( )

4. Cone spectral sensitivity L M S L = E( )P( ) L ( ) M = E( )P( ) M ( ) S = E( )P( ) S ( )

5. Cone Responses L = E( )P( ) L ( ) M = E( )P( ) M ( ) S = E( )P( ) S ( ) Each cone produces a univariant response L M S Colour perception stems from the comparative responses of the three cone responses Colour is a perception – the rays are not coloured

6. Colour Constancy Objects tend to retain their approximate daylight appearance when viewed under a wide range of different light sources P 0.01 0.99 Indoors (100 cd/m 2 ) 1 99 Outdoors (10,000 cd/m 2 ) 100 9900 The visual system is able to discount changes in the intensity or spectral composition of the illumination WHY? / HOW?

8. noonsunset

9. X

10. X

11. Computational Explanation L 1 = E 1 ( )P( ) L ( ) M 1 = E 1 ( )P( ) M ( ) S 1 = E 1 ( )P( ) S ( ) L 2 = E 2 ( )P( ) L ( ) M 2 = E 2 ( )P( ) M ( ) S 2 = E 2 ( )P( ) S ( ) L 1 / L 1W = L 2 / L 2W M 1 / M 1W = M 2 / M 2W S 1 / S 1W = S 2 / S 2W e 1 = De 2 D = L 1W /L 2W M 1W /M 2W S 1W /S 2W 000000 000000 e 1 = L1M1S1L1M1S1 e 2 = L2M2S2L2M2S2

12. Practical Use – Colour Correction Camera RGB values vary for a scene depending upon the light source colour correction In order to correct the images we need an estimate of the light source under which the original image was taken brightest pixel is white grey-world hypothesis

13. Colour Constancy Adaptation is too slow to explain colour constancy Any visual system that achieves colour constancy is making use of the constraints in the statistics of surfaces and lights – Maloney (1986) Is it possible for the visual system to recover the spectral reflectance factors of the surfaces in scenes from the cone responses? L = E( )P( ) L ( ) M = E( )P( ) M ( ) S = E( )P( ) S ( )

14. P w i B i ( ) Using a process such as SVD or PCA we can compute a set of basis functions B i ( ) such that each reflectance spectrum may be represented by a linear sum of basis functions - a linear model of low dimensionality. If we use n basis functions then each spectrum can be represented by just n scalars or weights. Basis Functions

15. 1 Basis Function Original1 BF P() = w 1 B 1 ()

16. 2 Basis Functions Original1 BF2 BF P() = w 1 B 1 () + w 2 B 2 ()

17. 3 Basis Functions Original1 BF2 BF3 BF P() = w 1 B 1 () + w 2 B 2 () + w 3 B 3 ()

18. About 99% of the variance can be accounted for by a 3-D model (Maloney & Wandell, 1986) But what proportion of the variance do we need to account for? How many Basis Functions are Required? 6-9 basis functions are required

19. Simultaneous Contrast

22. original original covered by filter original with small filter

23. Colour Constancy - spatial comparisons For the qualities of lights and colours are perceived by the eye only by comparing them with one another (Alhazen, 1025) … object colour depends upon the ratios of light reflected from the various parts of the visual field rather than on the absolute amount of light reflected (Marr) e i,1 /e i,2 = e' i,1 /e' i,2 (Foster) i = {L, M, S} Ratio under first light source Ratio under second light source

24. Spatial Comparison of Cone Excitations Retinex – Land and McCann (1971) Foster and Nascimento (1994) L1L1 L2L2 =k L1L1 L2L2

25. Transparency Perception (Ripamonti and Westland, 2001) e1e1 e2e2 e1e1 e2e2 e 1 /e 2 = e 1 /e 2

26. What is transparency? An object is (physically) transparent if some proportion of the incident radiation that falls upon the object is able to pass through the object.

27. What is perceptual transparency? Perceptual transparency is the process of seeing one object through another (Helmholz, 1867) Physical transparency is neither a necessary or sufficient condition for perceptual transparency (Metelli, 1974) Even in the complete absence of any physical transparency it is possible to experience perceptual transparency

28. Perceptual transparency

29. Research Questions What mechanisms could drive perceptual transparency? What are the chromatic conditions that cause transparency? Could transparency and colour constancy be linked?

30. Perceptual transparency

31. Transparency and Spatial Ratios e i,1 /e i,2 = e' i,1 /e' i,2 e i,1 e i,2 T( ) e' i,1 e' i,2

32. Experimental Computational analysis to investigate whether for physical transparency the cone ratios are preserved Psychophysical study to investigate whether the invariance of spatial ratios can predict chromatic conditions for perceptual transparency Psychophysical study to compare the performance of the ratio-invariance model when the number of surfaces is varied

33. opaque surface P T (1-b)bP 2 T 4 (1-b)PT 2 (1-b)b 2 P 3 T 6 Physical Model of Transparency b P'( ) = P( )[T( )(1-b) 2 ] 2 (Wyszecki & Stiles, 1982)

34. Monte Carlo Simulation 4. Steps 1-3 repeated 1000 times 1. A pair of surfaces P 1 ( ) and P 2 ( ) were randomly selected 2. A filter was randomly selected (defined by a gaussian distribution) m 3. The cone excitations were computed for the surfaces viewed directly (under D65) and through the filter e i,1 /e i,2 e' i,1 /e' i,2 P 1 ( ) P 2 ( )

35. e i,1 /e i,2 e' i,1 /e' i,2 i,2 e ' i,1 e i,2 ' e i,1 e Monte Carlo Results

36. i,2 e ' i,1 e i,2 ' e i,1 e Monte Carlo Results The ratios are approximately invariant Invariance is slightly better for the S cones Invariance decreases as the spectral transmittance decreases

37. Da Pos, 1989, DZmura et al., 1997 xBxB xPxP xQxQ g xAxA xPxP xBxB xQxQ x P = x A + (1- ) g x Q = x B +(1- ) g xAxA Convergence

38. (a) convergent (deviation 0) (b) invariant (deviation = 0) deviation i = 1 - [e i,1 /e i,2 ]/[ e' i,1 /e' i,2 ] Psychophysical Stimuli I

39. d' 0 indicates subjects' preference for invariant filter -3 1 3 5 00.10.20.30.40.50.6 LMS deviations d' LMSLog. (L)Log. (S)Log. (M) Psychophysical Results I

40. Psychophysical Stimuli II

41. Psychophysical Results II

42. Computational and pyschophysical studies show that the invariance of cone-excitation ratios may be a useful cue driving transparency perception Conclusions Colour constancy and transparency perception may be related. Could they result from similar mechanisms, perhaps even similar groups of neurones? There are still a mass of unsolved problems in computational colour vision including the relationship between cone excitations and the actual sensation of colour

43. xBxB xPxP xQxQ g xAxA xPxP xBxB xQxQ xAxA x P = x A + (1- ) g x Q = x B +(1- ) g x P = x A x Q = x B Cone excitations are transformed by a a diagonal matrix whose diagonal elements are all equal

44. xBxB xPxP xQxQ xAxA x P = x A x Q = x B Cone excitations are transformed by a a diagonal matrix whose diagonal elements are not necessarily all equal The two models can be made to be the same if the convergence model has no additive component and if the invariance model has equal cone scaling