NATURAL COLOR CONSTANCY Qasim Zaidi SUNY College of Optometry Hannah Smithson University College London Byung-Geun Khang Utrecht University

COLOR CONSTANCY [Lichtenberg, Letter to Goethe on “Farbige Schatten”, 1793] The objects we call white are the ones we infer would be colorless if they were illuminated by a colorless light. In reality all illuminants are spectrally selective, especially inter-reflected light. Dueling definitions: Constancy of subjective appearance Inference of physical invariants Identification strategies: Under limited conditions, appearance is constant. 1st-order similarity (between colors) is useful for identification of materials/objects. In general, appearance is not constant. Observers use 2nd-order similarity (between color differences) to identify materials. If neither similarity is applicable, accurate identification is not possible

Physical basis of color constancy? For natural reflectance spectra, the “color conversion” (Helson) between two illuminant conditions is simply multiplicative when expressed in terms of cone- coordinates, hence rank-orders are preserved (Dannemiller; Nascimento & Foster; Zaidi, Spehar & DeBonet).

Requirements for 1st-order similarity based color constancy To achieve a constant color of an object despite changes in the spectrum of the light reflected from that object, reverse the color conversion to “transform” (Helson) the perceived colors of objects under a test illuminant towards the colors of objects under a reference illuminant: Adaptation to illuminant coordinates (Ives) Rank orders anchored across a reference material (Helson)

In post-receptoral cordinates, the “color conversion” is translational for L/(L+M), and multiplicative for S/(L+M). For identification based on 1 st -order similarity: Adaptation to illuminant coordinates Rank orders anchored across the mean Will be as effective at a post-receptoral level as at receptors.

Is there constancy of color appearance under adaptation to single illuminants? Observer’s task: Session 1: Classify the appearance of square patch (3 deg) as red or green Session 2: Classify the appearance of square patch as yellow or blue

Sunlight illuminant Skylight illuminant EXPERIMENT 1 Tests (circles) Chromatically balanced background (plusses) L/(L+M) S/(L+M)

Fixed trial duration 1.5 seconds Time Procedure for each illuminant and background set: Initial adaptation period 80 trials, no response Trials continue until a response has been collected for all materials

EXPERIMENT 1 Single-trial classifications obtained from one observer L/(L+M) S/(L+M) “Red / Green” judgement“Blue / Yellow” judgement

ReflectancesChromaticities HES HS JEM L/(L+M) S/(L+M) S/(L+M) S/(L+M) HES HS JEM L/(L+M) EXPERIMENT 1 Classification boundaries in chromaticity & reflectance spaces Red lines: boundaries under sunlight Blue lines: boundaries under skylight. Circles: corresponding illuminants Illuminant has a large effect on the location of color boundaries in chromaticity space, but similar reflectances fall on color boundaries under both illuminants. Constancy indices 0.87, 0.68, 0.93 for the three observers. Appearance-based color constancy could result from spatially extended adaptation, or illuminant estimation using the mean.

EXPERIMENT 2 Single illuminants on chromatically biased backgrounds L/(L+M) S/(L+M) Sunlight illuminant Red-blue biasGreen-yellow bias Skylight illuminant Red-blue biasGreen-yellow bias

ReflectancesChromaticities HES HS JEM L/(L+M) S/(L+M) S/(L+M) S/(L+M) HES HS JEM L/(L+M) EXPERIMENT 2 Classification boundaries with red-blue backgrounds Red lines: boundaries under sunlight Blue lines: boundaries under skylight. Circles: corresponding illuminants Illuminant has a large effect on the location of color boundaries in chromaticity space, but similar reflectances fall on color boundaries under both illuminants. Constancy indices 0.87, 0.81, and 0.95 for the three observers. Appearance-based color constancy could not be a result of spatially extended adaptation, or illuminant estimation using the mean.

ReflectancesChromaticities HES HS JEM L/(L+M) S/(L+M) S/(L+M) S/(L+M) HES HS JEM L/(L+M) EXPERIMENT 2 Classification boundaries with yellow-green backgrounds Red lines: boundaries under sunlight Blue lines: boundaries under skylight. Circles: corresponding illuminants Illuminant has a large effect on the location of color boundaries in chromaticity space, but similar reflectances fall on color boundaries under both illuminants. Constancy indices 0.87, 0.81, and 0.95 for the three observers. Appearance-based color constancy could not be a result of spatially extended adaptation, or illuminant estimation using the mean.

ReflectancesChromaticities HES HS JEM L/(L+M) S/(L+M) S/(L+M) S/(L+M) HES HS JEM L/(L+M) Comparison of Experiments 1 and 2 Effect of mean background chromaticity BACKGROUNDS Balanced (black lines) Red-blue (purple lines) Green-yellow (lime lines) Pluses: mean chromaticities The chromaticity of the background has little effect on the location of color boundaries in chromaticity space, indicating that the constancy process is unlikely to be spatially extended.

Test: Skylight Background: Sunlight Test: Sunlight Background: Skylight Experiment 3 Conflicting illuminants on test and balanced background

Chromaticities HES HS JEM L/(L+M) S/(L+M) S/(L+M) S/(L+M) EXPERIMENT 3 vs 1 Classification boundaries with conflicting illuminant Red lines: boundaries under sunlight Blue lines: boundaries under skylight Circles: corresponding illuminants Dashed lines: consistent illuminant Appearance constancy is a result of a spatially local but temporally extended process: Local adaptation with either long time constants or gated by attention to the test. Illuminant estimation using spatially local statistics collated over time. Test appearance anchored on test mean. Constancy indices 0.70, 0.45, and 0.82 for the three observers.

Lichtenberg’s game: Color identification without appearance constancy Can you identify trees of similar foliage across sun and shade? How do you know which parts are sunny or shady?

Three patches are from the same material while one is different. Illumination on the right is skylight, on the left is sunlight. Which is the odd material?

% SIDE CORRECT: Discrimination within each illuminant % OBJECT CORRECT: Identification across illuminants Mean color of standards: red, green, yellow, blue, magenta, cyan. 6 standards x 5 distractors x 2 daylights x 6 deltas x 10 reps, Method of constant stimuli DELTA is distance between standard and distractor.

IDENTIFICATION VS DISCRIMINATION MATERIALS In most cases, observers can identify materials across illuminants almost as well as they can discriminate materials within the same illuminant. Cases where identification is considerably worse provide cues to judgment strategies.

ACCURATE IDENTIFICATION PERFORMANCE IN THE ABSENCE OF APPEARANCE CONSTANCY Color conversions contain information about object invariances and illuminant color shifts. This information can be used for material and illuminant identification: Affine-heuristic (translation & multiplication) based algorithms Second-order color similarities: similarity between color differences. Rank orders will give correct identification only if material statistics are constant across illuminants.

Affine heuristic based algorithm L/(L+M) Solve correspondence problem for a subset of reflectances under equal- energy light with the superset under skylight. Errors should reflect divergence from affine transform.

IDENTIFICATION BASED ON 2 ND -ORDER SIMILARITY Color differences between tests vs differences between backgrounds Errors will depend on the nature of judging 2 nd -order similarities: RED and BLUE will be confused if observers judge deviations from directions of color vectors better than lengths of color vectors? Means of Material Chromaticities

JUDGMENT STRATEGIES FOR COLOR IDENTIFICATION As a result of spatially local processes, material identification within single-illuminant conditions can be based on first-order similarity. Material identification across multiple illuminants is based on second order similarities. Even if the spatially local process underlying appearance constancy is retinal adaptation, inferences about color invariants of natural objects require 2 nd -order judgments that are probably cortical.