A Perceptual Heuristic for Shadow Computation in Photo-Realistic Images Wednesday, 2 August 2006 Peter VangorpOlivier DumontToon LenaertsPhilip Dutré Katholieke Universiteit Leuven

Introduction 3 types of realistic rendering

Introduction – radiometric accuracy “render everything a photometer can detect”

Introduction 3 types of realistic rendering – radiometric accuracy “render everything a photometer can detect” – physiological perception “render only what the eye can see”

Introduction 3 types of realistic rendering – radiometric accuracy “render everything a photometer can detect” – physiological perception “render only what the eye can see” – psychological perception “render only what the brain can see”

Radiometric accuracy measure shapes, light sources, materials,... photometer Cornell box [Meyer et al. 1986]

Physiological perception use low-level limitations of human visual system threshold vs intensity, contrast sensitivity,... referencethreshold mapvisibly indistinguishable adaptive rendering [Ramasubramanian et al. 1999]

Physiological perception use low-level limitations of human visual system threshold vs intensity, contrast sensitivity,... referencethreshold mapvisibly indistinguishable adaptive rendering [Ramasubramanian et al. 1999]

Psychological perception use higher-level heuristics “Is a shadow necessary for the realism of a scene?” [Thompson et al. 1998]

Psychological perception use higher-level heuristics “Is a shadow necessary for the realism of a scene?” “Do we need highlights to convey material properties?” [Thompson et al. 1998] [Fleming et al. 2004]

Psychological perception use higher-level heuristics “Is a shadow necessary for the realism of a scene?” “Do we need highlights to convey material properties?” “How detailed should the geometry be?” [Thompson et al. 1998] [Luebke 2001] [Fleming et al. 2004]

Motivation & Goals Motivation – shadows are important for perception of realism Goal – detect perceptually important shadows in the scene – render important shadows accurately – approximate unimportant shadows [Kersten et al. 1997]

Methodology 1. Psycho-physical experiments 2. Derive a heuristic predicting shadow importance 3. Design a perceptually driven algorithm 4. Experimental validation

Methodology 1. Psycho-physical experiments 2. Derive a heuristic predicting shadow importance 3. Design a perceptually driven algorithm 4. Experimental validation

Methodology 1. Psycho-physical experiments 2. Derive a heuristic predicting shadow importance 3. Design a perceptually driven algorithm 4. Experimental validation

Methodology 1. Psycho-physical experiments 2. Derive a heuristic predicting shadow importance 3. Design a perceptually driven algorithm 4. Experimental validation

1. Perceptual experiments Test setup: 162 images, varying sphere over 9 radii and 9 heights correct shadow: no shadow: (avg. illumination)

1. Perceptual experiments “Does the lighting in this image look realistic?” – single stimulus – decisions – avg. 2 sec / decision

2. Heuristic large difference in realism, e.g. small difference in realism, e.g.  sphere radius  sphere height difference in realism

3. A perceptually driven algorithm Ray tracing

3. A perceptually driven algorithm Ray tracing – shoot viewing ray

3. A perceptually driven algorithm Ray tracing – shoot viewing ray – evaluate heuristic in hit point to be shaded in function of distance and solid angle solid angledistance

3. A perceptually driven algorithm Preprocessing step – shadow photon map [Jensen and Christensen 1995]

3. A perceptually driven algorithm Preprocessing step – shadow photon map – shadow photons augmented with heuristic [Jensen and Christensen 1995] shadow photon map

3. A perceptually driven algorithm For each viewing ray Gather nearest shadow photons Calculate average perceptual value avg < threshold approximate: photon map render accurately: shadow rays yesno

3. A perceptually driven algorithm For each viewing ray Gather nearest shadow photons Calculate average perceptual value avg < threshold approximate: photon map render accurately: shadow rays yesno

3. A perceptually driven algorithm For each viewing ray Gather nearest shadow photons Calculate average perceptual value avg < threshold approximate: photon map render accurately: shadow rays yesno

For each viewing ray Gather nearest shadow photons Calculate average perceptual value avg < threshold approximate: photon map render accurately: shadow rays yesno user-defined threshold 3. A perceptually driven algorithm

For each viewing ray Gather nearest shadow photons Calculate average perceptual value avg < threshold approximate: photon map render accurately: shadow rays yesno user-defined threshold

3. A perceptually driven algorithm For each viewing ray Gather nearest shadow photons Calculate average perceptual value avg < threshold approximate: photon map render accurately: shadow rays yesno user-defined threshold

3. A perceptually driven algorithm

4. Validation Similar perceptual experiment – “Do the lighting and the shadows look realistic?”

4. Validation Similar perceptual experiment – “Do the lighting and the shadows look realistic?” Stimuli: 6 scenes – threshold 25%, 50%, 75% – reference rendering (threshold 0%)

4. Validation Similar perceptual experiment – “Do the lighting and the shadows look realistic?” Stimuli: 6 scenes – threshold 25%, 50%, 75% – reference rendering (threshold 0%) 15 subjects, almost 6000 decisions, avg. 5 seconds

4. Validation Up to 50% of the pixels can be approximated, without loss of perceptual realism Threshold 25% Approx px 13% Threshold 50% Approx px 24% Threshold 75% Approx px 48%

Results referencethreshold 90%

Results reference threshold 90%

Results referencethreshold 90%

Results referencethreshold 90%

reference Results threshold 80%

Results referencethreshold 80%

Results referencethreshold 80%

Results referencethreshold 80%

reference Results threshold 90%

Results referencethreshold 90%

Results referencethreshold 90%

Conclusions & Future Work Intuitions confirmed by statistical data Rendering algorithm driven by perceptual information

Conclusions & Future Work Intuitions confirmed by statistical data Rendering algorithm driven by perceptual information Extend methodology to other phenomena Different questions than “Does this look realistic?” Better ways to incorporate perceptual information – currently no significant acceleration yet

Questions?