Face Relighting with Radiance Environment Maps Zhen Wen 1, Zicheng Liu 2, Thomas Huang 1 Beckman Institute 1 University of Illinois Urbana, IL61801, USA {zhenwen, Microsoft Research 2 One Microsoft Way Redmond, WA 98052, USA

Problem Statement Given a single image of face, modify the lighting effect –Simulate environment rotation –Transfer the lighting from another face image –Interactive lighting editing Input Modified lighting

Related Work Inverse rendering – recover reflection properties from image samples and geometry. –Recover BRDF [Marschner1998]. –Recover basis [Debevec2000], ~2000 basis [Georphiades1999], 3 basis for Lambertian reflection. Relighting by illumination ratio –Preserve high frequency texture. [Riklin1999], [Stoschek2000] “quotient image” [Sato1999]

Assumptions Diffuse face surface Distant illumination Ignore cast shadow –(as in applications of environment maps)

Radiance Environment Map Suppose we capture the lighting using a sphere (radiance environment map). –For face

Approximate Radiance Environment Map from Image Hypothesis: face albedo has no lower order (1,2,3,4) coefficients: Use spherical harmonic representation of lighting: [Ramamoorthi2001] [Basri2001]

Algorithm: –Solve for the first 9 harmonics coefficients of –REM =

Relighting Step 1: Radiance environment map Step 2: To relight a rotated pixel:

Different Relighting Scenario –Different lighting: Light transfer: Light editing: Estimate from new face image Obtain by editing coefficients

Back Lighting Assumption It’s under-constrained to recover all 9 coefficients from a single frontal image –[Ramamoorthi2002] Make assumptions about back lighting –Symmetric lighting, i.e. lighting in the back is the same as front Good when only frontal lighting matters Equivalent to assuming 3 “asymmetric” coefficients to be zero. –Assumption based on scene, e.g. dark back lighting

Basic Algorithm Align image with generic 3D face model. Approximate radiance environment map. Synthesis appearance in –rotated lighting –different lighting using REM recovered from images in target lighting. –light editing by adjusting the 9 coefficients

Dynamic Range of Image Ratio-based relighting has large relative error when pixels values are too low or saturated. Use example-based texture synthesis to improve. –Relight all skin pixels to same normal. –Detect outliers using robust statistics. –Use the remaining pixels as examples to synthesis in the place of outliers Use patch-based approach Constrain that synthesized patch should be as close as the original patch.

Results – Rotation Example 1 The middle image is the input

Results – Rotation Example 2 – Ground Truth Comparison The middle image is the input. The upper row is the ground truth.

Results – Rotation Example 3 The middle image is the input

Results – Low Dynamic Range Input RGB B Basic algorithm With example- based synthesis

Results – Light Editing

Results – Lighting Transfer InputTargetResult

Conclusion Efficient technique for modifying, editing lighting in face images Capture variation due to diffuse lighting using spherical harmonics based environment maps Ratio image remove material dependency

Future Work More intuitive interface for light editing Handle non-Lambertian effects Recover personalized 3D geometry model