Illumination Estimation via Thin Plate Spline Weihua Xiong ( OmniVision Technology,USA ) Lilong Shi, Brian Funt ( Simon Fraser University, Canada) ( Simon Fraser University, Canada) Sung-Su Kim, Byoung-Ho Kang, Sung-Duk Lee, Sung-Su Kim, Byoung-Ho Kang, Sung-Duk Lee, and Chang-Yeong Kim and Chang-Yeong Kim (Samsung Advanced Institute of Technology, Korea) (Samsung Advanced Institute of Technology, Korea)

Automatic White Balancing Image Estimate the Illumination Original Surface Color

Illumination Estimation Methods Non-statistical SolutionNon-statistical Solution –Gray World (1980 Buchsbaum) –Max-RGB –Shades of Gray (2004 Finlayson) –Gray-Edge Hypothesis (Weijei & Gevers 2005) Statistics Based SolutionStatistics Based Solution –Color by Correlation (Finlayson at. el. 2001) –Neural Network (Funt. 2002) –Support Vector Regression (Xiong, 2004) –KL-Divergence (Rosenberg 2001)

Limitations of previous statistical solutions Color by Correlation Illuminations pre-fixed Neural Network Local Minima Support Vector Regression Many parameters to be determined by user

Thin Plate Spline (TPS) Interpolation Thin Plate Spline (TPS) interpolates between control pointsThin Plate Spline (TPS) interpolates between control points –Minimizes the bending energy function of a thin metal plate. –Originally designed for deformable matching between images AdvantageAdvantage –Output is always unique –Fits to all the training data smoothly –No extrapolation –Solution can never be singular Has been extended into 3D for mapping from RGB to XYZ (CIC 2005)Has been extended into 3D for mapping from RGB to XYZ (CIC 2005) Source Image Target Image

Thumbnail Input The method uses thumbnails as inputThe method uses thumbnails as input –8-by-8 input RGB images –Each of the 64 pixels is the average of pixels from original input image Original Image Change into chromaticity space [R/(R+G+B) G/(R+G+B)] So we extend TPS to 128 Dimensions here Thumbnail Image

TPS for AWB: Intuition Basic ideaBasic idea –Similar images require similar white balancing –Illumination rg-chromaticity interpolated Training set contains images and their respective white points (illumination color)Training set contains images and their respective white points (illumination color) For input imageFor input image –Interpolate white point based on the distance between the input and training images

Training set consists of N images Corresponding illumination chromaticity values (for case of 8x8x2=128 input image) {(I i,1,I i,2,…I i,128), (r i,g i )}. (i/o pair) TPS determines parameters w i and (a 0, a 1, a 2, …, a 128 ) Controls two non-rigid mapping functions f r, f g, such that (r i,g i ) = (f r (I i,1,I i,2,…I i,128 ), f g (I i,1,I i,2,…I i,128 )). TPS is defined by a non-linear function with an additional linear term. Without loss of generality, consider only f r definition in which w i and a i are coefficients to be determined: TPS Details

Weighted distance to every image in the training set X is either r or g chromaticity TPS Details (Continued) Linear Term Nonlinear Term

TPS Details (Continued) Smoothness constraint where is the total bending energy described in terms of the curvature of the energy is minimized when

Direct Solution (once per training set) N unknowns and N equations Define L: U ij = U(||(I i1,I i2,…I i128 )- (I j1,I j2,…I j128 )||, U(r) = r 2 *logr Additionally define W= (w 1, w 2,…, w N, a 0, a 1, a 2, …, a 128 ) T, and K = (r 1, r 2, r 3 … r N,0,0,0,….0) T. We have K =LW and solution W = L -1 K 0 is ZERO matrix with size of 129x129

Original Training Data Pairs Graphical Example of TPS Interpolation Nonlinear Term Linear Term Error

TPS Result Input Image (Bluish) TPS Output Ground Truth Image

TPS on Large Image Database Average of bright part on the grayball is assumed to the true illumination value

Performance Comparison Method Angular Distance (x10 2 ) MaxRMSMaxRMS SVR (3D) TPS SOG Gray World Max RGB The database is divided into two non-overlapped subsets Test set size Training data set size 3581

Algorithm Comparison Using Wilcoxon MethodTPSSVR(3D)SoGMAXGW TPS=+++ SVR(3D)=+++ SoG---- MAX---- GW---+ TPS is equal to SVR, but better than GW, SoG, and Max

Conclusion Thin Plate Interpolation is applied in illumination estimationThin Plate Interpolation is applied in illumination estimation Performs non-uniform interpolationPerforms non-uniform interpolation –Assumption that similar images require similar color correction Method is parameter independentMethod is parameter independent Both training and testing are fastBoth training and testing are fast The performance on natural images shows the accuracy of TPS illumination estimation to be goodThe performance on natural images shows the accuracy of TPS illumination estimation to be good

Funded by Samsung Advanced Technology Institute

Thanks & Questions