1. Illumination Estimation via Thin Plate Spline Weihua Xiong ( OmniVision Technology,USA ) Lilong Shi, Brian Funt ( Simon Fraser University, Canada) ( Simon Fraser University, Canada) Sung-Su Kim, Byoung-Ho Kang, Sung-Duk Lee, Sung-Su Kim, Byoung-Ho Kang, Sung-Duk Lee, and Chang-Yeong Kim and Chang-Yeong Kim (Samsung Advanced Institute of Technology, Korea) (Samsung Advanced Institute of Technology, Korea)

2. Automatic White Balancing Image Estimate the Illumination Original Surface Color

3. Illumination Estimation Methods Non-statistical SolutionNon-statistical Solution –Gray World (1980 Buchsbaum) –Max-RGB –Shades of Gray (2004 Finlayson) –Gray-Edge Hypothesis (Weijei & Gevers 2005) Statistics Based SolutionStatistics Based Solution –Color by Correlation (Finlayson at. el. 2001) –Neural Network (Funt. 2002) –Support Vector Regression (Xiong, 2004) –KL-Divergence (Rosenberg 2001)

4. Limitations of previous statistical solutions Color by Correlation Illuminations pre-fixed Neural Network Local Minima Support Vector Regression Many parameters to be determined by user

5. Thin Plate Spline (TPS) Interpolation Thin Plate Spline (TPS) interpolates between control pointsThin Plate Spline (TPS) interpolates between control points –Minimizes the bending energy function of a thin metal plate. –Originally designed for deformable matching between images AdvantageAdvantage –Output is always unique –Fits to all the training data smoothly –No extrapolation –Solution can never be singular Has been extended into 3D for mapping from RGB to XYZ (CIC 2005)Has been extended into 3D for mapping from RGB to XYZ (CIC 2005) Source Image Target Image

6. Thumbnail Input The method uses thumbnails as inputThe method uses thumbnails as input –8-by-8 input RGB images –Each of the 64 pixels is the average of pixels from original input image Original Image Change into chromaticity space [R/(R+G+B) G/(R+G+B)] So we extend TPS to 128 Dimensions here Thumbnail Image

7. TPS for AWB: Intuition Basic ideaBasic idea –Similar images require similar white balancing –Illumination rg-chromaticity interpolated Training set contains images and their respective white points (illumination color)Training set contains images and their respective white points (illumination color) For input imageFor input image –Interpolate white point based on the distance between the input and training images

8. Training set consists of N images Corresponding illumination chromaticity values (for case of 8x8x2=128 input image) {(I i,1,I i,2,…I i,128), (r i,g i )}. (i/o pair) TPS determines parameters w i and (a 0, a 1, a 2, …, a 128 ) Controls two non-rigid mapping functions f r, f g, such that (r i,g i ) = (f r (I i,1,I i,2,…I i,128 ), f g (I i,1,I i,2,…I i,128 )). TPS is defined by a non-linear function with an additional linear term. Without loss of generality, consider only f r definition in which w i and a i are coefficients to be determined: TPS Details

9. Weighted distance to every image in the training set X is either r or g chromaticity TPS Details (Continued) Linear Term Nonlinear Term

10. TPS Details (Continued) Smoothness constraint where is the total bending energy described in terms of the curvature of the energy is minimized when

11. Direct Solution (once per training set) N + 129 unknowns and N + 129 equations Define L: U ij = U(||(I i1,I i2,…I i128 )- (I j1,I j2,…I j128 )||, U(r) = r 2 *logr Additionally define W= (w 1, w 2,…, w N, a 0, a 1, a 2, …, a 128 ) T, and K = (r 1, r 2, r 3 … r N,0,0,0,….0) T. We have K =LW and solution W = L -1 K 0 is ZERO matrix with size of 129x129

12. Original Training Data Pairs Graphical Example of TPS Interpolation Nonlinear Term Linear Term Error

13. TPS Result Input Image (Bluish) TPS Output Ground Truth Image

14. TPS on Large Image Database Average of bright part on the grayball is assumed to the true illumination value

15. Performance Comparison Method Angular Distance (x10 2 ) MaxRMSMaxRMS SVR (3D) 24.556.7618.625.03 TPS34.817.0225.785.19 SOG37.018.9327.996.59 Gray World 43.849.6645.097.82 Max RGB 27.4212.8121.729.14 The database is divided into two non-overlapped subsets Test set size 4080. Training data set size 3581

16. Algorithm Comparison Using Wilcoxon MethodTPSSVR(3D)SoGMAXGW TPS=+++ SVR(3D)=+++ SoG---- MAX---- GW---+ TPS is equal to SVR, but better than GW, SoG, and Max

17. Conclusion Thin Plate Interpolation is applied in illumination estimationThin Plate Interpolation is applied in illumination estimation Performs non-uniform interpolationPerforms non-uniform interpolation –Assumption that similar images require similar color correction Method is parameter independentMethod is parameter independent Both training and testing are fastBoth training and testing are fast The performance on natural images shows the accuracy of TPS illumination estimation to be goodThe performance on natural images shows the accuracy of TPS illumination estimation to be good

18. Funded by Samsung Advanced Technology Institute

19. Thanks & Questions