Demosaicking for Multispectral Filter Array (MSFA) Lidan Miao AICIP Research August 24, 2004

Background Challenge in acquisition of multispectral images. The CyberEye 2100 is developed in SCP Laboratory at Syracuse University. A synchronized filter wheel and optical system provides 12 independent spectral bands from 400-1000 nm . http://www.cs.ucf.edu/~jlee/www-scp/html/real-time_multispectral_camera.html Multispectral filter array technique. System Structure Camera Len MSFA Sensor Mosaic Image Reconstruction Scene Multispectral http://www.dpreview.com/learn/?/Glossary/ Camera_System/Color_Filter_Array_01.htm

Terminology MSFA samples Mosaic Image Demosaicking & Interpolation Reconstructed image & Demosaicked image

CFA Demosaicking Review Bilinear Interpolation Constant Hue-based Interpolation Hue is defined as the ratio or difference of red and green, blue and green (i.e. R-G,B-G or R/G, B/G) The HVS is more sensitive to hue artifacts than luminance errors. Gradient Based Interpolation Interpolate along the edges instead of across them. Weighted sum Interpolation

MSFA Review Design requirement A generic MSFA generation algorithm Probability of appearance(POA) Spectral consistency Uniform distribution A generic MSFA generation algorithm Based on the binary tree decomposition. Input: the number of spectral band and the probability of each band.

MSFA Review Resulted MSFA 1 3 2 4 5 6 Decomposition Subsampling 7 8 1

Spectral Correlation Why?? Details are well preserved in spectral bands with more MSFA samples. Different spectral bands possess similar edge information. Validation metric: image similarity.

Spectral correlation:example Similarity between different spectral bands B1 B2 B3 B4 B5 B6 B7 1 0.68 0.12 0.35 0.37 0.42 0.44 0.45 0.40 0.34 0.32 0.84 0.76 0.72 0.90 0.85 0.94

Progressive Demosaicking Binary tree MSFA Band selection Pixel selection Interpolation Pixel distribution of each band

Band Selection Determine the order of selecting different spectral bands for interpolation. Band selection is associated with the tree level of each leaf, i.e. the probability of appearance. The spectral band with the highest POA is interpolated first.

Pixel Selection Binary tree traversal from leaf nodes Example: reconstruct spectral band “C”

Interpolation Gradient Based Interpolation Pros. vs. Cons. Adaptive edge-sensing Interpolation: weighted sum of neighboring pixels. p3 p1 e p2 p4 p1 p3 e p4 p2

Edge-sensing Interpolation Edge-likelihood in each direction of neighboring pixels. Sobel edge detector and the second order derivative. Weight estimator Estimation of target pixel value

Case studies POA=1/2 POA=1/4 POA=1/8

Experiments-Data Description Real Multispectral Data 92AV3C9(9 bands) FLC1(12 bands) TIPJUL(7 bands) Synthetic Multispectral Data Eight objects

Validation Original multispectral images are sampled using MSFA to generate mosaic images. Reconstruct the mosaic images using proposed method with and without binary tree bilinear interpolation with and without binary tree. Compare between the original multispectral images and the reconstructed images. Evaluation metric: subject visual comparison, objective RMSE, classification accuracy.

Visual Comparison Proposed with binary tree Proposed without binary tree Bilinear with binary tree Bilinear without binary tree Original

Visual Comparison Original Proposed with and without tree Bilinear with and without tree

Experiments-RMSE Root Mean Square Error where M and N are the size of image, and is the original image, represents the reconstructed image. The smaller the RMSE, the better the reconstructed image.

Experiments-RMSE Comparison Real multispectral data set Synthetic multispectral data set

Experiments-Pixel Classification Real multispectral data Each scene contains different classes. Choose the region we know the ground truth, from which we extract training and testing samples. Knn classifier. Synthetic multispectral data Extract 20 training samples from each targets, all the targets pixels are testing samples. Knn classifier

Pixel classification accuracy Real multispectral data set Synthetic multispectral data set

Experiments-Color Map Pixel classification of each targets, different colors represent different classification results Original Reconstructed Original Reconstructed Original Reconstructed

Experiments-Object classification Compare the spatial and spectral features. Spatial: hu-moments (7D) Spectral: average of target pixels (7D) Study the effect of viewing distance. A schematic representation for extraction of training and testing set

Object classification accuracy Original Proposed Bilinear

Conclusion The proposed progressive method outperforms non-progressive and bilinear interpolation. The classification accuracy using reconstructed images is comparable with the one generated using original multispectral images. The MSFA technique is a feasible solution for multispectral cameras.

Thanks ?