1. Demosaicking for Multispectral Filter Array (MSFA)
Lidan Miao
AICIP Research
August 24, 2004

2. Background Challenge in acquisition of multispectral images.
The CyberEye 2100 is developed in SCP Laboratory at Syracuse University. A synchronized filter wheel and optical system provides 12 independent spectral bands from nm .
Multispectral filter array technique.
System Structure
Camera
Len
MSFA
Sensor
Mosaic
Image
Reconstruction
Scene
Multispectral
Camera_System/Color_Filter_Array_01.htm

3. Terminology MSFA samples Mosaic Image Demosaicking & Interpolation
Reconstructed image & Demosaicked image

4. CFA Demosaicking Review
Bilinear Interpolation
Constant Hue-based Interpolation
Hue is defined as the ratio or difference of red and green, blue and green (i.e. R-G,B-G or R/G, B/G)
The HVS is more sensitive to hue artifacts than luminance errors.
Gradient Based Interpolation
Interpolate along the edges instead of across them.
Weighted sum Interpolation

5. MSFA Review Design requirement A generic MSFA generation algorithm
Probability of appearance(POA)
Spectral consistency
Uniform distribution
A generic MSFA generation algorithm
Based on the binary tree decomposition.
Input: the number of spectral band and the probability of each band.

6. MSFA Review Resulted MSFA 1 3 2 4 5 6 Decomposition Subsampling 7 8 1

7. Spectral Correlation Why??
Details are well preserved in spectral bands with more MSFA samples.
Different spectral bands possess similar edge information.
Validation metric: image similarity.

8. Spectral correlation:example
Similarity between different spectral bands B1 B2 B3 B4 B5 B6 B7 1
0.68
0.12
0.35
0.37
0.42
0.44
0.45
0.40
0.34
0.32
0.84
0.76
0.72
0.90
0.85
0.94

9. Progressive Demosaicking
Binary tree
MSFA
Band selection
Pixel selection
Interpolation
Pixel distribution of each band

10. Band Selection
Determine the order of selecting different spectral bands for interpolation.
Band selection is associated with the tree level of each leaf, i.e. the probability of appearance.
The spectral band with the highest POA is interpolated first.

11. Pixel Selection Binary tree traversal from leaf nodes
Example: reconstruct spectral band “C”

12. Interpolation Gradient Based Interpolation
Pros. vs. Cons.
Adaptive edge-sensing Interpolation: weighted sum of neighboring pixels. p3 p1 e p2 p4 p1 p3 e p4 p2

13. Edge-sensing Interpolation
Edge-likelihood in each direction of neighboring pixels.
Sobel edge detector and the second order derivative.
Weight estimator
Estimation of target pixel value

14. Case studies
POA=1/2
POA=1/4
POA=1/8

15. Experiments-Data Description
Real Multispectral Data
92AV3C9(9 bands)
FLC1(12 bands)
TIPJUL(7 bands)
Synthetic Multispectral Data
Eight objects

16. Validation
Original multispectral images are sampled using MSFA to generate mosaic images.
Reconstruct the mosaic images using
proposed method with and without binary tree
bilinear interpolation with and without binary tree.
Compare between the original multispectral images and the reconstructed images.
Evaluation metric: subject visual comparison, objective RMSE, classification accuracy.

17. Visual Comparison Proposed with binary tree
Proposed without binary tree
Bilinear with binary tree
Bilinear without binary tree
Original

18. Visual Comparison Original Proposed with and without tree
Bilinear with and without tree

19. Experiments-RMSE Root Mean Square Error
where M and N are the size of image, and
is the original image, represents the reconstructed image. The smaller the RMSE, the better the reconstructed image.

20. Experiments-RMSE Comparison
Real multispectral data set
Synthetic multispectral data set

21. Experiments-Pixel Classification
Real multispectral data
Each scene contains different classes. Choose the region we know the ground truth, from which we extract training and testing samples.
Knn classifier.
Synthetic multispectral data
Extract 20 training samples from each targets, all the targets pixels are testing samples.
Knn classifier

22. Pixel classification accuracy
Real multispectral data set
Synthetic multispectral data set

23. Experiments-Color Map
Pixel classification of each targets, different colors represent different classification results
Original
Reconstructed
Original
Reconstructed
Original
Reconstructed

24. Experiments-Object classification
Compare the spatial and spectral features.
Spatial: hu-moments (7D)
Spectral: average of target pixels (7D)
Study the effect of viewing distance.
A schematic representation for extraction of training and testing set

25. Object classification accuracy
Original
Proposed
Bilinear

26. Conclusion
The proposed progressive method outperforms non-progressive and bilinear interpolation.
The classification accuracy using reconstructed images is comparable with the one generated using original multispectral images.
The MSFA technique is a feasible solution for multispectral cameras.

27. Thanks ?