1. Demosaicking Problem in Digital Cameras
EE 7700
Demosaicking Problem in Digital Cameras

2. Multi-Chip Digital Camera
To produce a color image, at least three spectral components are needed at each pixel.
One approach is to use beam-splitters and multiple chips.
Lens
Scene
Spectral filters
Beam-splitters
Sensors
Bahadir K. Gunturk

3. Single-Chip Digital Camera
Multi-chip approach is expensive. Precise chip alignment is required.
The alternative is to use a color filter array.
Lens
Color filter array
Sensors
Scene
Bahadir K. Gunturk

4. Single-Chip Digital Camera
The missing color samples must be estimated to produce the full color image.
Since a mosaic of samples are available, this estimation (interpolation) process is called demosaicking.
Bahadir K. Gunturk

5. Single-Chip Digital Camera
Images suffer from color artifacts when the samples are not estimated correctly.
Original image
Bilinearly interpolated from CFA-filtered samples
Bahadir K. Gunturk

6. Demosaicking Approaches
Non-Adaptive Single-Channel Interpolation: Interpolate each color channel separately using a standard technique, such as nearest-neighbor interpolation, bilinear interpolation, etc.
Edge-Directed Interpolation: Estimate potential edges, avoid interpolating across the edges.
Edge-directed interpolation
Calculate horizontal gradient ΔH = |G1 – G2|
Calculate vertical gradient ΔV = |G3 – G4|
If ΔH > ΔV,
Gx = (G3 + G4)/2
Else if ΔH < ΔV,
Gx = (G1 + G2)/2
Else
Gx = (G1 + G2 + G3 + G4)/4 3 1 x 2 4
Bahadir K. Gunturk

7. Demosaicking Approaches
Edge-Directed Interpolation: Based on the assumption that color channels have similar texture, various edge detectors can be used.
Edge-directed interpolation
Calculate horizontal gradient ΔH = | (R3 + R7)/2 – R5 |
Calculate vertical gradient ΔV = | (R1 + R9)/2 – R5 |
If ΔH > ΔV,
G5 = (G2 + G8)/2
Else if ΔH < ΔV,
G5 = (G4 + G6)/2
Else
G5 = (G2 + G8 + G4 + G6)/4 1 2 3 4 5 6 7 8 9
Bahadir K. Gunturk

8. Demosaicking Approaches
Constant-Hue-Based Interpolation: Hue does not change abruptly within a small neighborhood.
Interpolate green channel first.
Interpolate hue (defined as either color differences or color ratios).
Estimate the missing (red/blue) from the interpolated hue.
Interpolate
Red
Interpolated Red
Interpolate
Green
Bahadir K. Gunturk

9. Demosaicking Approaches
Edge-Directed Interpolation of Hue: It is a combination of edge-directed interpolation and constant-hue-based interpolation. Hue is interpolated as in constant-hue-based interpolation approach, but this time, hue is interpolated based on the edge directions (as in the edge-directed interpolation algorithm).
Bahadir K. Gunturk

10. Demosaicking Approaches
Using Laplacian For Enhancement: Use the second-order gradients of red/blue channels to enhance green channel.
Calculate horizontal gradient ΔH = |G4 – G6| + |R5 – R3 + R5 – R7|
Calculate vertical gradient ΔV = |G2 – G8| + |R5 – R1 + R5 – R9|
If ΔH > ΔV,
G5 = (G2 + G8)/2 + (R5 – R1 + R5 – R9)/4
Else if ΔH < ΔV,
G5 = (G4 + G6)/2 + (R5 – R3 + R5 – R7)/4
Else
G5 = (G2 + G8 + G4 + G6)/4 + (R5 – R1 + R5 – R9 + R5 – R3 + R5 – R7)/8 1 2 3 4 5 6 7 8 9
Bahadir K. Gunturk

11. Aliasing Frequency spectrum of an image: After CFA sampling:
Green channel
Red/Blue channel
Bahadir K. Gunturk

12. Demosaicking Approach
Alias Cancelling: Based on the assumption that red, green, and blue channels have similar frequency components, the high-frequency components of red and blue channels are replaced by the high-frequency components of green channel.
Red/Blue channel
Bahadir K. Gunturk

13. Experiment HL HL HL LL LL LL HH LH LH LH HL HL HL LL LL LL HH LH LH LH
Full Red/Green/Blue channels
Subband decomposition LL LL LL HH LH LH LH
CFA Sampling
Interpolate HL HL HL
Subband decomposition LL LL LL HH LH LH LH
Bahadir K. Gunturk

14. Constraint Sets
Detail Constraint Set: Detail subbands of the red and blue channels must be similar to the detail subbands of the green channel. HL HL HH HH LH LH
Bahadir K. Gunturk

15. Constraint Sets
Observation Constraint Set: Interpolated channels must be consistent with the observed data.
Sensors
CFA
Bahadir K. Gunturk

16. Projection Operations
Projection onto the Detail Constraint Set:
Decompose the color channels.
Update the detail subbands of red and blue channels. HL HH LH
Apply synthesis filters to reconstruct back the channels.
Bahadir K. Gunturk

17. Projection Operations
Projection onto the Observation Constraint Set:
Insert the observed data to their corresponding positions.
Sensors
CFA
Bahadir K. Gunturk

18. Alternating Projections Algorithm
Samples of color channels
Initial interpolation
Projection onto the detail constraint set
Projection onto the observation constraint set
Insert the observed data
Update
Iteration
Bahadir K. Gunturk

19. Results Original Hibbard 1995 Laroche and Prescott 1994
Hamilton and Adams 1997
Kimmel 1999
Gunturk 2002
Bahadir K. Gunturk

20. Results Laroche and Prescott 1994 Hibbard 1995 Original
Hamilton and Adams 1997
Kimmel 1999
Gunturk 2002
Bahadir K. Gunturk

21. Previous Methods [Gunturk02]
Gunturk et al, “Demosaicking: Color Filter Array Interpolation in Single-Chip Digital Cameras,” to appear in IEEE Signal Processing Magazine.
Bahadir K. Gunturk

22. References
[Gunturk02] Gunturk et al, “Color Plane Interpolation Using Alternating Projections,” IEEE Trans. Image Processing, 2002.
[Hibbard 1995] R. H. Hibbard, “Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients,” U.S. Patent 5,382,976, January, 1995.
[Laroche and Prescott 1994] C. A. Laroche and M. A. Prescott, “Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients,” U.S. Patent 5,373,322, December, 1994.
[Hamilton and Adams 1997] J. F. Hamilton Jr. and J. E. Adams, “Adaptive color plane interpolation in single sensor color electronic camera,” U.S. Patent 5,629,734, May, 1997.
[Kimmel 1999] R. Kimmel, “Demosaicing: Image reconstruction from CCD samples,” IEEE Trans. Image Processing, vol. 8, pp , 1999.
Bahadir K. Gunturk