Logarithmic CMOS image sensors Dr. Dileepan Joseph Dept. of Engineering Science University of Oxford, UK

Outline MotivationBackgroundMethodConclusions Future work

Motivation: social Society has invested over many millennia in developing technology to record observed scenes on an independent medium Artistic license aside, the aim has been to render images with a maximum of perceptual accuracy using a minimum of effort The digital camera is a culmination of the above but its development is far from complete… Although digital cameras have in many ways surpassed film cameras, human vision routinely outperforms the best cameras

Motivation: economic A digital camera consists of many components (optics, housing, battery, memory etc.), of which the image sensor is considered principal With market revenues of $1.7 billion in 2003, there is widespread research and development in a variety of image sensor designs Modern designs may be either charge coupled device (CCD) sensors or complementary metal- oxide-semiconductor (CMOS) sensors

Motivation: technological Criterion Human eye Film photo CCD sensor CMOS sensor Pixel pitch 2–3 μm 10–20 μm 5–10 μm Image pitch 3 cm Film size 1 mm–11 cm 1 mm–2 cm Dynamic range 2–5 decades 1–4 decades 4 decades 3–5 decades Max. frame rate ≈ 15 Hz 1 shot only 10 kHz >> 10 kHz Pre-processingExtensiveNoneNonePossible Unit price Invaluable € 0.1 € 100 € 10

Background: CCD image sensor Marches photo generated charge systematically from an array of pixels to an output amplifier Established technology High resolution, high sensitivity, low noise Fabrication process is optimised for imaging Market share of 93% in 1999 (49% in 2004?)

Background: CMOS image sensor Works like memory array with photosensitive pixels instead of memory cells Signal processing may be included on the same die High yield and good video performance May be fabricated by the makers of microchips Market share of 7% in 1999 (51% in 2004?)

Background: linear pixels Linear pixels (CCD or CMOS) integrate photons over discrete periods of time They produce a voltage directly proportional to the light intensity The response may saturate white or black easily

Background: logarithmic pixels Logarithmic pixels (CMOS only) can measure photon flux continuously They produce a voltage proportional to the logarithm of light intensity The response is similar to that of human vision

Background: image quality Images are noisy with logarithmic sensors Colour is worse than with linear sensors Quality improves with digital processing No comprehensive treatment of either problem or solution

Method: theory Model logarithmic CMOS image sensors using optical & integrated circuit theory Use the model to hypothesize the cause and solution of image quality problems Calibrate the model and test hypotheses using constrained regression theory Optimise digital image processing using multilinear (or array) algebra

Method: simulation Simulation of integrated circuits is more accurate than a theoretical analysis Cost of simulation in time and money is small compared to that of experiment Integrated circuits may be studied under controlled and well-defined conditions Internal states and variables may be observed without specialised equipment, circuit disruption and/or foresight

Method: experiment Experiments were performed using a Fuga 15RGB camera from C-Cam Technologies The camera was operated from a portable PC via a custom Windows application The image sensor had 512 × 512 pixels and a full frame rate of about 8 Hz Until recently, it was the most successful commercial logarithmic image sensor

Conclusions: fixed pattern noise y = a + b ln (c + x) + ε for illuminance x and response y of a pixel Variation of offset a, gain b, bias c or a combination thereof causes FPN Calibration possible within limits of the stochastic error ε

Conclusions: fixed pattern noise Left to right: FPN correction for single, double and triple variation models Top to bottom: two decade attenuation of illuminance in half decade steps Inter-scene plus intra- scene dynamic range of 3.5 decades

Conclusions: transient response The transient response of a pixel is fast enough for most applications Greater demands are placed on the row and column readouts Premature digitization results in a predictable non-uniformity or FPN Affects only a few rows due to slow scanning

Conclusions: transient response Premature digitization is more serious for column readout due to speed For example, columns need scanning at 100 MHz for HDTV video Column-to-column gain variation is caused by transient response Resolve with careful timing and design

Conclusions: temperature dependence Unlike with humans, digital cameras do not regulate temperature Hence, responses to illuminance depend on temperature When temperature dependence varies from pixel to pixel, FPN occurs

Conclusions: temperature dependence The dark response of a pixel depends only on temperature It may be used to correct FPN due to temperature in the light response Experiments support this conclusion but simulation results are shown for clarity

Conclusions: colour rendition Combine the theories of colour linear sensors and b/w logarithmic sensors Calibrate FPN, using images of uniform stimuli, by a relative analysis Calibrate colour, using images of a colour chart, by an absolute analysis Fuga 15RGB competes with conventional digital cameras (which have a perceptual error of 15)

Conclusions: colour rendition Image of a colour chart, in 11 lux of illuminance, was rendered using calibrated models Single, double and triple variation results and ideal colours are shown As with vision, rendition improves in brighter lighting and worsens in dimmer lighting

Future work Digital cameras aim to render images with a maximum of perceptual accuracy using a minimum of effort By modelling and calibrating logarithmic CMOS image sensors, problems with image quality may be solved Past work has focused on maximising perceptual accuracy but future work will focus on minimising effort

Future work Shrinking feature sizes may be used to improve imaging There are challenges with deep submicron CMOS processes that need overcoming What about industrial and biomedical uses of the technology?

Acknowledgements This work was funded thanks to the engineering research councils of both Canada and the UK Thanks also to colleagues at the Microelectronic Circuits and Analogue Devices research group