1. A Digital Camera to Rival the Human Eye Dr. Dil Joseph, Assistant Professor Electrical and Computer Engineering University of Alberta

2. Synopsis Although digital cameras have in many ways surpassed the capabilities of film cameras, the human eye remains the ultimate standard for comparison and it vastly outperforms the best cameras in many respects Although digital cameras have in many ways surpassed the capabilities of film cameras, the human eye remains the ultimate standard for comparison and it vastly outperforms the best cameras in many respects Inspiration from biology may help us to “perfect” the technology of imaging Inspiration from biology may help us to “perfect” the technology of imaging

3. Outline Film cameras Film cameras Digital cameras Digital cameras Human eyes Human eyes My research My research Your questions Your questions However, let us first put things in perspective… However, let us first put things in perspective…

4. In Perspective Society has invested over millennia in developing technology to record observed scenes on an independent medium Society has invested over millennia in developing technology to record observed scenes on an independent medium Artistic license aside, the aim has been to render images with a maximum of perceptual accuracy using a minimum of effort Artistic license aside, the aim has been to render images with a maximum of perceptual accuracy using a minimum of effort The digital camera is merely a culmination of the above but that does not mean the technology development is over The digital camera is merely a culmination of the above but that does not mean the technology development is over Norbert Aujoulat © National Centre of Prehistory, France http://www.culture. gouv.fr/culture/arcn at/lascaux/en/

5. History of Film Cameras Two scientific processes required development: optical and chemical Two scientific processes required development: optical and chemical During the Renaissance, the optical process was understood well enough for artists to begin using the camera obscura as a drawing aid (see right) During the Renaissance, the optical process was understood well enough for artists to begin using the camera obscura as a drawing aid (see right)

6. History of Film Cameras Thomas Wedgwood published his work on photograms in 1802; he recorded short-lived images using silver nitrate on leather Thomas Wedgwood published his work on photograms in 1802; he recorded short-lived images using silver nitrate on leather John Herschel, who popularized the word “photography”, discovered in 1819 how to dissolve (unexposed) silver salts John Herschel, who popularized the word “photography”, discovered in 1819 how to dissolve (unexposed) silver salts Although Nicéphore Niépce produced the first photograph in 1827, Louis Daguerre was the most famous of the early inventors Although Nicéphore Niépce produced the first photograph in 1827, Louis Daguerre was the most famous of the early inventors

7. History of Film Cameras The Daguerreotype was revealed to the public in 1839 to much acclaim The Daguerreotype was revealed to the public in 1839 to much acclaim Images were produced on copper plates coated with silver iodide Images were produced on copper plates coated with silver iodide The drawbacks of the process were the cost, long exposure time and irreproducibility The drawbacks of the process were the cost, long exposure time and irreproducibility

8. History of Film Cameras Fox Talbot introduced the Calotype in 1840, the first process to involve negatives Fox Talbot introduced the Calotype in 1840, the first process to involve negatives Positive prints could be made cheaply on paper Positive prints could be made cheaply on paper Others went on to create processes that had both better image quality and shorter exposure times Others went on to create processes that had both better image quality and shorter exposure times

9. Film Cameras Today Film cameras today have come a long way since the nineteenth century, especially by improved spatial and contrast resolution Film cameras today have come a long way since the nineteenth century, especially by improved spatial and contrast resolution Thanks to the early work of Maxwell and “recent” work of others, including industry, colour photography thrives today Thanks to the early work of Maxwell and “recent” work of others, including industry, colour photography thrives today However, except for low cost and large format photography, amateurs and professionals are replacing their film cameras with digital ones However, except for low cost and large format photography, amateurs and professionals are replacing their film cameras with digital ones

10. History of Digital Cameras Einstein published three papers in 1905, one of which explained the photoelectric effect Einstein published three papers in 1905, one of which explained the photoelectric effect Photons striking a metal liberate electrons, which can carry a current Photons striking a metal liberate electrons, which can carry a current Kinetic energy depends on light frequency while the number of electrons depends on intensity Kinetic energy depends on light frequency while the number of electrons depends on intensity

11. History of Digital Cameras The advent of Quantum Mechanics in the early 1900s led to advances in solid-state physics The advent of Quantum Mechanics in the early 1900s led to advances in solid-state physics Russell Ohl invented the p-n junction, a photosensitive diode that (usually) passes current in only one direction, in 1940 Russell Ohl invented the p-n junction, a photosensitive diode that (usually) passes current in only one direction, in 1940 John Bardeen, Walter Brattain and William Shockley invented the transistor in 1947, a semiconductor device that can act as an amplifier or a switch John Bardeen, Walter Brattain and William Shockley invented the transistor in 1947, a semiconductor device that can act as an amplifier or a switch

12. History of Digital Cameras Working independently, Jack Kilby and Robert Noyce invented the integrated circuit in 1959 Working independently, Jack Kilby and Robert Noyce invented the integrated circuit in 1959 Transistors, diodes, capacitors, resistors and wiring could be fabricated on a single crystal of semiconductor material, e.g. silicon Transistors, diodes, capacitors, resistors and wiring could be fabricated on a single crystal of semiconductor material, e.g. silicon George Smith and Willard Boyle invented the CCD camera in 1969 George Smith and Willard Boyle invented the CCD camera in 1969 Digital, as opposed to analog, CCD cameras came in the late 1980s Digital, as opposed to analog, CCD cameras came in the late 1980s

13. Digital Cameras Today A digital camera consists of many components (optics, housing, battery, memory etc.), of which the image sensor is principal A digital camera consists of many components (optics, housing, battery, memory etc.), of which the image sensor is principal With market revenues of $1.7 billion in 2003, there is widespread research and development in a variety of image sensor designs With market revenues of $1.7 billion in 2003, there is widespread research and development in a variety of image sensor designs Modern designs may be either charge coupled device (CCD) sensors or complementary metal-oxide-semiconductor (CMOS) sensors Modern designs may be either charge coupled device (CCD) sensors or complementary metal-oxide-semiconductor (CMOS) sensors

14. CCD Image Sensors March photo-generated charge systematically from an array of pixels to an output amplifier March photo-generated charge systematically from an array of pixels to an output amplifier Established technology Established technology High resolution, high sensitivity, low noise High resolution, high sensitivity, low noise Fabrication process is optimised for imaging Fabrication process is optimised for imaging Market share of 93% in 1999 (49% in 2004?) Market share of 93% in 1999 (49% in 2004?)

15. CMOS Image Sensors Work like memory array with photosensitive pixels instead of memory cells Work like memory array with photosensitive pixels instead of memory cells Signal processing may be included on the same die Signal processing may be included on the same die High yield of working chips and good video performance High yield of working chips and good video performance May be fabricated by the makers of microchips May be fabricated by the makers of microchips Market share of 7% in 1999 (51% in 2004?) Market share of 7% in 1999 (51% in 2004?)

16. Test Your Knowledge In what ways are the digital cameras on the market better than human eyes? In what ways are the digital cameras on the market better than human eyes? And, in what ways are human eyes better than the digital cameras on the market? And, in what ways are human eyes better than the digital cameras on the market?

17. History of the Human Eye Darwin wrote in 1859: “To suppose that the eye…could have been formed by natural selection, seems, I freely confess, absurd in the highest possible degree. Yet reason tells me…the difficulty of believing that a perfect and complex eye could be formed by natural selection…can hardly be considered real.” Darwin wrote in 1859: “To suppose that the eye…could have been formed by natural selection, seems, I freely confess, absurd in the highest possible degree. Yet reason tells me…the difficulty of believing that a perfect and complex eye could be formed by natural selection…can hardly be considered real.” Biologists have since filled in many gaps Biologists have since filled in many gaps

18. The Human Eye Today There are 6–7 million cones, for bright light vision with fine detail and colour, and 75–150 million rods, for dim light vision with coarse detail and no colour There are 6–7 million cones, for bright light vision with fine detail and colour, and 75–150 million rods, for dim light vision with coarse detail and no colour Today, the resolution of digital pixels (5–10 μm) is closing on the resolution of foveal cones (2–3 μm) Today, the resolution of digital pixels (5–10 μm) is closing on the resolution of foveal cones (2–3 μm) © John W. Kimball http://users.rcn.com /jkimball.ma.ultran et/BiologyPages/

19. The Human Eye Today The human eye is a remarkable organ not only because of its ability to sense images but especially because of its ability to process the image before sending a signal to the brain The human eye is a remarkable organ not only because of its ability to sense images but especially because of its ability to process the image before sending a signal to the brain The eye encodes the abundant visual input in such a way that the limited neural output retains the most significant descriptors of the scene while the rest are discarded The eye encodes the abundant visual input in such a way that the limited neural output retains the most significant descriptors of the scene while the rest are discarded Most digital cameras do little pre-processing Most digital cameras do little pre-processing

20. Dynamic Range The human eye will capture brightness and colour detail (left) that a typical digital camera will fail to capture (right) The human eye will capture brightness and colour detail (left) that a typical digital camera will fail to capture (right) These images were computed from a series of photographs— no human eye or digital camera was harmed in the process These images were computed from a series of photographs— no human eye or digital camera was harmed in the process Photographic data came from Paul Debevec and the visual model came from Gregory Larson and his colleagues Photographic data came from Paul Debevec and the visual model came from Gregory Larson and his colleagues

21. Colour Constancy The human eye factors out the illumination to large measure when identifying colours The human eye factors out the illumination to large measure when identifying colours A typical digital camera uses a simple method to balance colours that will fail noticeably at times A typical digital camera uses a simple method to balance colours that will fail noticeably at times Better methods exist but they are complicated Better methods exist but they are complicated Acquired coloursPerceived colours Typical correctionBetter correction © Computational Vision Lab, Simon Fraser University (annotation added) http://www.cs.sfu.ca/~colour/research/col our-constancy.html

22. Fixed Pattern Noise Any two photoreceptors in a retina and any two photodetectors in an image sensor are never perfectly identical Any two photoreceptors in a retina and any two photodetectors in an image sensor are never perfectly identical A varying response to light stimulus causes “fixed pattern noise” A varying response to light stimulus causes “fixed pattern noise” Camera designers work to correct or reduce the FPN; neurons adapt (?) Camera designers work to correct or reduce the FPN; neurons adapt (?)

23. Temperature Stability Unlike the human body, a digital camera cannot regulate temperature Unlike the human body, a digital camera cannot regulate temperature The response of a pixel to a light stimulus will depend on temperature The response of a pixel to a light stimulus will depend on temperature When the temperature dependence varies from one pixel to another, FPN will appear When the temperature dependence varies from one pixel to another, FPN will appear

24. My Research Pixel & readout circuits and image processing may be improved Pixel & readout circuits and image processing may be improved Camera designers face challenges when using CMOS processes with feature sizes ≤ 180 nm Camera designers face challenges when using CMOS processes with feature sizes ≤ 180 nm Photodetectors may be fabricated in a thin film deposited on top of the CMOS microchip Photodetectors may be fabricated in a thin film deposited on top of the CMOS microchip

25. Linear Pixels Linear pixels (either CCD or CMOS type) “count” photons over a discrete period of time Linear pixels (either CCD or CMOS type) “count” photons over a discrete period of time They produce a voltage directly proportional to the light intensity They produce a voltage directly proportional to the light intensity Unfortunately, the response may saturate white or black easily Unfortunately, the response may saturate white or black easily © IMS Chips http://www.ims-chips.de/

26. Logarithmic Pixels Logarithmic pixels (CMOS only) measure the “rate” of photon incidence continuously Logarithmic pixels (CMOS only) measure the “rate” of photon incidence continuously They produce a voltage directly proportional to the logarithm of the light intensity They produce a voltage directly proportional to the logarithm of the light intensity The response is similar to that of human vision The response is similar to that of human vision © IMS Chips http://www.ims-chips.de/

27. My Research Logarithmic pixels are great for high dynamic range imaging but… Logarithmic pixels are great for high dynamic range imaging but… FPN is worse compared to typical linear pixels FPN is worse compared to typical linear pixels Colours are worse than for typical linear pixels Colours are worse than for typical linear pixels Temperature stability is hardly understood for log (and linear) pixels Temperature stability is hardly understood for log (and linear) pixels

28. Fixed Pattern Noise By studying the causes of FPN, I developed a method to correct it By studying the causes of FPN, I developed a method to correct it Images of a uniform surface are used to define corrections Images of a uniform surface are used to define corrections My correction reduces the FPN to the same order as the random temporal noise My correction reduces the FPN to the same order as the random temporal noise

29. Colour Rendition I also showed how to render accurate colours with logarithmic pixels I also showed how to render accurate colours with logarithmic pixels Images of a reference chart are used to define a colour mapping Images of a reference chart are used to define a colour mapping The perceptual error of the rendered colours is comparable to the error of typical cameras The perceptual error of the rendered colours is comparable to the error of typical cameras Rendered Reference

30. Temperature Stability The dark response of a pixel depends only on temperature The dark response of a pixel depends only on temperature Thus, it may be used to correct FPN due to temperature in the light response Thus, it may be used to correct FPN due to temperature in the light response Experiments support this conclusion but simulation results are shown for clarity Experiments support this conclusion but simulation results are shown for clarity

31. My Research A digital camera should render images with a maximum of perceptual accuracy (as per the human eye) using a minimum of effort A digital camera should render images with a maximum of perceptual accuracy (as per the human eye) using a minimum of effort A logarithmic imager responds to light much like the human eye; pre-processing and better electronics will improve the image quality A logarithmic imager responds to light much like the human eye; pre-processing and better electronics will improve the image quality My past work has focused on maximising the perceptual accuracy; my future work will focus on minimising the effort required My past work has focused on maximising the perceptual accuracy; my future work will focus on minimising the effort required

32. Your Questions

33. Resources Robert Leggat, “A History of Photography”, http://www.rleggat.com/photohistory/ Robert Leggat, “A History of Photography”, http://www.rleggat.com/photohistory/ http://www.rleggat.com/photohistory/ Mary Bellis, “The History of Computers”, http://inventors.about.com/ Mary Bellis, “The History of Computers”, http://inventors.about.com/ http://inventors.about.com/ Bob Patterson, “The Evolution of Eyes”, http://www.origins.tv/darwin/eyes.htm Bob Patterson, “The Evolution of Eyes”, http://www.origins.tv/darwin/eyes.htm http://www.origins.tv/darwin/eyes.htm Microsoft Office Online, Clip Art and Media, http://office.microsoft.com/clipart/ Microsoft Office Online, Clip Art and Media, http://office.microsoft.com/clipart/ http://office.microsoft.com/clipart/