Mobile Robotics: 6. Vision 1 Dr. Brian Mac Namee (

2 of 25 2 of 34 Acknowledgments These notes are based (heavily) on those provided by the authors to accompany “Introduction to Autonomous Mobile Robots” by Roland Siegwart and Illah R. Nourbakhsh More information about the book is available at: The book can be bought at: The MIT Press and Amazon.com The MIT PressAmazon.com

3 of 25 3 of 34 Today’s Lecture Why is vision hard? Brief historical overview –From early cameras to digital cameras Low-level robot vision –Camera as sensor –Color representation –Object detection

4 of 25 4 of 34 Vision In General Vision is our most powerful sense providing us with an enormous amount of information about our environment and enables us to interact intelligently with the environment It is therefore not surprising that an enormous amount of effort has occurred to give machines a sense of vision Vision is also our most complicated sense –Whilst we can reconstruct views with high resolution on photographic paper, understanding how the brain processes the information from our eyes is still in its infancy

5 of 25 5 of 34 Vision In General (cont…) When an image is recorded through a camera, a 3- D scene is projected onto a 2-D plane In order to try and recover some “useful information” from the scene, usually edge detectors are used to find the contours of the objects From these edges or edge fragments, much research time has to been spent attempting to produce fool proof algorithms which can provide all the necessary information required to reconstruct the 3-D scene which produced the 2-D image The interpretation of 3-D scenes from 2-D images is not a trivial task

6 of 25 6 of 34 Vision Is Hard! (Segmentation)

7 of 25 7 of 34 Vision Is Hard! (Classification)

8 of 25 8 of 34 Vision Is Hard! (Perspectives)

9 of 25 9 of 34 Vision Is Hard! (Brightness Adaptation) For more great illusion examples take a look at:

10 of of 34 Vision Is Hard! (Illusions) Our visual systems play lots of interesting tricks on us

11 of of 34 Vision Is Hard! (Illusions)

12 of of 34 Vision Is Hard! (Illusions) Stare at the cross in the middle of the image and think circles

13 of of 34 Camera Obscura Mo Ti, Chinese philosopher, 5th Century B.C. –Described linear light paths, pinhole image formation Leonardo da Vinci ( ) –Demonstrated camera obscura (lens added later) Frisius (1544) Portmerion Village, North Wales Photograph of camera obscura interior:

14 of of 34 Toward Photography People sought a way to “fix” the images at the back of the camera obscura Pursued decades of experimentation with light- sensitive salts, acids, etc. First photograph produced when?

15 of of 34 Harry Ransom CenterKodak (reproduction) First Photograph Joseph Nicéphore Niépce “View from the Window at Le Gras”, c Aluminum plate coated with light-sensitive material More information on the first photograph is available at: Joseph Nicéphore Niépce

16 of of 34 First Digital Cameras Photoelectric effect (Hertz 1887; Einstein 1905) Charge-coupled devices as storage (late 1960’s) Light sensing, pixel row readout (early 1970’s) First electronic CCD still- image camera (1975): –Fairchild CCD element –Resolution: 100 x 100 b&w –Image capture time: 23 sec., mostly writing cassette tape –Total weight: 8½ pounds Kodak, c. 1975

17 of of 34 Modern Digital Cameras Today, fifty Euro buys a camera with: –640 x 480 pixel resolution at 30Hz –1280 x 960 still image resolution –24-bit RGB pixels (8 bits per channel) –Automatic gain control, color balancing –On-chip lossy compression algorithms –Uncompressed images if desired –Integrated microphone, USB interface –Limitations Narrow dynamic range Narrow FOV, with fixed spatial resolution No motion / active vision capabilities

18 of of 34 Vision-Based Sensors: Hardware CCD (light-sensitive, discharging capacitors of 5 to 25 micron) CMOS (Complementary Metal Oxide Semiconductor) technology

19 of of 34 What Is A Digital Image? A digital image is a 2-D representation of a 3-D scene as a finite set of digital values, called picture elements or pixels

20 of of 34 What Is A Digital Image? (cont…) Pixel values typically represent gray levels, colours, heights, opacities etc Remember digitization implies that a digital image is an approximation of a real scene 1 pixel

21 of of 34 Digital Image Contents Why are pixels represented as “RGB”? –Is world made of red, green, and blue “stuff”? … Answer requires a digression (or two) about human vision, cameras as sensors Image coordinates (pixels) u v IOIO width height

22 of of 34 Visible Light Spectrum Freedman & Kaufmann, Universe Solar (ECI, Oxford) Incandescent (Wikipedia)

23 of of 34 Image As Measurement What does eye/camera actually observe? … the product of illumination spectrum with absorption or reflection spectrum! = (at each image point) X Illumination spectrum IJVS Reflection spectrum

24 of of 34 Eye Anatomy Spectrum incident on light-sensitive retina Incident spectral distribution After Isaka (2004) (View of R eye from above)Rods and cones

25 of of 34 Blind-Spot Experiment Draw an image similar to that below on a piece of paper (the dot and cross are about 6 inches apart) Close your right eye and focus on the cross with your left eye Hold the image about 20 inches away from your face and move it slowly towards you The dot should disappear!

26 of of 34

27 of of 34 Cone Sensitivities Three cone types (S, M, and L) are roughly blue, green, and red sensors, respectively Their peak sensitivities occur at ~430nm, ~560nm, and ~610nm for the “average” human Rods & cones, ~1.35 mm from center of fovea Rods & cones, ~8 mm from center of fovea Cone sensitivities as a function of wavelength 4 m

28 of of 34 Color Perception The cones form a spectral “basis” for visible light; incident spectral distribution differentially excites S,M,L cones, leading to color vision = (at each cone site) X IJVS X

29 of of 34 Origin Of RGB CCD Sensors So, in a concrete sense, CCD chips are designed as RGB sensors in order to emulate the human visual system (Vaytek)

30 of of 34 RGB Colour Model Think of R, G, B as color orthobasis (0,1,0) – pure green (0,0,1) – pure blue (1,0,0) pure red (1,1,1) - white (0,0,0) - black (hidden)

31 of of 34 HSV Colour Model More robust against illumination changes Still must confront noise, specularity etc.

32 of of 34 Object Detection Suppose we want to detect an object (e.g., a colored ball) in the field of view We simply need to identify the pixels of some desired colour in the image … right? Image coordinates (pixels) u v IOIO width height

33 of of 34 Real-World Images Occluded light source Specular highlights Mixed pixels Complex surface geometry (self- shadowing) Noise!

34 of of 34 Summary Vision is our most useful sense, but is also the most difficult to replicate Digital cameras have evolved to be powerful, cheap and accurate Artificial vision systems tend to be modelled after the human eye How do we do it?

35 of of 34 Questions? ?