Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 1 Computational Architectures in Biological Vision, USC, Spring 2001 Lecture 4: Introduction to Vision. Reading Assignments: Chapters 2 and 3 of textbook.

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 2 Today’s lecture - The challenges: -Optics and image formation -Sampling and image representation -Theoretical limits - The biological approach: -Organization of the primate retina -Trading accuracy for coverage: moving eyes - The engineering approach: -Arrays of photosensitive sensors -On-board processing and VLSI sensors -Trading accuracy for coverage: multiple moving cameras

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 3 Projection

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 4 Projection

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 5 Convention: Visual Angle Rather than reporting two numbers (size of object and distance to observer), we will combine both into a single number: visual angle e.g., the moon: about 0.5deg visual angle your thumb nail at arm’s length: about 1.5deg visual angle 1deg visual angle: 0.3mm on retina

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 6 Charge-Coupled Devices Uniform array of sensors Very little on-board processing Very inexpensive

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 7 Optics limitations: acuity

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 8 Sampling

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 9 Sampling

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 10 Sampling

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 11 The Big Question

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 12 Convolution & Fourier Transforms

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 13 Sampling in the frequency domain

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 14 Reconstruction

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 15 Aliasing

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 16 Sampling Theorem

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 17 Aliasing

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 18 Eye Anatomy

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 19 Visual Pathways

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 20 Image Formation Accomodation: ciliary muscles can adjust shape of lens, yielding an effect equivalent to an autofocus.

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 21 Phototransduction Cascade Net effect: light (photons) is transformed into electrical (ionic) current.

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 22 Rods and Cones Roughly speaking: 3 types of cones, sensitive to red, green and blue.

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 23 Processing layers in retina

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 24 Retinal Processing

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 25 Center-Surround Center-surround organization: neurons with receptive field at given location receive inhibition from neurons with receptive fields at neighboring locations (via inhibitory interneurons).

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 26 Early Processing in Retina

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 27 Color Processing

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 28 Over-representation of the Fovea Fovea: central region of the retina (1-2deg diameter); has much higher density of receptors, and benefits from detailed cortical representation.

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 29 Fovea and Optic Nerve

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 30 Blind Spot

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 31 Retinal Sampling

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 32 Retinal Sampling

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 33 Seeing the world through a retina

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 34 Sampling & Optics Because of blurring by the optics, we cannot see infinitely small objects…

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 35 Sampling & optics The sampling grid optimally corresponds to the amount of blurring due to the optics!

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 36 Retinal Implants Problems: - long-term toxicity (neurons close to electrodes die) - image quality so low that users hate it!

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 37 Silicon Retina Smoothing network: allows system to adapt to various light levels.

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 38 VLSI sensor vith retinal organization

Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC Lecture 4: Introduction to Vision 39 Combining cameras From fixed cameras with overlapping field of view, we can reconstruct a virtual camera with any point of view: virtual PTZ (pan-tilt-zoom).