1. CS 414 - Spring 2012 CS 414 – Multimedia Systems Design Lecture 4 – Visual Perception and Digital Image Representation Klara Nahrstedt Spring 2012

2. CS 414 - Spring 2012 Administrative Groups are formed and names have been sent to TSG and Barb Leisner We will inform you about group directories as soon as we have information from TSG

3. Administrative Leasing Process from Barb Leisner  Lease one Logitech camera - two cameras within one group to start MP1, and then for MP2/MP3.  Leasing process starts on January 25  Pick up the camera from Barb Leisner office, 2312 SC  Bring your student ID to sign for the camera  Each cs414 student is responsible for his/her own camera if you loose it (or badly damage) and you don’t have police report, you pay for it (charged to your student account at the end of the semester)  Hours to pick up camera: Monday –Friday 9am-5pm  No camera pickup on Saturday and Sunday CS 414 - Spring 2012

4. Today Introduced Concepts Two Important Metrics for Digital Audio  Signal-to-Noise Ratio (dB)  Digital Audio Data Rates (bits per second) Human Visual System Digital Images  Sampling  Quantization  Spatial Resolution CS 414 - Spring 2012

5. Signal-to-Noise Ratio (metric to quantify quality of digital audio) CS 414 - Spring 2012

6. Signal To Noise (SNR) Ratio Measures strength of signal to noise SNR (in DB)= Given sound form with amplitude in [-A, A] Signal energy = A 0 -A CS 414 - Spring 2012

7. Modeling of Noise - Quantization Error Difference between actual and sampled value  amplitude between [-A, A]  quantization levels = N e.g., if A = 1, N = 8, = 1/4 CS 414 - Spring 2012

8. Compute Signal to Noise Ratio Signal energy = ; Noise energy = ; Noise energy = Signal-to-Noise = SNR depends on number of bits (number of quantization levels) assigned to signal Every bit increases SNR by ~ 6 decibels CS 414 - Spring 2012

9. Audio Data Rate Data rate = best sample rate * quantization (bits per sample) * channel Derived from  Nyquist Data Rate = 2*H log 2 N, where H is basic sampling rate, N is number of quantization levels Compare rates for CD vs. mono audio  8000 samples/second * 8 bits/sample * 1 channel = 8 kBytes / second  44,100 samples/second * 16 bits/sample * 2 channel = 176 kBytes / second ~= 10MB / minute CS 414 - Spring 2012

10. Integrating Aspects of Multimedia CS 414 - Spring 2012 Image/Video Capture Image/Video Information Representation Media Server Storage Transmission Compression Processing Audio/Video Presentation Playback Audio/Video Perception/ Playback Audio Information Representation Transmission Audio Capture A/V Playback

11. Human Visual System Eyes, optic nerve, parts of the brain Transforms electromagnetic energy

12. Human Visual System Image Formation  cornea, sclera, pupil, iris, lens, retina, fovea Transduction  retina, rods, and cones  Retina has photosensitive receptors at back of eye Processing  optic nerve, brain

13. Rods vs Cones (Responsible for us seeing brightness and color) Contain photo-pigment Respond to low energy Enhance sensitivity Concentrated in retina, but outside of fovea One type, sensitive to grayscale changes Contain photo-pigment Respond to high energy Enhance perception Concentrated in fovea, exist sparsely in retina Three types, sensitive to different wavelengths ConesRods CS 414 - Spring 2012

14. Tri-stimulus Theory 3 types of cones (6/7 Mil. of them)  Red = L cones, Green = M cones, Blue = S cones  Ratio differentiates for each person  E.g., Red (64%), Green (32%), rest S cones  E.g., L(50.6%), M(44.2%), rest S cones Each type most responsive to a narrow band electro-magnetic waves  red and green absorb most energy, blue the least Light stimulates each set of cones differently, and the ratios produce sensation of color CS 414 - Spring 2012

15. Color and Visual System Color refers to how we perceive a narrow band of electromagnetic energy  source, object, observer Visual system transforms light energy into sensory experience of sight

16. Color Perception (Color Theory) Hue  Refers to pure colors  dominant wavelength of the light Saturation  Perceived intensity of a specific color  how far color is from a gray of equal intensity Brightness (lightness)  perceived intensity CS 414 - Spring 2012 Hue Scale Saturation Original lightness Source: Wikipedia

17. Color Perception (Hue) CS 414 - Spring 2012 Relation between “hue” of colors (spectrum of colors) with maximal saturation in HSV and HSL with Their corresponding RGB coordinates HSV = Hue, Saturation, Value (Lightness) HSL = Hue, Saturation, Lightness

18. Digitalization of Images – Capturing and Processing CS 414 - Spring 2012

19. Capturing Real-World Images Picture – two dimensional image captured from a real-world scene that represents a momentary event from the 3D spatial world CS 414 - Spring 2012 W3 W1 W2 r s Fr= function of (W1/W3); s=function of (W2/W3)

20. Image Concepts - Sampling An image is a function of intensity values over a 2D plane I(r,s) Sample function at discrete intervals to represent an image in digital form  matrix of intensity values for each color plane  intensity typically represented with 8 bits Sample points are called pixels CS 414 - Spring 2012

21. Digital Image Sampling Sample = pixel Image Size (in pixels) Image Size = Height x Width (in pixels) 320x240 pixels 640x480 pixels 1920x1080pixels CS 414 - Spring 2012

22. Digital Images - Quantization Quantization = number of bits per pixel Example: if we would sample and quantize standard TV picture (525 lines) by using VGA (Video Graphics Array),  video controller creates matrix 640x480pixels, and  each pixel is represented by 8 bit integer (256 discrete gray levels) CS 414 - Spring 2012

23. Image Representations Black and white image  single color plane with 2 bits Grey scale image  single color plane with 8 bits Color image  three color planes each with 8 bits  RGB, CMY, YIQ, etc. Indexed color image  single plane that indexes a color table Compressed images  TIFF, JPEG, BMP, etc. 2gray levels4 gray levels

24. Digital Image Representation (3 Bit Quantization) CS 414 - Spring 2012

25. Color Quantization Example of 24 bit RGB Image CS 414 - Spring 2012 24-bit Color Monitor

26. Image Representation Example 128135166138190132 129255105189167190 229213134111138187 135190 255167 213138 128138 129189 229111 166132 105190 134187 24 bit RGB Representation (uncompressed) Color Planes

27. Graphical Representation CS 414 - Spring 2012

28. Image Properties (Color) CS 414 - Spring 2012

29. Color Histogram CS 414 - Spring 2012

30. Spatial and Frequency Domains Spatial domain  refers to planar region of intensity values at time t Frequency domain  think of each color plane as a sinusoidal function of changing intensity values  refers to organizing pixels according to their changing intensity (frequency) CS 414 - Spring 2012

31. Spatial Resolution and Brightness Spatial Resolution (depends on: )  Image size  Viewing distance Brightness  Perception of brightness is higher than perception of color  Different perception of primary colors Relative brightness: green:red:blue= 59%:30%:11% CS 414 - Spring 2011 Source: wikipedia

32. Image Size (in Bits) Image Size = Height x Width X Bits/pixel Example:  Consider image 320x240 pixels with 8 bits per pixel  Image takes storage 7680 x 8 bits = 61440 bits or 7680 bytes CS 414 - Spring 2012

33. Summary Important Image Processing Functions (see Computer Vision/Image Processing classes)  Filtering  Edge detection  Image segmentation  Image recognition Formatting Conditioning Marking Grouping Extraction Matching  Image synthesis CS 414 - Spring 2012