Klara Nahrstedt Spring 2009 CS 414 – Multimedia Systems Design Lecture 5 – Digital Video Representation Klara Nahrstedt Spring 2009 CS 414 - Spring 2009

Administrative MP1 is out (January 28) Deadline of MP1 is February 9 (Monday) Demonstration of MP1 will be Monday, February 9 at 5-7pm in 216 Siebel Center Sign-up sheet will be available in class on February 9 (during class) CS 414 - Spring 2009

Color and Visual System Color refers to how we perceive a narrow band of electromagnetic energy source, object, observer Visual system transforms light energy into sensory experience of sight ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Human Visual System Eyes, optic nerve, parts of the brain Transforms electromagnetic energy ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Human Visual System Image Formation Transduction Processing cornea, sclera, pupil, iris, lens, retina, fovea Transduction retina, rods, and cones Processing optic nerve, brain ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Retina and Fovea Retina Fovea Retina has photosensitive receptors at back of eye Fovea is small, dense region of receptors only cones (no rods) gives visual acuity Outside fovea fewer receptors overall larger proportion of rods Retina Fovea ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Transduction (Retina) Transform light to neural impulses Receptors signal bipolar cells Bipolar cells signal ganglion cells Axons in the ganglion cells form optic nerve Bipolar cells Rods Ganglion Cones Optic nerve ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Rods vs Cones Cones Rods Contain photo-pigment Respond to high energy Enhance perception Concentrated in fovea, exist sparsely in retina Three types, sensitive to different wavelengths Contain photo-pigment Respond to low energy Enhance sensitivity Concentrated in retina, but outside of fovea One type, sensitive to grayscale changes CS 414 - Spring 2009 ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Tri-stimulus Theory 3 types of cones (6 to 7 million of them) Red = L cones, Green = M cones, Blue = S cones Ratio differentiates for each person E.g., Red (64%), Green (32%), rest S cones E.g., L(75.8%), M(20%), rest S cones E.g., L(50.6%), M(44.2%), rest S cones Source of information: See ‘cone cell’ in wikipedia www.colorbasics.com/tristimulus/index.php Each type most responsive to a narrow band red and green absorb most energy, blue the least Light stimulates each set of cones differently, and the ratios produce sensation of color The three types of cone cells are named: Short (S) for the short wavelengths Middle (M) for the medium wavelenghts Long (L) for the long wavelengths OK, not the most creative names, but they are quite easy to remember. The three types of cones are also known as blue, green or red receptors. They are able to receive three different wavelength of light: Short - Blue Medium - Green Long - Red That is why we need three parameters (tristimulus values) to describe a color. The tristimulus values measure the relative brightness of each primary color needed to stimulate the three color receptors of the eye to create the sensation of seeing a certain color. ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Color Perception (Color Theory) Hue distinguishes named colors, e.g., RGB dominant wavelength of the light Saturation Perceived intensity of a specific color how far color is from a gray of equal intensity Brightness (lightness) perceived intensity Hue Scale Original Saturation lightness CS 414 - Spring 2009 Source: Wikipedia ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Visual Perception: Resolution and Brightness Spatial Resolution (depends on: ) Image size Viewing distance Brightness Perception of brightness is higher than perception of color Different perception of primary colors Relative brightness: green:red:blue=59%:30%:11% B/W vs. Color CS 414 - Spring 2009 Source: wikipedia

Visual Perception: Temporal Resolution Effects caused by inertia of human eye Perception of 16 frames/second as continuous sequence Special Effect: Flicker CS 414 - Spring 2009

Temporal Resolution Flicker Higher refresh rate requires: Perceived if frame rate or refresh rate of screen too low (<50Hz) Especially in large bright areas Higher refresh rate requires: Higher scanning frequency Higher bandwidth CS 414 - Spring 2009

Visual Perception Influence Viewing distance Display ratio (width/height – 4/3 for conventional TV) Number of details still visible Intensity (luminance) CS 414 - Spring 2009

Television History 1927, Hoover made a speech in Washington while viewers in NY could see, hear him AT&T Bell Labs had the first “television” 18 fps, 2 x 3 inch screen, 2500 pixels ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Television Concepts Re-construction Production (capture) 2D array of light energy to electrical signals signals must adhere to known, structured formats Representation and Transmission popular formats include NTSC, PAL, SECAM Re-construction CRT technology and raster scanning display issues (refresh rates, temporal resolution) relies on principles of human visual system CS 414 - Spring 2009 ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Video Representations Composite NTSC - 6MHz (4.2MHz video), 29.97 fps PAL - 6-8MHz (4.2-6MHz video), 25 fps Component Maintain separate signals for color Color spaces RGB, YUV, YCRCB, YIQ CS 414 - Spring 2009 ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Color Coding: YUV PAL video standard YUV from RGB Y is luminance UV are chrominance YUV from RGB Y = .299R + .587G + .114B U = 0.492 (B - Y) V = 0.877 (R - Y) U-V plane at Y=0.5 CS 414 - Spring 2009 Source: wikipedia ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

YCrCb Subset of YUV that scales and shifts the chrominance values into range 0..1 Y = 0.299R + 0.587G + .114B Cr = ((B-Y)/2) + 0.5 Cb = ((R-Y)/1.6) + 0.5 CS 414 - Spring 2009 ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

YIQ NTSC standard YIQ from RGB Y = .299R + .587G + .114B I = .74 (R - Y) - .27 (B - Y) Q = 0.48 (R - Y) + 0.41 (B - Y) YIQ with Y=0.5 CS 414 - Spring 2009 Source: wikipedia ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

NTSC Video 525 scan lines per frame; 29.97 fps 33.37 msec/frame (1 second / 29.97 frames) scan line lasts 63.6 usec (33.37 msec / 525) aspect ratio of 4/3, gives 700 horizontal pixels 20 lines reserved for control information at the beginning of each field so only 485 lines of visible data CS 414 - Spring 2009 ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

NTSC Video Interlaced scan lines divide each frame into 2 fields, each of which is 262.5 lines phosphors in early TVs did not maintain luminance long enough (caused flicker) scanning also interlaced; can cause visual artifacts for high motion scenes CS 414 - Spring 2009 ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

HDTV Digital Television Broadcast (DTB) System Twice as many horizontal and vertical columns and lines as traditional TV Resolutions: 1920x1080 (1080p) – Standard HDTV Frame rate: options 50 or 60 frames per second CS 414 - Spring 2009

Pixel Aspect Ratio Computer Graphics parameter pixel aspect ratio is used in the context of computer graphics to describe the layout of pixels in a digitized image. Most digital imaging systems use a square grid of pixels—that is, they sample an image at the same resolution horizontally and vertically. But there are some devices that do not (most notably some common standard-definition formats in digital television and DVD-Video) so a digital image scanned at a vertical resolution twice that of its horizontal resolution (i.e. the pixels are twice as close together vertically as horizontally) might be described as being sampled at a 2:1 pixel aspect ratio, regardless of the size or shape of the image as a whole. Increasing the aspect ratio of an image makes its use of pixels less efficient, and the resulting image will have lower perceived detail than an image with an equal number of pixels, but arranged with an equal horizontal and vertical resolution. Beyond about 2:1 pixel aspect ratio, further increases in the already-sharper direction will have no visible effect, no matter how many more pixels are added. Hence an NTSC picture (480i) with 1000 lines of horizontal resolution is possible, but would look no sharper than a DVD. The exception to this is in situations where pixels are used for a purpose other than resolution - for example, a printer that uses dithering to simulate gray shades from black-or-white pixels, or analog videotape that loses high frequencies when dubbed. Computer Graphics parameter Mathematical ratio describing horizontal length of a pixel to its vertical height Used mainly in digital video editing software to properly scale and render video Into a new format CS 414 - Spring 2009 Source: wikipedia

Pixel aspect ratio (Standard 16:9) Video Format Description Resolution (WxH) Pixels Aspect Ratio Pixel aspect ratio (Standard 16:9) Video Format Description 1024×768 786,432 16:9 4:3 720p/XGA Used on PDP HDTV 1280×720 921,600 1:1 (square) 720p/WXGA Used on Digital television, 1366×768 1,049,088 Approx. 1:1 (square) 720p/WXGA - HDTV standard format Used on LCD/PDP HDTV displays 1024×1080 1,105,920 15:8 1080p Used on PDP displays HDTV 1280×1080 1,382,400 3:2 Used on PDP HDTV displays 1920×1080 2,073,600 1080p - HDTV standard format Used on all types of HDTV technologies 3840x2160 8,294,400 2160p Quad HDTV, CS 414 - Spring 2009

HDTV Interlaced and/or progressive formats MPEG-2 compressed streams Conventional TCs – use interlaced formats Computer displays (LCDs) – use progressive scanning MPEG-2 compressed streams In Europe (Germany) – MPEG-4 compressed streams CS 414 - Spring 2009

Aspect Ratio and Refresh Rate Conventional TV is 4:3 (1.33) HDTV is 16:9 (2.11) Cinema uses 1.85:1 or 2.35:1 Frame Rate NTSC is 60Hz interlaced (actually 59.94Hz) PAL/SECAM is 50Hz interlaced Cinema is 24Hz non-interlaced CS 414 - Spring 2009 Source: wikipedia ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

SMPTE Time Codes Society of Motion Picture and Television Engineers defines time codes for video HH:MM:SS:FF 01:12:59:16 represents number of pictures corresponding to 1hour, 12 minutes, 59 seconds, 16 frames If we consider 30 fps, then 59 seconds represent 59*30 frames, 12 minutes represent 12*60*30 frames and 1 hour represents 1*60*60*30 frames. For NTSC, SMPTE uses a 30 drop frame code increment as if using 30 fps, when really NTSC has only 29.97fps defines rules to remove the difference error SMPTE timecode is a set of cooperating standards to label individual frames of video or film with a timecode defined by the Society of Motion Picture and Television Engineers in the SMPTE 12M specification. Timecodes are added to film, video or audio material, and have also been adapted to synchronize music. They provide a time reference for editing, synchronisation and identification. Timecode is a form of media metadata. The invention of timecode made modern videotape editing possible, and led eventually to the creation of non-linear editing systems. SMPTE (pron :sim-tee) timecodes contains binary coded decimal hour:minute:second:frame identification and 32 bits for use by users. There are also drop-frame and colour framing flags and three extra 'binary group flag' bits used for defining the use of the user bits. The formats of other forms SMPTE timecodes are derived from that of the longitudinal timecode. Time code can have any of a number of frame rates: common ones are 24 frame/s (film) 25 frame/s (PAL colour television) 29.97 (30*1.000/1.001) frame/s (NTSC color television) 30 frame/s (American black-and-white television) (virtually obsolete) In general, SMPTE timecode frame rate information is implicit, known from the rate of arrival of the timecode from the medium, or other metadata encoded in the medium. The interpretation of several bits, including the "colour framing" and "drop frame" bits, depends on the underlying data rate. In particular, the drop frame bit is only valid for a nominal frame rate of 30 frame/s: see below for details. More complex timecodes such as Vertical interval timecode can also include extra information in a variety of encodings. SMPTE time code is a digital signal whose ones and zeroes assign a number to every frame of video, representing hours, minutes, seconds, frames, and some additional user/specified information such as tape number. For instance, the time code number 01:12:59:16 represents a picture 1 hour, 12 minutes, 59 seconds, and 16 frames into the tape. CS 414 - Spring 2009 ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Take Home Exercise Given a SMPTE time stamp, convert it back to the original frame number e.g., 00:01:00:10 CS 414 - Spring 2009 ORCHID Research Group Department of Computer Science, University of Illinois at Urbana-Champaign

Summary Digitization of Video Signals Digital Television (DTV) Composite Coding Component Coding Digital Television (DTV) DVB (Digital Video Broadcast) Satellite connections, CATV networks – best suited for DTV DVB-S – for satellites (also DVB-S2) DVB-C – for CATV CS 414 - Spring 2009