Video Processing & Communications Basics of Video Courtesy of Professor Yao Wang Polytechnic University, Brooklyn, NY11201 yao@vision.poly.edu

Outline Video capture Analog video Digital video Video display Photometric model Geometric model Analog video Progressive vs. interlaced rasters in analog TV system Different color representations: YUV/YIQ Digital video Sampling/quantization Y’CbCr format Video display Spatial/temporal/bit-depth resolution Adapted from Yao Wang, 2004 Video Basics

Photometric Model of Video Capture Courtesy of Onur Guleryuz Adapted from Yao Wang, 2004 Video Basics

Geometric Model of Video Capture point Camera center The image of an object is reversed from its 3-D position. The object appears smaller when it is farther away. 2-D image Image plane Adapted from Yao Wang, 2004 Video Basics

Implication of Models in Analog World Miniature building Explosion from The Mummy Lighting in Filmmaking Adapted from Yao Wang, 2004 Video Basics

Progressive and Interlaced Raster Scans Field 1 Field 2 Progressive Frame Interlaced Frame Horizontal retrace Vertical retrace Interlaced scan is developed to provide a trade-off between temporal and vertical resolution, for a given, fixed data rate (number of line/sec). Adapted from Yao Wang, 2004 Video Basics

Color TV Broadcasting and Receiving Adapted from Yao Wang, 2004 Video Basics

Why not using RGB directly? R,G,B components are correlated Transmitting R,G,B components separately is redundant More efficient use of bandwidth is desired RGB->YC1C2 transformation Decorrelating: Y,C1,C2 are uncorrelated C1 and C2 require lower bandwidth Y (luminance) component can be received by B/W TV sets Color transformation is a compromised solution, but the ultimate one Adapted from Yao Wang, 2004 Video Basics

YIQ in NTSC I (in-phase): orange-to-cyan Q (quadrature): green-to-purple (human eye is less sensitive) Q can be further bandlimited than I Phase=Arctan(Q/I) = hue, Magnitude=sqrt (I^2+Q^2) = saturation Hue is better retained than saturation Recall: Quadrature amplitude modulation (QAM) in digital communication Adapted from Yao Wang, 2004 Video Basics

Color Image Y image I image (orange-cyan) Q image (green-purple)

I and Q on the color circle Q: green-purple I: orange-cyan Adapted from Yao Wang, 2004 Video Basics

Conversion between RGB and YIQ RGB -> YIQ Y = 0.299 R + 0.587 G + 0.114 B I = 0.596 R -0.275 G -0.321 B Q = 0.212 R -0.523 G + 0.311 B YIQ -> RGB R =1.0 Y + 0.956 I + 0.620 Q, G = 1.0 Y - 0.272 I -0.647 Q, B =1.0 Y -1.108 I + 1.700 Q. Adapted from Yao Wang, 2004 Video Basics

YUV in PAL Adapted from Yao Wang, 2004 Video Basics

YUV/RGB Conversion Numerical approximations Y = (( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16) U = ( ( -38 * R - 74 * G + 112 * B + 128) >> 8) + 128 V = ( ( 112 * R - 94 * G - 18 * B + 128) >> 8) + 128 Adapted from Yao Wang, 2004 Video Basics

YIQ/YUV Comparison Adapted from Yao Wang, 2004 Video Basics

Different Color TV Systems Adapted from Yao Wang, 2004 Video Basics

Who uses what? From http://www.stjarnhimlen.se/tv/tv.html#worldwide_0 Adapted from Yao Wang, 2004 Video Basics

Digital Video Sampling Quantization Taken from EE465: Image Acquisition Adapted from Yao Wang, 2004 Video Basics

BT.601* Video Format * BT.601 is formerly known as CCIR601 Adapted from Yao Wang, 2004 Video Basics

RGB <--> Y’CbCr Analog video Digital video Adapted from Yao Wang, 2004 Video Basics

YUV vs. Y’CbCr Adapted from Yao Wang, 2004 Video Basics

Chrominance Subsampling Formats Adapted from Yao Wang, 2004 Video Basics

Digital Video Formats Adapted from Yao Wang, 2004 Video Basics

4:2:0 YUV Video U: 144-by-176 V: 144-by-176 Y: 288-by-352 Adapted from Yao Wang, 2004 Video Basics

Tricky Photometric Situations Shadow causes problem to background extraction Video enhancement Adapted from Yao Wang, 2004 Video Basics

Geometric Invariance Adapted from Yao Wang, 2004 Video Basics

Video Display High-end Low-end If the resolution of display device is higher than that of video sequence, what can we do? Tradeoff between quality and complexity Subjective evaluation of video quality Low-end If the resolution of display device is lower than that of video sequence, what can we do? What if the bit-depth resolution is lower? (e.g., display video on PDAs and portable DVDs) It is the last and the least-researched component in visual communication systems Adapted from Yao Wang, 2004 Video Basics

Resolution, Resolution, Resolution temporal 300fps 30fps 1M 10M spatial 8bpp 32bpp Bit-depth Adapted from Yao Wang, 2004 Video Basics

High Dynamic Range Imaging Q: Can we generate a HDR image (16bpp) by a standard camera? A: Yes, adjust the exposure and fuse multiple LDR images together Adapted from Yao Wang, 2004 Video Basics

HDR Display (after Toner Mapping) Note that any commercial display devices we see these days are NOT HDR Adapted from Yao Wang, 2004 Video Basics