1. Klara Nahrstedt Spring 2009
CS 414 – Multimedia Systems Design Lecture 5 – Digital Video Representation
Klara Nahrstedt
Spring 2009
CS Spring 2009

2. Administrative MP1 is out (January 28)
Deadline of MP1 is February 9 (Monday)
Demonstration of MP1 will be Monday, February 9 at 5-7pm in 216 Siebel Center
Sign-up sheet will be available in class on February 9 (during class)
CS Spring 2009

3. Color and Visual System
Color refers to how we perceive a narrow band of electromagnetic energy
source, object, observer
Visual system transforms light energy into sensory experience of sight
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

4. Human Visual System Eyes, optic nerve, parts of the brain
Transforms electromagnetic energy
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

5. Human Visual System Image Formation Transduction Processing
cornea, sclera, pupil, iris, lens, retina, fovea
Transduction
retina, rods, and cones
Processing
optic nerve, brain
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

6. Retina and Fovea Retina Fovea
Retina has photosensitive receptors at back of eye
Fovea is small, dense region of receptors
only cones (no rods)
gives visual acuity
Outside fovea
fewer receptors overall
larger proportion of rods
Retina
Fovea
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

7. Transduction (Retina)
Transform light to neural impulses
Receptors signal bipolar cells
Bipolar cells signal ganglion cells
Axons in the ganglion cells form optic nerve
Bipolar cells
Rods
Ganglion
Cones
Optic nerve
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

8. Rods vs Cones Cones Rods Contain photo-pigment Respond to high energy
Enhance perception
Concentrated in fovea, exist sparsely in retina
Three types, sensitive to different wavelengths
Contain photo-pigment
Respond to low energy
Enhance sensitivity
Concentrated in retina, but outside of fovea
One type, sensitive to grayscale changes
CS Spring 2009
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

9. Tri-stimulus Theory 3 types of cones (6 to 7 million of them)
Red = L cones, Green = M cones, Blue = S cones
Ratio differentiates for each person
E.g., Red (64%), Green (32%), rest S cones
E.g., L(75.8%), M(20%), rest S cones
E.g., L(50.6%), M(44.2%), rest S cones
Source of information:
See ‘cone cell’ in wikipedia
Each type most responsive to a narrow band
red and green absorb most energy, blue the least
Light stimulates each set of cones differently, and the ratios produce sensation of color
The three types of cone cells are named:
Short (S) for the short wavelengths
Middle (M) for the medium wavelenghts
Long (L) for the long wavelengths
OK, not the most creative names, but they are quite easy to remember.
The three types of cones are also known as blue, green or red receptors. They are able to receive three different wavelength of light:
Short - Blue
Medium - Green
Long - Red
That is why we need three parameters (tristimulus values) to describe a color. The tristimulus values measure the relative brightness of each primary color needed to stimulate the three color receptors of the eye to create the sensation of seeing a certain color.
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

10. Color Perception (Color Theory)
Hue
distinguishes named colors, e.g., RGB
dominant wavelength of the light
Saturation
Perceived intensity of a specific color
how far color is from a gray of equal intensity
Brightness (lightness)
perceived intensity
Hue Scale
Original
Saturation
lightness
CS Spring 2009
Source: Wikipedia
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

11. Visual Perception: Resolution and Brightness
Spatial Resolution (depends on: )
Image size
Viewing distance
Brightness
Perception of brightness is higher than perception of color
Different perception of primary
colors
Relative brightness: green:red:blue=59%:30%:11%
B/W vs. Color
CS Spring 2009
Source: wikipedia

12. Visual Perception: Temporal Resolution
Effects caused by inertia of human eye
Perception of 16 frames/second as continuous sequence
Special Effect: Flicker
CS Spring 2009

13. Temporal Resolution Flicker Higher refresh rate requires:
Perceived if frame rate or refresh rate of screen too low (<50Hz)
Especially in large bright areas
Higher refresh rate requires:
Higher scanning frequency
Higher bandwidth
CS Spring 2009

14. Visual Perception Influence
Viewing distance
Display ratio (width/height – 4/3 for conventional TV)
Number of details still visible
Intensity (luminance)
CS Spring 2009

15. Television History
1927, Hoover made a speech in Washington while viewers in NY could see, hear him
AT&T Bell Labs had the first “television”
18 fps, 2 x 3 inch screen, 2500 pixels
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

16. Television Concepts Re-construction Production (capture)
2D array of light energy to
electrical signals
signals must adhere to known, structured formats
Representation and Transmission
popular formats include NTSC,
PAL, SECAM
Re-construction
CRT technology and raster scanning
display issues (refresh rates, temporal resolution)
relies on principles of human visual system
CS Spring 2009
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

17. Video Representations
Composite
NTSC - 6MHz (4.2MHz video), fps
PAL - 6-8MHz (4.2-6MHz video), 25 fps
Component
Maintain separate signals for color
Color spaces
RGB, YUV, YCRCB, YIQ
CS Spring 2009
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

18. Color Coding: YUV PAL video standard YUV from RGB Y is luminance
UV are chrominance
YUV from RGB
Y = .299R G B U = (B - Y) V = (R - Y)
U-V plane at Y=0.5
CS Spring 2009
Source: wikipedia
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

19. YCrCb
Subset of YUV that scales and shifts the chrominance values into range 0..1
Y = 0.299R G B Cr = ((B-Y)/2) Cb = ((R-Y)/1.6) + 0.5
CS Spring 2009
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

20. YIQ NTSC standard YIQ from RGB
Y = .299R G B I = .74 (R - Y) (B - Y) Q = 0.48 (R - Y) (B - Y)
YIQ with Y=0.5
CS Spring 2009
Source: wikipedia
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

21. NTSC Video 525 scan lines per frame; 29.97 fps
33.37 msec/frame (1 second / frames)
scan line lasts 63.6 usec (33.37 msec / 525)
aspect ratio of 4/3, gives 700 horizontal pixels
20 lines reserved for control information at the beginning of each field
so only 485 lines of visible data
CS Spring 2009
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

22. NTSC Video
Interlaced scan lines divide each frame into 2 fields, each of which is lines
phosphors in early TVs did not maintain luminance long enough (caused flicker)
scanning also interlaced; can cause visual artifacts for high motion scenes
CS Spring 2009
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

23. HDTV Digital Television Broadcast (DTB) System
Twice as many horizontal and vertical columns and lines as traditional TV
Resolutions:
1920x1080 (1080p) – Standard HDTV
Frame rate: options 50 or 60 frames per second
CS Spring 2009

24. Pixel Aspect Ratio Computer Graphics parameter
pixel aspect ratio is used in the context of computer graphics to describe the layout of pixels in a digitized image. Most digital imaging systems use a square grid of pixels—that is, they sample an image at the same resolution horizontally and vertically. But there are some devices that do not (most notably some common standard-definition formats in digital television and DVD-Video) so a digital image scanned at a vertical resolution twice that of its horizontal resolution (i.e. the pixels are twice as close together vertically as horizontally) might be described as being sampled at a 2:1 pixel aspect ratio, regardless of the size or shape of the image as a whole.
Increasing the aspect ratio of an image makes its use of pixels less efficient, and the resulting image will have lower perceived detail than an image with an equal number of pixels, but arranged with an equal horizontal and vertical resolution. Beyond about 2:1 pixel aspect ratio, further increases in the already-sharper direction will have no visible effect, no matter how many more pixels are added. Hence an NTSC picture (480i) with 1000 lines of horizontal resolution is possible, but would look no sharper than a DVD. The exception to this is in situations where pixels are used for a purpose other than resolution - for example, a printer that uses dithering to simulate gray shades from black-or-white pixels, or analog videotape that loses high frequencies when dubbed.
Computer Graphics parameter
Mathematical ratio describing horizontal length of a pixel to its vertical height
Used mainly in digital video editing software to properly scale and render video
Into a new format
CS Spring 2009
Source: wikipedia

25. Pixel aspect ratio (Standard 16:9) Video Format Description
Resolution (WxH)
Pixels
Aspect Ratio
Pixel aspect ratio (Standard 16:9)
Video Format
Description
1024×768
786,432
16:9
4:3
720p/XGA
Used on PDP HDTV
1280×720
921,600
1:1 (square)
720p/WXGA
Used on Digital television,
1366×768
1,049,088
Approx. 1:1 (square)
720p/WXGA - HDTV standard format
Used on LCD/PDP HDTV displays
1024×1080
1,105,920
15:8
1080p
Used on PDP displays HDTV
1280×1080
1,382,400
3:2
Used on PDP HDTV displays
1920×1080
2,073,600
1080p - HDTV standard format
Used on all types of HDTV technologies
3840x2160
8,294,400
2160p
Quad HDTV,
CS Spring 2009

26. HDTV Interlaced and/or progressive formats MPEG-2 compressed streams
Conventional TCs – use interlaced formats
Computer displays (LCDs) – use progressive scanning
MPEG-2 compressed streams
In Europe (Germany) – MPEG-4 compressed streams
CS Spring 2009

27. Aspect Ratio and Refresh Rate
Conventional TV is 4:3 (1.33)
HDTV is 16:9 (2.11)
Cinema uses 1.85:1 or 2.35:1
Frame Rate
NTSC is 60Hz interlaced (actually 59.94Hz)
PAL/SECAM is 50Hz interlaced
Cinema is 24Hz non-interlaced
CS Spring 2009
Source: wikipedia
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

28. SMPTE Time Codes
Society of Motion Picture and Television Engineers defines time codes for video
HH:MM:SS:FF
01:12:59:16 represents number of pictures corresponding to 1hour, 12 minutes, 59 seconds, 16 frames
If we consider 30 fps, then 59 seconds represent 59*30 frames, 12 minutes represent 12*60*30 frames and 1 hour represents 1*60*60*30 frames.
For NTSC, SMPTE uses a 30 drop frame code
increment as if using 30 fps, when really NTSC has only 29.97fps
defines rules to remove the difference error
SMPTE timecode is a set of cooperating standards to label individual frames of video or film with a timecode defined by the Society of Motion Picture and Television Engineers in the SMPTE 12M specification.
Timecodes are added to film, video or audio material, and have also been adapted to synchronize music. They provide a time reference for editing, synchronisation and identification. Timecode is a form of media metadata. The invention of timecode made modern videotape editing possible, and led eventually to the creation of non-linear editing systems.
SMPTE (pron :sim-tee) timecodes contains binary coded decimal hour:minute:second:frame identification and 32 bits for use by users. There are also drop-frame and colour framing flags and three extra 'binary group flag' bits used for defining the use of the user bits. The formats of other forms SMPTE timecodes are derived from that of the longitudinal timecode.
Time code can have any of a number of frame rates: common ones are
24 frame/s (film)
25 frame/s (PAL colour television)
29.97 (30*1.000/1.001) frame/s (NTSC color television)
30 frame/s (American black-and-white television) (virtually obsolete)
In general, SMPTE timecode frame rate information is implicit, known from the rate of arrival of the timecode from the medium, or other metadata encoded in the medium. The interpretation of several bits, including the "colour framing" and "drop frame" bits, depends on the underlying data rate. In particular, the drop frame bit is only valid for a nominal frame rate of 30 frame/s: see below for details.
More complex timecodes such as Vertical interval timecode can also include extra information in a variety of encodings.
SMPTE time code is a digital signal whose ones and zeroes assign a number to every frame of video, representing hours, minutes, seconds, frames, and some additional user/specified information such as tape number. For instance, the time code number 01:12:59:16 represents a picture 1 hour, 12 minutes, 59 seconds, and 16 frames into the tape.
CS Spring 2009
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

29. Take Home Exercise
Given a SMPTE time stamp, convert it back to the original frame number
e.g., 00:01:00:10
CS Spring 2009
ORCHID Research Group
Department of Computer Science, University of Illinois at Urbana-Champaign

30. Summary Digitization of Video Signals Digital Television (DTV)
Composite Coding
Component Coding
Digital Television (DTV)
DVB (Digital Video Broadcast)
Satellite connections, CATV networks – best suited for DTV
DVB-S – for satellites (also DVB-S2)
DVB-C – for CATV
CS Spring 2009