1. Digital Video Digital video is basically a sequence of digital images
Processing of digital video has much in common with digital image processing
First we review the basic principles of analog television

2. Television Fundamentals
Color television cameras and television receivers use the RGB (red, green, blue) color system to create any color
We have seen how raster scan devices operate
Commercial television systems, however, use interlaced scanning as opposed to the progressive scanning of computer monitors

3. Television Fundamentals
In interlaced scanning, one half of the horizontal scan lines (every other line) are transmitted and “drawn” by the receiver
Then the other half of the lines are transmitted and are drawn in between the first scan lines
Each half is known as a field, and two fields together are known as a frame

4. Television Fundamentals
Since the phosphors retain their values for longer than the time that it takes to transmit two fields, and since rate of transmission of a field is shorter than the human eye can perceive, the viewer does not perceive this interlacing
If the frame rate is at least frames per second the viewer does not perceive motion in an image sequence as discrete, but as continuous

5. Interlaced Scanning
In the figures, the first field is transmitted at time t = 0 and displayed at time t = f / 2
f is the frame rate
The second field is transmitted at time t = f / 2 and displayed at time t = f
Note that the display at time t = f consists of information (scan lines) from two distinct points in time

6. The first field of a television transmission

7. The second field of a television transmission

8. A complete frame

10. Interlacing
Interlaced scanning is used in commercial television systems to decrease the bandwidth of the transmitted signal and to reduce the phenomenon known as large area flicker
These problems had been overcome by the time bit-mapped computer monitors were being developed

11. Deinterlacing
There are several common operations that you might want to perform on interlaced video
Producing stills
resizing the video
changing the frame etc.
Performing these operations on raw, interlaced, video can produce undesirable artifacts

12. Interlacing Artifact

13. Deinterlacing Deinterlacing provides a way around these problems
All deinterlacing methods involve turning the field-based image into a frame-based image by modifying one of the fields in the image
Popular methods include duplication and interpolation

14. Deinterlacing
Duplication
Interpolation

15. Television Systems
The exact frame rate depends on the system as does the number of scan lines per frame
There are currently three conventional commercial television systems in use throughout the world
North America, South America and Japan use NTSC
The United Kingdom, Western Europe, Africa and Australia use PAL
France, Eastern Europe and Russia use the SECAM

16. Television Systems System Scan lines/ frame Frame rate Pixels / frame
Bandwidth
NTSC
525
30 / sec
130,000
4.3 MhZ
PAL
625
25 / sec
210,000
6 MhZ
SECAM

17. Digital Television
Considering the bandwidth of the NTSC signal, how would digital transmission compare to today’s analog? We have:
30 frames/second x 130,000 pixels/frame x 24 bits/pixel = 93.6 Mbits/second
To be competitive with analog transmission, a data compression of more than 20:1 is required
All digital television standards therefore include some form of compression.
The disadvantage of digital television therefore, is the extra bandwidth required

18. Digital Television
The real advantage may be seen by examining the signal-to-noise ratio of digital vs. analog television
This figure shows the approximate ratio of error rate to signal-to-noise ratio for digital transmission

19. Digital Television
An error rate of 10-8 or one bit in 100 million bits is practically undetectable
Channel error rates of 10-5 still permit acceptable pictures, especially if error correction techniques are used
An analog TV signal requires a channel with a signal-to-error ratio (SER) of 55dB

20. Digital Television
If we use PCM for a digital television signal, the principal source of error is due to quantization
The error is a maximum of + or - 1/2 the least significant bit
For a quantization level of 8 bits, this is + or - 0.2%
This “fine” quantization would appear as white noise if viewed as a picture

21. Digital Television
Theoretically, the SER with 8 bits is 59 dB and for each 1 bit reduction in quantization, the SER is reduced 6 dB
The actual SER of a composite color TV signal is about 4 dB less
Thus, 8-bit PCM encoding of a noise-free NTSC composite color signal yields a SER of 55 dB

22. Digital Television
A bit error rate of 10-8 is practically undetectable
From the figure above, this requires a SER of only 21 dB
If we use the rate 10-5 with error correction bits added, a SER of 18 dB may be sufficient
This requires less than 1 bit/pixel
The essential problem in digital TV coding is therefore to reduce the picture bandwidth at the expense of the bit error rate and retain acceptable picture quality

23. Aspect Ratios
Each of the systems listed above has an aspect ratio (ratio of width to height) of 4:3
Cinematic films and high-definition television (HDTV) systems have aspect ratios of approximately 16:9

24. Aspect Ratios
4:3 aspect ratio
16:9 aspect ratio

25. Compatible Color TV
In order to permit compatibility of color TV transmission with preexisting black and white receivers, the RGB image generated by a television camera is converted to a YIQ image by using the transform

26. Compatible Color TV

27. Compatible Color TV See book notes for more info on YIQ
The bandwidth allocated to a black and white television signal is illustrated on the next slide

28. Compatible Color TV

29. Compatible Color TV
In order to maintain compatibility, the color TV signal has to fit in the same bandwidth
This is accomplished by first combining the I and Q signals using a method called quadrature modulation
The two signals are multiplied by a sine and cosine function, respectively, added and become a single composite signal
The second idea is to choose the color subcarrier to be an odd multiple of one half the line frequency
The resulting bandwidth allocation is illustrated on the next slide

30. Compatible Color TV

31. Compatible Color TV
At the receiver, the inverse transformation given on the next slide reforms R, G, B from the received Y’, I’, Q’ signals

32. Compatible Color TV

33. Pixel Aspect Ratio
When we are displaying a digital video stream on a standard television receiver, we have another parameter to consider - the pixel aspect ratio
This is related to the aspect ratio of the television screen and to the sampling rate

34. Pixel Aspect Ratio
If we have a screen with an aspect ratio of 4:3 and we have a digital image of size 711x487, then in order to maintain the 4:3 aspect ratio we must have a pixel aspect ratio p, where p can be found as follows.
3/4 = 487/711 * p
p = 0.75 * 711/487 =
Computer monitors generally have pixel ratios of 1.0 (square pixels)

35. Aspect Ratio Conversion
We are given a video sequence with an aspect ratio of 16:9 and we want to display the sequence on a device with an aspect ratio of 4:3
For example, the source image may be 640x360 and the display device may have a resolution of 480x360
We have several alternatives

36. Aspect Ratio Conversion
In the letterbox technique, we scale the source image by the same amount in both the vertical and horizontal directions: 480/640=.75
First, we create a 480x270 image:
S2[i,j] = S1[INT(4/3*i),INT(4/3*j)]
where S1 is the original image and INT is a function which rounds a floating point value to the nearest whole number

37. Aspect Ratio Conversion
We now use this image to form the target 480x360 image S3 as follows:
S3[i,j] = S2[i,j-45] for 45≤j≤314
S3[i,j] = 0 for 0≤j≤44, 315≤j≤359
The second line gives the black bars of the letterbox format
In horizontal compression, we map the source image so that it exactly fills screen of the target device
This results in a stretching of the source image in the vertical direction
The transformation is:
S2[i,j] = S1[INT(3/4*i),j]

38. Aspect Ratio Conversion
The crop and pan transformation is different from the previous two in that the transformation varies over time, depending on the contents of the source image
We will basically be showing one of the three transformed images:
S2[i,j] = S1[i+80,j]
S3[i,j] = S1[i,j]
S4[i,j] = S1[i+160,j]

39. Aspect Ratio Conversion
S2 is the center 3/4 of the source image, S3 is the left 3/4 of the source image and S4 is the right 3/4 of the source image
If there is an object of interest in the leftmost 1/4 of the source image, we use the transformation S3
if there is an object of interest in the rightmost 1/4 of the source image, we use S4
otherwise, we use S2
We view the original video sequence to determine when we need to focus on the corners of the source image, and we define the transformation accordingly

40. Aspect Ratio Conversion
The use of only these three transformations will lead to jerkiness when we shift from one transform to another
We can get the effect of a camera panning by making a time-dependent transformation
For example, imagine we want to pan from the center view to the right view over some period
We can define the transformation as follows:
S5[i,j,t] = S1[i+80+INT(80*(t-t1)/(t2-t1)),j,t], t1≤t≤t2
assuming that there is no parallel change of frame rate

41. Sample rate conversion
At times, it will be necessary to convert the sampling rate in a source signal to some other sampling rate
Consider converting from a CCIR 601 signal (a digital video standard) to an MPEG SIF signal
MPEG is a compression standard for video
SIF is the source input format for the compression)
CCIR 601 is an interlaced signal with a 50 Hz field rate
The signal consists of three components: Y, U and V

42. Sample rate conversion
The Y component (luminance) is sampled at a resolution of 720x480
The U and V components (chrominance) are sampled at a resolution of 360x480
The sampling pattern is shown on the next slide

43. CCIR 601 Sampling Pattern

44. Sample rate conversion
MPEG SIF samples luminance at a resolution of 360x240 and chrominance at a resolution of 180x120
The sampling pattern is different as well, as shown on the next slide

45. MPEG SIF Sampling Pattern

46. Sample rate conversion
The conversion to the lower sampling resolution begins with the discarding of one of the interlaced fields
This reduces the picture rate to 25 Hz (non-interlaced) and reduces the vertical resolution by one half

47. Sample rate conversion
Now, the luminance is decimated by one half in the horizontal direction
One possibility is to subsample the values, however better results are obtained by applying an FIR filter before subsampling
One filter which has been found to give good results in decimating the luminance is shown on the following slide

48. Luminance subsampling filter weights

49. Sample rate conversion
The results of multiplying the filter weights by the original values is divided by 256
Use of a power of two allows a simple hardware implmentation
An example of the use of this filter followed by subsampling is shown on the next slide
At the ends of lines, some special technique such as renormalizing the filter or replicating the last pixel must be used
In the example below, the data in the line is reflected at each end

50. Example of filtering and subsampling of a line of pixels

51. Sample rate conversion
The chrominance samples have to be placed at a horizontal position in the middle of the luminance samples
A linear filter with a phase shift of half a sample is used for this task e.g.

52. Chrominance subsampling filter weights

53. Sample rate conversion
The result is divided by eight
The chrominance values are vertically decimated after they have been horizontally decimated
The conversion process is pictured on the next slide

54. CCIR to SIF Conversion

55. CCIR to SIF Conversion
After this conversion, further compression takes place and the MPEG video is transmitted
At the receiver, the signal must be uncompressed and then reconverted
After zeroes have been inserted between samples, a linear phase FIR filter is applied for upsampling
For reconversion, both luminance and chrominance components are upsampled both horizontally and vertically

56. Upsampling filter for luminance

57. Upsampling filter for chrominance

58. Temporal resampling
We may also find it necessary to convert from one picture rate to another
Consider film digitized at 24 frames per second which is to be shown on television at 60 fields per second
The required conversion can be accomplished by the technique of 3:2 pulldown as shown

59. 3:2 Pulldown

60. Temporal resampling
Video coded at 25 pictures per second can be converted to 50 fields per second by displaying the original decoded lines in the odd fields and the interpolated lines in the even fields
Video coded at 24 pictures per second may be converted to 50 fields per second by speeding it up by about 4% and decoding it as if it had been encoded at 25 pictures per second

61. Temporal resampling
The decoded pictures can be displayed in the odd fields, and interpolated pictures in the even field
The audio must be maintained in synchronization, either by increasing the pitch, or by speeding it up without a pitch change

62. High Definition Television
High-Definition Television (HDTV) is high-resolution digital television (DTV) combined with Dolby Digital surround sound (AC-3)
HDTV is the highest DTV resolution in the new set of standards

63. High Definition Television
History of HDTV in the United States
The Federal Communications Commission (FCC) formed the Advanced Television Systems Committee (ATSC) to examine proposals for a standard for HDTV in the United States
In March 1990, the FCC announced that it would consider only simulcast systems for a standard - systems in which one broadcast channel carries an NTSC signal and a second channel carries an HDTV signal

64. High Definition Television
There were five proposals
Four were from American groups
All of these employed digital broadcasting
The fifth proposal, from the Japan Broadcasting Corp. (NHK) was an analog system called MUSE (Multiple Sub-Nyquist Encoding)
MUSE was used on a test basis in Japan
The simulcast signals must make use of the currently unoccupied and unavailable taboo channels
These channels are those which would cause interference if they were used for broadcasting NTSC signals

65. High Definition Television
With UHF channels, for example, the minimum distance between co-channel transmitters (those with the same channel allocation) varies between 250 and 355 kilometers ( miles)
For adjacent channels, the minimum specified distance is 90 km (55 miles)

66. High Definition Television
The digital proposals all advocate using the taboo channels due to the fact that digital signals can be transmitted with far less power than can analog signals
The required average transmitter power for the digital portion of an HDTV signal can be less than 10 percent of that required by an NTSC transmitter with the same service area

67. High Definition Television
All of the proposed digital systems used some form of video bandwidth compression after analog-to-digital conversion
All systems also used some form of error correction coding so that the digital signal input to the source decoder portion of HDTV decoders will be accurate

68. High Definition Television
HDTV signals require compression because they are much larger than NTSC signals
The signals have twice as much luminance definition, both vertically and horizontally (four times as many luminance pixels)
Further additional pixels are needed for the wider screen (16:9 aspect ratio instead of 4:3)
The increase in luminance detail requires about five times the video bandwidth of conventional television systems

69. High Definition Television
When the additional chrominance detail is factored in, the total bandwidth required is 6 to 8 times that of conventional systems
Eventually, the 4 competitors joined into a “Grand Alliance” which combined the best of the proposals into a single proposal
This proposal was accepted in 1996

70. High Definition Television
Of the 18 DTV formats, six are HDTV formats, five of which use progressive scanning and one interlaced scanning
Of the remaining formats, eight are SDTV (four wide-screen formats with 16:9 aspect ratios, and four conventional formats with 4:3 aspect ratios), and the remaining four are video graphics array ( VGA ) formats
Stations are free to choose which formats to broadcast

71. High Definition Television
The formats used in HDTV are:
720p x720 pixels progressive
1080i x1080 pixels interlaced
1080p x1080 pixels progressive

72. Digital Television
DTV uses MPEG-2 compression to fit in the 6 MhZ bandwidth used by analog television
MPEG-2 is also used in
DVD videos
Some satellite TV broadcast systems
The FCC has mandated that all stations be capable of broadcasting HDTV by 2006

73. DTV Transition Timeline
November Affiliates from the top 30 markets (which reach 50 percent of U.S. households) must have constructed digital facilities.
December Stations in some of the major markets, including New York, Los Angeles, Atlanta and Chicago, were broadcasting digital signals
May The remaining markets after the top 30 (there are 211 total) will have constructed digital facilities.
April All stations must simulcast at least 50 percent of their NTSC programs on their digital TV channel.

74. DTV Transition Timeline
April Stations must simulcast 75 percent of their NTSC programs on their digital TV channel.
April Stations must simulcast 100 percent of their NTSC programs on their digital TV channel.
The Federal Communications Commission has targeted this year for the complete conversion from analog to digital broadcasting.