1. Chapter 1: Overview

2. What is multimedia? Multimedia is when you have too many cables.
Multimedia combines audio and visual materials to provide computerized interaction of text, sound, graphics, images, animation & video to enhance communication and to enrich its presentation.
Multimedia systems handle at least one type of “continuous media” as well as “static media”.

3. Multimedia applications
Videodisc applications
A laser disc (LD) can store about 1 hour of video (per side), or 100,000 video frames.
A DVD can hold 2-8 hrs of high-quality video
Electronic games
Each game is a highly interactive multimedia application that presents layered-2D/3D animation with synchronized sound effects and music.
New generation of computer games usually use optical disks (CD/DVD/GD-ROM) to store game data.

4. Multimedia applications
Web browsers
A hyper-document is a document composed of multiple media types.
Multimedia presentation systems
An “engine” that displays, synchronizes, provides interaction with, and generally manipulates multimedia material (e.g., Macromedia Director).

5. Multimedia applications
Multimedia mail systems
Handle electronic messages containing audio, graphics, and other media.
3 components: Composition, Transmission, Presentation.
Teleconferencing
A computer equipped with microphone, speakers, and a video camera, and placed on a multimedia network, can establish audio and video connections between other, similarly equipped, machines.
Multi-user tools, such as group editors. A group editor allows conference participants to share documents, and to edit the documents simultaneously.

6. Multimedia applications
Multimedia services
Interactive TV
Interactive shopping
Education
Medical services (telemedicine)
Video-on-demand

7. Analog vs. Digital Two ways to process information: analog and digital
Examples:
LP vs. CD
radio vs. web-radio
slide-rule vs. calculator

8. Analog recording
Sound is caused by a variation of air pressure – a sound wave

9. Analog recording
An audio tape records an analog signal that represents the sound wave.
An analog signal is continuous – there are an infinite number of points (values) and each value has an infinite precision.

10. Digital recording
To record an analog signal by a computer, we need to perform analog-to-digital conversion (ADC):
Sampling
Quantization
Coding

11. ADC
Consider an analog signal
time

12. Sampling
convert continuous signal into discrete values by taking samples

13. Sampling
The original signal can be approximated by interpolation using the sampled values.

14. Sampling
More samples  more accurate approximation

15. Quantization
A computer cannot record the values with infinite precision. A value has to be quantized.
a quantization
level

16. Quantization
A computer cannot record the values with infinite precision. A value has to be quantized.
a coarser
quantization

17. Quantization
Each value is replaced by the nominal value of its quantization level.
a sample
value
a nominal
value

18. Quantization
Each value is replaced by the nominal value of its quantization level.

19. Quantization
The difference between a sample value and its quantization value is called the quantization error.
quantization
error

20. Quantization
More quantization levels gives a more accurate representation.

21. Quantization
More quantization levels gives a more accurate representation.

22. Quantization
More quantization levels gives a more accurate representation.

23. Quantization
Note that quantization does not necessarily have to be uniform or linear.
wider
quantization
level
narrower
quantization
level

24. Coding assign a codeword to each quantization level. 0101 0100 0011
0010
0001
0000
1000
1001
1010
1011
1100
1101

25. Coding
An analog signal could then be represented digitally by a string of 0’s and 1’s
… …

26. Why digital? ADC introduces error so … Example: for CD-DA:
the waveform constructed by interpolating samples is not a verbatim copy of the original
quantization introduces quantization error
so …
Why digital representation?
What should be the sampling frequency?
How many quantization levels (or bits/sample) shall we use?
Example: for CD-DA:
sampling frequency: 44,100 samples per second
16 bits per sample (i.e., 65,536 quantization levels)

27. Why digital?
easy integration & sharing of resources (storage, transmission network).
can be processed by a computer, e.g., encryption, watermarking, compression, digital effects, etc.
more “reliable” storage and transmission (error-correction).
can be used and copied many times without losing quality.
analog signal transmission
channel
channel
digital signal transmission
channel
reconstruct

28. original

29. “blurred”

30. “pixelated”

31. Media types
Non-temporal (Discrete) – do not have a time dimension, and their contents & meanings do not depend on the presentation time.
Text
Image
Graphics

32. Media types
Temporal (Continuous, Isochronous) – have a time dimension. They convey meanings only if “displayed” at a specific rate.
Video
Digital Audio
Music

33. Text not visually exciting conveys essential and precise information
text representation: e.g., ASCII
storage “friendly”
sometimes, certain information is too abstract to be captured by words

34. Digital images
To digitize a photo, we again perform ADC

35. Digital images
sampling

36. Digital images
pick a sample value from each “box” to record

37. Digital images
finer sampling gives a clearer image

38. Digital images
finer sampling yet

39. Digital cameras
A Charge-coupled-device (CCD) camera has a 2-dimensional array of photosites that convert the amount of light intensities into equivalent electrical charges.
Each photosite is covered by a color filter (red, green, or blue).

40. Digital cameras
The amount of charge at each photosite tells the intensity of
a color component of light detected at that point. R G B B R G G B R

41. Digital cameras R G B B R G G B R red
The intensity of the intensity read-outs are of course quantized. R G B B R G G B R
red

42. Digital cameras R G B B R G G B R
green

43. Digital cameras R G B B R G G B R
blue

44. Digital images
two-dimensional arrays of pixels (picture elements) of varying color and intensity.
color model: how to specify the color of a pixel?
RGB: colors can be represented by a numeric triple specifying red (R), green (G), and blue (B) intensities.
YCRCB (for digital images and video)
CMY(K) (for printing)
Cyan (no red)
Magenta (no green)
Yellow (no blue)
K — black ink

45. Digital images Compression Image transformation and processing.
A page-sized 24-bit color image with 300 pixels per inch takes up about 20Mbytes.
Lossless and lossy compression.
Many different standards: JPEG, GIF (Graphic Interchange Format), ... Example: run-length encoding.
Image transformation and processing.
E.g., morphing (one image transforms into another).

46. Graphics
Graphics data are represented by a geometric model + a set of graphics operations.
Geometric model
A collection of 2D/3D geometric primitives (lines, circles, polygons, curves).
Transformations: rotation, translation, scaling.

47. Graphics

48. 3D models
taken from
Planet
Architecture

49. Graphics Graphics operations Coloring, shading, lighting, and viewing.
ambient light (from all directions)
point light (inverse square law)
spot light (a cone-shaped volume)

50. Graphics Animation Texture mapping (applying an image onto a surface)
Rendering
converts a model + shading, lighting + viewing ... into an image.
Animation
Eye-catching
Good for demonstration

51. Graphics
taken from
ACM Computer
Graphics ‘97

52. Graphics
taken from
Planet
Architecture

53. Analog video
A sequence of images called frames, “persistence of vision”.
Originally, motion pictures are shown at an (insufficient) frame rate of 16fps.
It was found that for smooth motion, 24fps is needed.
Attributes: frame rate, resolution, aspect ratio, interlacing, refresh rate

54. Analog video
Formats. e.g., NTSC (National Television Systems Committee) PAL (Phase Alternation Line).
format frame rate scan lines aspect ratio NTSC :3 PAL :3 HDTV(US) :9 HDTV(EURO) :9
MUSE(Japan) :9
picture
width to
height

55. Analog video
Theoretically, most color can be produced by mixing 3 primary colors (red, green, blue). An analog video camera produces 3 distinct continuous signals, one for each color component.
scan line
signal
horizontal
blanking
black
white …
black
vertical blanking
white

56. Analog video
A video signal is applied to a TV (a CRT) to control the power of an electron beam that strikes on the phosphors on the inside of the CRT surface.
With the 625/50 system (PAL), for example, a frame is displayed every 1/25 seconds.
Unfortunately, the phosphors do not stay lit that long  flickering
To prevent flicker, the picture has to be refreshed at least 50 times per second.
Interlacing: a frame is divided into 2 fields: odd-lines and even-lines, a field is displayed every 1/50 seconds.

57. Analog video
frame 1
progressive
frame 2
time
1/50
1/25
interlaced

58. Analog video
Luminance/chrominance principle: the three primary colors can be converted into 2 parts:
Luminance: information on the lightness of the image.
Chrominance: information on the color of the image.
Because the human eye is not very sensitive to color information, the bandwidth of these 2 color components is reduced before transmission.

59. Analog video Video storage video tape e.g., VHS tape
about 240 scan lines resolution
problem: loss of quality when copying and repeated playback due to stretches and magnetic material wearing off.

60. Digital video
A video can also be represented by a sequence of digital images.
broadcast quality video (uncompressed):
1 sec  > 20 MB
For lesser quality, and a good compression technique, it is possible to achieve:
1 sec = 1.2 Mb 
transfer rate of (single-speed) CD-ROM 
VCD
Data rate of a DVD movie is about 2GB/hr.
Compression: lossless and lossy.
B – byte
b – bit

61. Digital video
For lossy compression, we can achieve a 50:1 or higher compression ratio.
MPEG: The Moving Pictures Expert Group
MPEG-1: 1.5Mbps VHS quality video
MPEG-2: 4-10 Mbps (digital TV)
MPEG-4: a system which allows a scene structure to be composed of multiple different objects (video, audio, natural, synthetic)
MPEG-7: multimedia information retrieval

62. Digital video formats CCIR 601(4:2:2 chromatic subsampling) 4:2:0
for video exchange
525/60 system: 720 x 480 (take 720 samples from each active scan line)
625/50 system: 720 x 576
4:2:0
for video broadcast

63. Digital video formats SIF CIF for storage 525/60 system: 360 x 240
4:1:1 chromatic subsampling
CIF
for video conferencing
360 x 288 at 30 fps using 4:1:1 chromatic subsampling

64. Digital audio Digital audio representation Sampling frequency
produced by sampling a continuous signal generated by a sound source
a process called analog-to-digital conversion (ADC)
Sampling frequency
Human ear is sensitive to frequencies of up to about 20kHz (c.f., rat: 1k – 10k Hz; cat: 100 – 60k Hz)
Sampling frequency > 40kHz
audio-CD  44.1kHz, 16 bits per sample ( 96dB max. SNR)
DVD audio  196kHz (max), 24 bits per sample (max)
(Q: what is the throughput requirement?)
1.5Mbps/9.6Mpbs

65. Digital audio Number of channels Storage 2 for stereo
some audio editing equipment may have 16 or 32 channels
Storage
An hour of high quality stereo digital audio requires > 500MBytes of storage.
A CD-ROM can store about 650MBytes of data.
A CD-DA can store about 74 minutes of audio data.

66. Music MIDI -- Musical Instrument Digital Interface
Digital musical instruments send MIDI messages to a sequencer.
The sequencer composes the music according to the messages received.
The sequencer/synthesizer has a “palette” of sounds for each type of instrument
MIDI sequencer

67. Challenges
Multimedia stresses all components of a computer system (data volume & time constraints)
CPU processing power
fast speed for data capturing, CODEC, data enhancement
large amounts of data being processed in real-time
Storage and Memory
high capacity, fast access time, high transfer rates
System architecture
high bus bandwidth, efficient I/O 6

68. Challenges Software Operating systems Networks
tools for retrieval and data management of continuous media data
Operating systems
support for new data types, real-time scheduling, multimedia file systems, time-critical synchronization
Networks
high bandwidth, low latency, low jitter

69. Research areas 1. fast processors 2. high-speed networks
3. large capacity storage devices
4. video & audio compression algorithms
5. graphics systems
6. human-computer interfaces
7. real-time operating systems
8. information storage and retrieval
9. hypertext & hypermedia
10. languages for scripting
11. parallel processing methods

70. Compression Throughput and storage Compression makes it possible.
“If a picture is worth a thousand words, then a video is worth 414 million (4-byte) words per minute,” Benny said. (or 25GBytes/hr !!!!)
What are the requirements on network data transfer rate? On disk I/O?
Compression makes it possible.
50:1 yields 0.5 GB/hr.
200:1 for HDTV.
How expensive is the processing? Hardware solution? Software solution?

71. Compression MM systems require data compression for 3 reasons:
the large storage requirements of MM data.
relatively slow storage devices that cannot play MM data in real time
network bandwidth that does not allow real-time video data transmission

72. Some compression standards

73. Multimedia networking
Many multimedia applications, such as video mail, video conferencing, and video-on-demand, require the support of a high performance network system.
In these applications, the multimedia objects are stored at a server and played back at the clients’ sites.
Remote retrieval of multimedia objects has stringent time constraints.

74. Multimedia networking
Delay: the amount of time it takes to transmit a data unit (e.g., a video frame) from a sender to a receiver.
Jitter: delay variation.
delay
jitter
source
destination

75. Multimedia networking
for some loss: can apply lightweight transmission protocols. These protocols cannot accept retransmission, since that might introduce unacceptable delays.
Traditional networks do not suit multimedia. Ethernet provides only 10 Mbps, its access time is not bounded, and its latency and jitter are unpredictable. Token-ring networks provide 16 Mbps and are deterministic; from this point of view, they can handle multimedia. However, the predictable worst-case latency can be very high.
FDDI
fast ethernet: 100 Mbps; priority token ring.

76. Multimedia information retrieval
To retrieve a text document from the Web, we use keyword search via “Alta Vista”, for example.
To retrieve a record from a relational database, such as Oracle, we use an SQL statement.
How shall we formulate a query to retrieve pictures?
What about audio? How do we describe a sound? How do we describe a song?

77. Multimedia software tools
Music sequencing
Cakewalk
supports general MIDI
provides several editing views (staff, piano roll, event list) and Virtual Piano 28

78. Multimedia software tools
Image and video editing
Adobe Photoshop
allows layering of images, graphics and text
includes many graphics drawing and painting tools
sophisticated lighting effects, various image processing filters
Adobe Premiere
provides many video and audio tracks, superimposition and virtual clips
supports various transitions, filters and motions for video clips
a reasonable desktop video editing tool 29

79. Multimedia software tools
Multimedia authoring
Microsoft Power Point
building slide show sequence
slides include objects of different media, e.g., sound and video
limited animation ability
Macromedia Director, Flash
Movie metaphor (cast of bitmapped sprites, scripts, music, sound, and palettes)
Lingo script language allows more control of the presentation sequence. Control of devices, e.g., VCRs and disk players.
ready for building interactivities using buttons, etc.