1. Hardware that Enables Multimedia
Input and Output Devices
Virtual Reality Devices
Modems and Network Interfaces

2. Input and Output Devices
Monitors
Speakers and MIDI interfaces
VR helmets and immersive displays
Keyboards and OCR devices
Digital cameras, scanners, & CD-ROMs
MIDI keyboards and microphones
Video cameras and frame grabbers
Mice, track balls, joysticks, and VR gloves and wands
Input

3. Monitors
A simplified cathode ray tube (CRT)

4. Monitors Pixel — a picture element; a dot of color on the screen
Three different phosphors at each pixel to create the color
RGB (Red, green, blue)
CYM (Cyan, yellow, magenta)

5. Monitors
Raster Scanning

6. Monitors
Refresh rate — the frequency at which the phosphors are excited
Normally the refresh rate is given in Hertz
For flicker-free images 75 Hz or faster is desirable
The refresh rate for a projector needs to be coordinated with the monitor

7. Monitors
The digitized image to be displayed must be stored in a buffer
The stored image is said to be “bit-mapped,” because, for monochrome images, the map used just one bit per pixel
Multimedia monitors use 24 bits per pixel (8 for each color); can define >16 million colors

8. A Good Multimedia Monitor
Large enough for comfortable viewing, probably 15” or greater
Pixel size of no more than 0.28mm
Refresh rate of at least 75 Hz
Capable of displaying 24-bit color
Designed for the CPU and operating system
Ergonomically comfortable and attractive

9. Speakers and MIDI Interfaces
Storage of digitized sound files
Reproduction via digital-to-analog conversion sent to a loudspeaker
Built-in speakers often do not have sufficient fidelity
Low-powered (3- to 5-watt) external speakers or head-phones will serve a single user and provide excellent fidelity

10. Speakers and MIDI Interfaces
Storage of synthesizer command files
Create the sounds by sending the commands to a synthesizer
Musical Instrument Digital Interface (MIDI) standard (1982)
MIDI includes both a hardware and a message standard

11. Speakers and MIDI Interfaces
MIDI hardware standard defines cables, connectors, circuits, and electrical signals
MIDI message standard defines a
Device number for multiple device systems
Control segment that gives an instrument-specific command
Data segment containing the information needed by that instrument for that command

12. Alphanumeric Keyboards
For entering commands, text, and data
Each key is a switch that closes when it is depressed, sending a code to the CPU
The arrangement of the keys may vary
The most common is QWERTY
Another arrangement is Dvorak

13. Choosing a Keyboard
Does it include all of the needed characters, including command keys?
Is it ergonomically comfortable and safe, preventing repetitive stress syndrome?

14. Optical Character Recognition (OCR)
Hardware — scans the text image
Software — systematically checks the entire image for patterns of light and dark that it recognizes as alphabetic, numeric, or punc- tuation characters
OCR software entails pattern recognition, a sophisticated logic problem

15. Optical Character Recognition (OCR)
E e E E e e E E e E
e e E E E e e E e e
It is relatively easy for a human to
recognize each of these characters
as the letter “e.” For the pattern
recognition logic in OCR software,
this is very difficult.

16. Digital Cameras and Scanners
Real Image — a portion of what is physically present in nature
Digital Image — a representation of a real image in which individual points are encoded to represent the wavelength and intensity of light at that point
Still Image — a single snapshot of an instant; may be real or digital
Motion Image — a sequence of images that, when viewed consecutively at the appro- priate rate, gives the impression of con- tinuous motion; may be digital or analog

17. Scanners
Schematic Drawing of a Scanner

18. Digital Cameras
Schematic Drawing of a Digital Camera

19. Digital Cameras and Scanners
Image quality depends on the:
Quality of the optics and the scanning mechanism, which determines focus
Precision of the photosensitive cells, which determines the accuracy of the encoding of intensity and wavelength data
Resolution of the instrument in dots per inch, which determines graininess
Amount of storage available, which deter- mines the total size of an image that can be digitized

20. Inputting Images Memory required to store a 5” x 7” snapshot
Dots/inch resolution
of snapshot image
100
300
600
1200
Bytes required
for storage
1.05 Mb
9.45 Mb
37.8 Mb
151.2 Mb
Assuming no compression, 24 bits per pixel
Memory required to store a 5” x 7” snapshot

21. Video Cameras and Frame Grabbers
Video cameras are similar to digital cameras
Except that a video camera takes image after image continuously
The output from many video cameras is analog and requires digitizing circuitry to make the image usable in a computer
Digital camcorders are now available
Frame grabber software allows the capture of a single still image from the video stream
Frame grabbed images are of rather low resolution, however, <80-90 dots/inch

22. Microphones and MIDI Keyboards
For input of sound
Microphones capture sound waves from the air as an analog signal
The analog signal must be digitized to be stored and then replayed by the computer
Digitizing at <10,000 Hz is adequate for speech; 20,000 Hz is needed for music
MIDI keyboards usually look like piano key- boards with extra switches and controls
MIDI keyboards encode and transmit musical information according to the MIDI standard

23. Inputting Positional Information
Mice
Trackballs
Track pads
Joysticks
Drawing tablets

24. Inputting Positional Information
Specifying a point on a surface requires two dimensions, as with latitude and longitude
A third dimension could be added, as with altitude
For multimedia, what is commonly needed is position on the monitor in terms of left-right (X) and up- down (Y) distances

25. Inputting Positional Information
X and Y coordinates are obtained relative to a fixed point, usually one corner of the screen
The coordinates are entered in analog form as output from roll- ing wheels inside a device such as a mouse
The analog values are digitized to specify the X and Y coordinates

26. The Mechanism of a Mouse

27. Using a Drawing Tablet

28. Collage by Janet Anderson
Created in part using a drawing tablet © Janet Anderson

29. CD-ROMs, DVDs, and Video Disks
Media for external storage and transport of data
Compact disk—read-only memory
Rewritable compact disk (CD-RW)
DVD
Video disk (laser disk); analog format

30. CD-ROMs Digital format Write once, read many times
A rewritable version (CD-RW) is available, but not in common use
Information is “written” by burning tiny holes in the disk surface with a laser
The hole pattern is read by a laser and inter- preted as the bits comprising the data
Can store megabytes of data; about 300,000 pages of double-spaced text, or more than an hour of high fidelity sound

31. Creating Multimedia CD-ROMs
Requires a hard disk large enough to store ~650 megabytes of data to be written to the CD
Requires a CD-ROM recorder that writes the data to the blank CD using a laser
The developer creates the multimedia mater- ial, stores it on the hard disk, and then tests it as completely as possible
When the material is in final form, it is written to the blank CD as if it were being copied from one disk to another

32. Video Disks or Laser Disks
Much larger than a CD-ROM; ~12” in diameter
Hold ~54,000 video frames per side
Hold ~30 minutes of video per side
Read-only
Analog format
Requires a conversion board to be used with a computer
Excellent for large-scale, video-based multimedia projects

33. Virtual Reality Devices
Non interactive
Slow image update rate
Simple image
Nonengaging content
and presentation
No sound
Basic Screen display
Low resolution image
Monoscopic image
Small field of view
No head tracking
No body motion sensing
No tactile feedback
Highly interactive
Fast image update rate
Highly complex image
Highly engaging content
and presentation
Three-dimensional sound
Head-mounted display
High resolution image
Stereoscopic image
Full field of view
Full head tracking
Full body motion sensing
Full tactile feedback
Factors affecting the degree of immersion in virtual reality

34. NCSA’s CAVE

35. NCSA’s CAVE
Virtual Reality Room with stereo glasses and magnetic head/hand tracking
Fully immersive using three of four walls to display the graphics
Uses an SGI Power Onyx with Reality Engine 2 software

36. Miniature version of NCSA’s CAVE
NCSA’s ImmersaDesk
Miniature version of NCSA’s CAVE

37. Drafting-table format virtual prototyping device
NCSA’s ImmersaDesk
Drafting-table format virtual prototyping device

38. Uses CAVE’s stereo glasses and magnetic head/hand tracking
NCSA’s ImmersaDesk
Uses CAVE’s stereo glasses and magnetic head/hand tracking

39. Semi-immersive, fills the user’s field of vision
NCSA’s ImmersaDesk
Semi-immersive, fills the user’s field of vision

40. NCSA’s ImmersaDesk
Uses an identical SGI Power Onyx with the same Reality Engine 2 software as the CAVE

41. VR Head-Mounted Display

42. VR Head-Mounted Display
Limitations:
Liquid Crystal Displays (LCDs)
pixels not as small as a CRT
pixels not as bright as a CRT
cannot change as quickly as a CRT
short focal distance makes precision, high resolution, and rapid response even more essential
Muscle receptor feedback confusion
light rays indicate “distant”
muscles indicate “very close”

43. VR Head-Mounted Display
Parallax — the apparent change in position of a stationary object when viewed from slightly different positions
A person’s eyes each see a slightly different view of an object
As the brain receives these two images, it interprets the the distance to the object in terms of the difference in position of the object in the two images
Parallax can be used to fool the brain into “seeing” images as being at various distances

44. Demonstrating Parallax
Pencil
Demonstrating Parallax
Looking at a pencil aligned What is seen using
with the corner of a room both eyes
What is seen with What is seen with
right eye covered left eye covered

45. Stereoscope
Courtesy of Special Collections, M. I. King Library, University of Kentucky

46. Parallax Problems with VR Head-Mounted Displays
Images may not be perfectly realistic, especially with motion images
When the observer’s head moves and the eyes are refocused, muscle receptor feedback data does not correlate with visual cues
The perspective is always that of the camera, never the viewer’s eyes
A viewer motion feedback mechanism is needed to change the perspective
This all contributes to “cybersickness”

47. VR Aural Output
Refer to the discussion in chapter 2 regarding the perception of sound
Two key factors
Localization
Identification
The brain interprets differences in the signals it receives from the two ears in a manner analogous to binocular vision
For multimedia sound to be completely realistic, it requires head-position sensing feedback and enormous computational power — not practical for most multimedia

48. VR Input Devices The terminology of three-dimensional motion y x z Yaw
Roll
Origin z
Pitch
The terminology of three-dimensional motion

49. VR Position Sensing
A point in space is defined in terms of distance along three mutually perpen- dicular axes, usually termed X, Y, and Z
Motion is defined in terms of changes in position, which requires six parameters
Devices that can sense and record motion are termed six-degrees-of-freedom (6-DOF) devices

50. VR Position Sensing Sensor output from a 6-DOF device can be
continuous
polled, or sent only upon request
Parameters to consider in evaluating a tracking device:
Lag or Latency — the delay between the actual time of the motion and when it is available as input data; should be <50 mSec
Update rate — Rate at which measurements are made; should be as fast as possible
Precision or accuracy of the measurements
Range over which the sensors operate
Rejection of interference

51. VR Voice Input Speech Recognition Complications due to variations in
Pitch
Timbre
Volume
Speed of Delivery
Inflection
Accent
Natural language processing

52. Natural Language Processing
Put out the light.
Turn off the light.
Close the light, please.
The light, turn it off.
Please, shut the light.
Kill the lights.

53. Natural Language Processing
Only the son praised his sister. (The rest of the family did not.)
The only son praised his sister. (There was just one son.)
The son only praised his sister. (He never found fault with her.)
The son praised only his sister. (But never anyone else.)
The son praised his only sister. (He had just one sister.)
The son praised his sister only. (In this instance, he praised no one but her.)

54. Modems and Network Interfaces
Serial and Parallel
Serial — the bits arrive sequentially
Parallel — the bits arrive simultaneously
Character encoding
ASCII — American standard code for information interchange
EBCDIC — Extended binary-coded decimal interchange code
Unicode

55. ASCII and Unicode
ASCII is limited because it is only a or 8-bit code; even using “escape sequences” only a small number of characters can be encoded
Unicode is a 16-bit code that can encode many primary scripts plus special character sets known as secondary scripts

56. Unicode Scripts Primary scripts: Arabic Georgian Hebrew Malayalam
Armenian Greek Hiragana Oriya
Bengali Gujarati Kannada Phonetic
Bopomofo Gurmkhi Katakana Tamil
Cyrillic Han Latin Telugu
Devanagari Hangul Lao Thai
Secondary scripts:
Numbers General diacritics
General symbols Miscellaneous symbols
General Punctuation Technical symbols
Dingbats Mathematical symbols
Presentation forms
Arrows, blocks, box drawing forms, and geometric shapes

57. Modems and Network Interfaces
Start bits
Opposite bit from the system idle state
Necessary to alert the receiver to the beginning of a new character
Stop bits
Provide a short delay at the end of each character to give the receiver enough time to convert from serial to parallel
Error-checking codes
Parity bits, CRC bits, etc.
Discussed in chapter 6

58. Modems and Network Interfaces
Transmission rate
Internal transfer rates are much faster than data rates over networks
The interface needs to “interrupt” the computer when it has new data, not keep it from doing other processing while data is being received
Transmission form
Connection via a telephone line requires a modem (MOdulator-DEModulator) to translate the internal data transfer format into an audio signal, and vice-versa

59. Modems and Network Interfaces
Use of modems and telephone lines
for connectivity