1. The Computer Continuum9-1 Chapter 9: Multimedia Most of the outside information entering our brains, enters through our eyes or our ears. They are, indeed, input devices through which our world communicates with us.

2. The Computer Continuum9-2 Multimedia n In this chapter: What is multimedia? How are pictures stored within the computer? What can be done with digitized pictures? How can pictures be created using a computer and software? What is the importance of vocal communication? How is speech stored and played back by the computer? What are some problems the computer has when analyzing speech? What areas are computers used in audio communication?

3. The Computer Continuum9-3 What is Multimedia? n Multimedia has become interactive Interactive multimedia: The use of media such as text, graphics, animation, video, and audio in an interactive way that allows a participant to control it. To understand the value of interactive multimedia, we will examine visual and audio concepts in detail.

4. The Computer Continuum9-4 Visual Media: Manipulating Images n Previously created images (Modifying or processing images that already exist) Three types of techniques used to manipulate images: –Minor Processing Techniques –Enhancement –Restoration

5. The Computer Continuum9-5 Visual Media: Manipulating Images n Common example of digitized images: Satellite weather maps. GEOS-7 satellite picture taken on Aug. 24, 1992. Outlines created by a computer program. Program color coded the picture to give information about the hurricane’s wind speed.

6. The Computer Continuum9-6 Visual Media: Manipulating Images n Landsat 4 Satellite: Uses 3 of the 7 frequencies available from the Thematic Mapper of the satellite. Like having a camera with different filters, each letting a different frequency through (including infrared).

7. The Computer Continuum9-7 Visual Media: Manipulating Images n Image enhancement A type of digital image processing whose goal is to highlight or enhance particular aspects of an image or change an image’s structure. Common example of image enhancement: False- coloring: An image enhancement technique that consists of changing the colors of an image or assigning colors to various parts of an image.

8. The Computer Continuum9-8 Visual Media: Manipulating Images n Image restoration A type of digital image processing whose goal is to eliminate known, but unwanted image flaws or degradations.

9. The Computer Continuum9-9 Visual Media: Manipulating Images n Image enhancement example: It makes you wonder if photographs can be trusted.

10. The Computer Continuum9-10 Visual Media: Creating Images n Artists follow rules (or intentionally don’t follow rules) to create original images: Hiding parts of things that wouldn’t normally be visible. Sources of light that illuminate the objects. n Creating images using the computer Line images. Solid forms to 3D. Animation and video.

11. The Computer Continuum9-11 Visual Media: Creating Images n Creating Line Images What are some uses of line drawings? –Map making. –Architectural drawings. –Business graphs. –Plans for building machinery, aircraft, furniture-in-a-box. –City population density maps. How can line images be created? –Using drawing programs such as Adobe Illustrator. Have drawing related features including geometric shapes, free-hand tools, fills, patterns, text.

12. The Computer Continuum9-12 Visual Media: Creating Images n Bit-mapped or Raster graphical images Constructed of individual pixels. Only crude manipulation can be done (modification of each pixel). Retains “bit-mapped” appearance. n Object-oriented or Vector graphic images Stored in the computer as lines, curves, or geometric shapes. Formulas are used to draw (circles, twice the size, are still circles!) Objects can be moved or modified easily.

13. The Computer Continuum9-13 Visual Media: Creating Images n Designing 3-dimensional pictures drawn in 2- dimensions Curved surfaces Color Texture Shading

14. The Computer Continuum9-14 Visual Media: Creating Images n One of the classic problems in computer graphics: Hidden-line problem: Concerns itself with how to hide the outlines or surfaces of a solid object that shouldn’t be seen from the direction of the observation. Example: Studies of an airline pilot in action.

15. The Computer Continuum9-15 Visual Media: Creating Images n Perspective: The quality that allows 3-dimensional images to be drawn on 2-dimensional surfaces and yet retain the look of a 3- dimensional image.

16. The Computer Continuum9-16 Visual Media: Creating Images n Shading: A technique used to give the appearance of illumination by some combination of light sources. The architecture program used to draw this building has a 3D description of the building It can produce pictures from any angle.

17. The Computer Continuum9-17 Visual Media: Creating Images n Shading flat surfaces of the head (as in b) then blending the surfaces to give a more natural look (as in c). abc

18. The Computer Continuum9-18 Visual Media: Creating Images n Texture: A property of a surface. It is observed and identified through the reflection of the light off the surface.

19. The Computer Continuum9-19 Visual Media: Animation and Video n Animation: Early animation started with pictures flipping hand-cranked players. Animation 30 years ago: –Cartoons had to have the figures drawn by artists. –The animator would make them move while taking pictures, frame by frame. Animation now: –Computers are used to create full length feature cartoons. Toy Story Individual frames stored on the computer. Special player programs can be used to view them on the computer screen or recorded on videotape using special hardware and software.

20. The Computer Continuum9-20 Visual Media: Animation and Video n Jurassic Park dinosaurs Wire frames were created in the computer to depict dinosaurs. Each wire frame is moved by the computer which calculates positions of each part viewed from a certain angle and distance. The computer covered the surfaces of the wire frames with a texture layer. –Computer takes into account sources of light illuminating the figure.

21. The Computer Continuum9-21 Visual Media: Animation and Video n Morphing: A graphics technique used in animation A beginning image will distort and change in a predetermined number of frames into a final image.

22. The Computer Continuum9-22 Visual Media: Animation and Video n Digitizing and Manipulating Video Digital video hardware systems: Systems capable of capturing and manipulating video. Analog video uses standards developed over 50 years ago. –Each differs in the number of horizontal lines that make up a single video frame and in the number of frames per second that are displayed on the TV set. European broadcast standards (PAL). United States broadcast standards (NTSC). »30 frames per second. »VHS.

23. The Computer Continuum9-23 Visual Media: Animation and Video n Video capture: The process of digitizing an analog TV signal. Done using hardware that produces a digital form of the analog TV signal. –Uses the VCR to feed the video signal into a digital hardware card that plugs into the computer. –Each frame is converted into a bitmapped image. –The digital video signal is compressed. Eliminates data that our eye doesn’t see or that is hardly noticed. Codec: A scheme for coding & decoding large amounts of data. (The data is economical and uncomplicated to store.)

24. The Computer Continuum9-24 Visual Media: Animation and Video n Common codecs: Motion Picture Experts Group (MPEG) –MPEG-1 Developed to handle slower media such as videoCDs. Deteriorates when high-speed action occurs. Roughly the quality of VHS. 70 minutes with sound fits on a CD-ROM disc. –MPEG-2 Full-screen codec standard that brings twice the resolution of VHS. Four times the resolution of MPEG-1. Optimized for higher demands of broadcast, satellite, and DVD.

25. The Computer Continuum9-25 Visual Media: Animation and Video n Common codecs: (continued) Motion Picture Experts Group (MPEG) –M-JPEG Requires more memory space than MPEG-2 Each frame is digitized separately using JPEG compression. »Makes it easier to edit. As we move from analog to digital broadcast, coding and decoding will become unnecessary. (Everything will already be digital!)

26. The Computer Continuum9-26 Visual Media: Animation and Video n DVD: The Digital Versatile Disc DVD: An optical disk storage technology that looks like the standard CD, but is faster and has a much larger storage capacity. –Two forms: DVD-Video Discs: Intended as a replacement of VHS. DVD-ROM: Intended as a computer storage medium. –Digital Video Players: Software products are available that will play back video coded with one of these codec systems. »QuickTime, ActiveMovie

27. The Computer Continuum9-27 Visual Media: Animation and Video n Streaming media: Information in one of many different visual or audio forms that is sent from a server in packets to the requesting computer. Contains parts of the medium that may be a recording or an actual live event. The information is played back in the order sent, starting soon after a few packets arrive, but before all of them have been sent.

28. The Computer Continuum9-28 Audio Media: Human Speech n The importance of vocal communication: It is the fastest method of communication. Most communication is vocal. Researchers believe that human intelligence is closely linked to the development of vocal communication and language.

29. The Computer Continuum9-29 Audio Media: Human Speech n Creating Speech: Recorded Speech Recording and digitizing individual words and phrases. –Common uses: Talking cash registers. Automated national telephone information service. –Problems with recorded speech: Creating whole sentences of human speech would take a huge amount of computer memory. Several versions of each word would need to be digitized to account for pitch and volume levels (for punctuation and variations of the same word).

30. The Computer Continuum9-30 Audio Media: Human Speech n Creating Speech: Speech Synthesis Speech synthesis: The electronic production of sounds and sound patterns that closely resemble human speech. –Uses phonemes: The fundamental sounds of any given human language. –Three additional factors that have an affect on how a word or phrase sounds: Inflection - The rising and falling pattern of pitch. Duration - The time spent on or the time between individual phonemes. Elision - Splicing phonemes together so that when one ends, the other begins. The connection will sound natural.

31. The Computer Continuum9-31 Audio Media: Human Speech n Creating Speech: Speech Synthesis The sounds produced are electronically generated. –The sounds simulate those produced by the human vocal tract. Can produce both major types of human sounds: »Voiced sounds: Sounds produced by the vibration of vocal cords in conjunction with specific positioning of teeth, tongue, and lips. Include all vowels and some consonants.(d, g, b) »Voiceless sounds: Characterized by the lack of vibration of the vocal cords. (s, k, t)

32. The Computer Continuum9-32 Audio Media: Human Speech n Recognizing Spoken Words More difficult for the computer than speech synthesis. Voiceprint: A visual plot of frequency versus time of sound produced by a human speaker. –No two voiceprints are completely identical. Human voice is determined at least by: »Pitch: The number of cycles per second of a particular sound’s vibration. »Resonance: The reverberation or amplification of the voice in the cavities of the vocal tract. Problem: Because each individual has uniquely shaped vocal cords and resonating chambers, each has a uniquely individual voice.

33. The Computer Continuum9-33 Audio Media: Human Speech n Recognizing Speech Disjointed speech: Words spoken one at a time with silence between words. –Software programs are available for home use. –Each person goes through a recording session to train it. –Total vocabulary to fewer than 1000 words spoken by an individual. Problems: –Speaking in a disjointed manner is unnatural. –Any changes in the person’s voice (head cold, sinus infection) may make the speech unrecognizable.

34. The Computer Continuum9-34 Audio Media: Human Speech n Recognizing Speech Continuous speech: Words spoken in a continuous stream of sounds, usually with no pauses between words. –More difficult for the computer to understand. Computer has difficulty when one word ends and next begins.

35. The Computer Continuum9-35 Audio Media: Music n The Computer and Recorded Music Compact Disc Technology –CDs don’t wear out like records and magnetic tape. After numbers are recorded, they don’t change. Scratching the surface can affect quality of sound. –Compact discs can be produced from a master disc very inexpensively.

36. The Computer Continuum9-36 Audio Media: Music n The Computer and Recorded Music DAT (Digital Audio Tape) technology –Advantages of DAT: Binary information is recorded onto magnetic tape. DAT can easily record sound information. Perfect duplication of CDs can be made onto tape.

37. The Computer Continuum9-37 Audio Media: Music n Recorded Audio Files and Formats Individual standards have been developed for special purposes or for specific computers. –Sound files can be huge. Sound quality affects size. –The four most common sound file formats: WAV: Originally developed with Windows 3.1 AU: Developed by SunAudio for UNIX systems but now supported by Windows and Macintosh. AIFF: Originated on the Macintosh computer. MP3: (MPEG - level 3) Can be played on Windows, Macintosh, and UNIX systems.

38. The Computer Continuum9-38 Audio Media: Music n Hardware and Software Needs Playing back Web sound –Need a sound card. Newer “multimedia” computers have sound cards included. –Need external speakers. Receiving sound files from the Web. –Must be a type of file your browser recognizes. Most new browsers handle Web sound.

39. The Computer Continuum9-39 Audio Media: Music n MIDI: A Revolution in Music MIDI (Musical Instrument Digital Interface): Musical standard created in 1982 that makes possible communication of audio information –Between keyboards, drum pads, etc. –To a computer-controlled mixer. –Each instrument connected to a MIDI bus usually has a small special-purpose computer with some memory. MIDI bus: The main communications channel for MIDI information that can consist of signals sent to as many as 16 different instruments or groups of instruments.

40. The Computer Continuum9-40 Audio Media: Music n All MIDI instruments have three-pronged connectors that are used to connect other MIDI instruments or computers. Thru port: Sends MIDI signals obtained from the IN port on to other MIDI instruments. OUT port: Sends out MIDI signals. IN port: Receives MIDI signals. The three connectors necessary for a MIDI interface:

41. The Computer Continuum9-41 Audio Media: Music n Several ways to use a MIDI instrument: By itself. –A MIDI synthesizer can connect to an amplifier with speakers. Using one MIDI instrument as a master to a second, acting as a slave. –Can not only play its own notes, but can also control the slave’s notes. In conjunction with a MIDI sequencer. –MIDI sequencer: A device, usually a computer, that records and plays back MIDI signals. –Can be either a computer with sequencing software or a dedicated sequencer.

42. The Computer Continuum9-42 Internet Music and Audio Net n Three techniques for both downloading sound to your computer and listening to live audio over the WWW. 1.Streaming audio: A request to hear music is made by clicking on a button. The Website sends back the sound which you hear immediately. You don’t receive a copy of the sound. (You may need to add a plug-in to hear the sound.) –Plug-in: A piece of software that is added to your browser to give it desired capability. 2.Sound file is sent over the WWW. It allows you to store the sound file on your computer. 3.MIDI files: Created by a performer on a MIDI instrument. Needs a MIDI player (available on the Internet.)