2. Week 4 LBSC 690 Information Technology Multimedia

3. Agenda Questions XML review Images Audio Streaming SMILe

4. What’s a Document? Content Structure Appearance Behavior

5. History of Structured Documents Early standards were “typesetting languages” –NROFF, TeX, LaTeX, SGML HTML was developed for the Web –Too specialized for other uses Specialized standards met other needs –Change tracking in Word, annotating manuscripts, … XML seeks to unify these threads –One standard format for printing, viewing, processing

6. Goals of XML Metalanguage –A toolkit for design markup languages Unambiguous markup –Clear span of tags Separate markup from presentation –Style info => stylesheet, so easy to change Be simple

7. A Family of Standards Definition: DTD –Names known types of entities with “labels” –Defines part-whole and is-a relationships Markup: XML –“Tags” regions of text with labels Markup: XLink –Defines “hypertext” (and other) link relationships Presentation: XSL –Specifies how each type of entity should be “rendered”

8. Some XML Applications Text Encoding Initiative –For adding annotation to historical manuscripts –http://www.tei-c.org/http://www.tei-c.org/ Encoded Archival Description –To enhance automated processing of finding aids –http://www.loc.gov/ead/http://www.loc.gov/ead/ Metadata Encoding and Transmission Standard –Bundles descriptive and administrative metadata –http://www.loc.gov/standards/mets/http://www.loc.gov/standards/mets/

9. The Gullibility of Human Senses Three simple tricks for producing –Images –Video –Audio But… how do you move the bits around fast enough?

10. Seurat, Georges, A Sunday Afternoon on the Island of La Grande Jatte

13. Visual Perception Closely spaced dots appear solid –But irregularities in diagonal lines can stand out Any color can be produced from just three –Red, Blue and Green: “additive” primary colors High frame rates produce apparent motion –Smooth motion requires about 24 frames/sec Visual acuity varies markedly across features –Discontinuities easily seen, absolutes less crucial

14. Basic Image Coding Raster of picture elements (pixels) –Each pixel has a “color” Binary - black/white (1 bit) Grayscale (8 bits) Color (3 colors, 8 bits each) –Red, green, blue Screen –A 1024x768 image requires 2.4 MB So a picture is worth 400,000 words!

15. Monitor Characteristics Technology (CRT, Flat panel) Size (15, 17, 19, 21 inch) –Measured diagonally –For CRT, key figure is “viewable area” Resolution –640x480, 800x600, 1024x768, 1280x1024 pixels Layout (three dot, lines) Dot pitch (0.26, 0.28) Refresh rate (60, 72, 80 Hz)

16. Some Questions How many images can a 64 MB flash card store? –But mine holds 120. How? How long will it take to send an image at 64kb/s? –But my Web page loads faster than that. How? But in reality images don’t have these problems –How do we get around these problems?

17. Compression Goal: reduce redundancy –Send the same information using fewer bits Originally developed for fax transmission –Send high quality documents in short calls Two basic strategies: –Lossless: can reconstruct exactly –Lossy: can’t reconstruct, but looks the same

18. Palette Selection Opportunity: –No picture uses all 16 million colors –Human eye does not see small differences Approach: –Select a palette of 256 colors –Indicate which palette entry to use for each pixel –Look up each color in the palette … …

19. Run-Length Encoding Opportunity: –Large regions of a single color are common Approach: –Record # of consecutive pixels for each color An example of lossless encoding

20. GIF Palette selection, then lossless compression Opportunity: –Common colors are sent more often Approach: –Use fewer bits to represent common colors 1Blue75%75x1= 7575x2=150 01White20%20x2= 4020x2= 40 001Red 5% 5x3= 15 5x2= 10 130 200

21. JPEG Opportunity: –Eye sees sharp lines better than subtle shading Approach: –Retain detail only for the most important parts –Accomplished with Discrete Cosine Transform Allows user-selectable fidelity Results: –Typical compression 20:1

22. Variable Compression in JPEG 37 kB (20%)4 kB (95%)

23. Discussion Point: JPEG vs. GIF in Web images Which format should I use for images in my Web pages? Color photos Scanned black & white text Line drawings

24. Hands-On Exercise: Convert Between Formats Download and save two images –http://www.umiacs.umd.edu/~daqingd/image1.jpghttp://www.umiacs.umd.edu/~daqingd/image1.jpg –http://www.umiacs.umd.edu/~daqingd/image2.gifhttp://www.umiacs.umd.edu/~daqingd/image2.gif Use Microsoft Paint to convert each to the other format, and compare quality and the size Why the difference?

25. Discussion Point: When is Lossless Compression Important? For images? For text? For sound? For video?

26. Basic Video Coding Display a sequence of images –Fast enough for smooth motion and no flicker NTSC Video –60 “interlaced” half-frames/sec, 512x486 HDTV –30 “progressive” full-frames/sec, 1280x720

27. Video Compression Opportunity: –One frame looks very much like the next Approach: –Record only the pixels that change Standards: –MPEG-1: Web video (file download) –MPEG-2: HDTV and DVD –MPEG-4: Web video (streaming)

28. Basic Audio Coding Sample at twice the highest frequency –One or two bytes per sample Speech (0-4 kHz) requires 8 kB/s –Standard telephone channel (1-byte samples) Music (0-22kHz) requires 88 kB/s –Standard for CD-quality audio (2-byte samples)

29. Speech Compression Opportunity: –Human voices vary in predictable ways Approach: –Predict what’s next, then send only any corrections Standards: –Real audio can code speech in 6.5 kb/sec Demo at http://www.data-compression.com/speech.html http://www.data-compression.com/speech.html –Scroll down to near the bottom

30. How do MP3s work? Opportunity: –The human ear cannot hear all frequencies at once, all the time Approach: –Don’t represent things that the human ear cannot hear

31. Human Hearing Response Experiment: Put a person in a quiet room. Raise level of 1kHz tone until just barely audible. Vary the frequency and plot the results.

32. Frequency Masking Experiment: Play 1kHz tone (masking tone) at fixed level (60db). Play test tone at a different level and raise level until just distinguishable. Vary the frequency of the test tone and plot the threshold when it becomes audible.

33. Temporal Masking If we hear a loud sound, then it stops, it takes a while until we can hear a soft tone at about the same frequency.

34. Putting it all together… Psychoacoustic compression: –Eliminate sounds below threshold of hearing –Eliminate sounds that are frequency masked –Eliminate sounds that are temporally masked –Eliminate stereo information for low frequencies

35. Transmission Download –Transfer the whole file, then start replay –Can be very slow for large files Streaming –Play the file as it is received Also suitable for live broadcasts –Requires a sufficiently fast connection

36. The “Last Mile” Traditional modems –“56” kb/sec modems really move ~3 kB/sec Digital Subscriber Lines –384 kb/sec downloads (~38 kB/sec) –128 kb/sec uploads (~12 kB/sec) Cable modems –10 Mb/sec downloads (~1 MB/sec) –256 kb/sec uploads (~25kB/sec)

37. Streaming Audio and Video Begins replay after only a portion received Buffer provides time to recover lost packets Interrupts replay when “rebuffering” Media Sever Internet Buffer

38. Hands On: RealPlayer View streaming real video –http://www.c-span.org Select “Tools/Playback statistics” Pay attention to bandwidth and lost packets

39. Narrated PowerPoint Create your slides Slide Show -> Record Narration –Set microphone level Record the narration –Slide transitions are automatically captured Narration plays automatically when displayed

40. Adding Video to PowerPoint Insert->Movies and Sounds –Movies from file (a.mpg file) Decide whether you want autostart –If not, it starts when you click on it

41. Illustrating RealAudio Create a.ram file –URL for the RealAudio –Dimensions of the picture –URL for the picture http://www.umiacs.umd.edu/~oard/teaching/690/fall05/notes/4/media.html

42. Synchronizing Multiple Media Scripting Languages –Synchronized Multimedia Integration Language (SMIL) Custom applications –Macromedia Flash Content representation standards –MPEG 4

43. SMILe W3C standard –Player-specific extensions are common XML, with a structure similar to HTML …

44. Elements in SMIL Window controls (in ) –Controlling layout:, Timeline controls (in ) –Sequence control:,, –Timing control:,, Content types (in ) –,,,

45. SMIL Examples Implemented in RealOne Player Example: http://www.umiacs.umd.edu/~oard/teaching/690/fall05/notes/4/media.html –First, run the executable –Then, view.smil file

46. Before You Go! On a sheet of paper (no names), answer the following question: What was the muddiest point in today’s class?