1. Colour and Sound Introduced with Too Little on Computing and the Handicapped Dan Eaves CSE5900 – Introduction to Multimedia Computing, Semester 1, 2006 Lecture 2 Note: Non-anime images are from Ted’s Photographics: http://www.ted.photographer.org.uk Very highly recommended! Please spend some time at this wonderful site.

2. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced2 Colour onto Paper Magazine printing uses 4 colours: CYMK –Cyan (a shade of light blue) –Yellow –Magenta (a shade of purplish red) –Black Scanning or photographing through filters, four different printing plates are made. The process is subtractive: Magenta is white light less the cyan and yellow. The paper is run through 4 presses, one doing each colour Inks consist of pigment and a solvent Each adjacent colour blends in the eye/brain: what you see is not what’s there!

3. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced3 Subtractive Colour

4. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced4 Subtractive Colour Red, Green and Blue No Filter

5. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced5 Subtractive Colour Red, Green and Blue Cyan Filter

6. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced6 Subtractive Colour Red, Green and Blue Magenta Filter

7. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced7 Subtractive Colour Red, Green and Blue Yellow Filter

8. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced8 But There’s More Colour (or black) aren’t continuous but in an array of dots, a screen (as in screen door). (There are types of colour printing, e.g., lithography, where this isn’t the case.) Dots (of colour or black) in a position on a screen can be of different sizes. This means that any combination of CMYK colours are possible in a particular position. Screens and varying dot sizes are best seen looking at a low resolution black and white photograph. In a black area, the dot is big, in a light grey area the dot is small.

9. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced9 Black and White “Screen”

10. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced10 Colour “Screen” The 4 Printing Plates: Cyan, Yellow, Magenta & Black: CMYK

11. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced11 Colour “Screen” Final Appearance, with Closeup

12. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced12 The Horrors of a Colourful Life Ink varies from batch to batch. Ink changes in characteristics as it gets older. Manufacturers’ formulas for the “same” colour vary. Papers can be coated or uncoated. There are different kinds of coatings. Different kinds of paper absorb at different rates. The same paper absorbs ink at different rates depending on the amount of water in it. There are several proprietary ways of specifying exactly what colour you want (for spot colour printing). Pantone is the best known. None work perfectly.

13. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced13 Colour onto Screen Computers use RGB colour –Red –Green –Blue The process is additive. There are three electron guns at the back of the TV tube. There are three types of phosphors (rare earths) which coat the inside of the front of the monitor. There is a grid (mask) or set of vertical wires (aperture grill) through which the electrons go on their way from the gun to the face of the monitor. (Aperture grill is better.) Each position on the screen has the three phosphors. The colour of the position is their combined light.

14. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced14 Colour onto Screen: Additive Colour

15. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced15 Additive Colour: RGB to CMYK

16. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced16 Monitor Phosphors versus Eye Sensitivity (The lines indicate the 3 different cones)

17. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced17 Monitor (With Shadow Mask)

18. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced18 Monitor (With Aperture Grill)

19. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced19 Monitors: Aperture Grill versus Shadow Mask

20. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced20 LCD Displays For liquid crystal displays –A freak of nature. Such crystals are Liquid Present a twisted path (blocks light) But electricity untwists it, so light can get through! –Voila

21. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced21 From Digital Home Magazine: How LCDs work

22. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced22 LCD Problems: Traditional and Contemporary Limited viewing angle – but greatly improved Colour intensity not too good – but greatly improved Limited response time precluded using for games – but greatly improved … for a price Colour accuracy not good enough for artists, graphic applications, but fine otherwise Only one optimal resolution, e.g., 1280 x 1024 on my LCD monitors Aspect ratio difference: Monitors = 1024/768 = 1.333 = 4:3 aspect ratio – like your (old) TV 1280/1024 = 1.25 = 5:4 aspect ratio Were more expensive than CRTs, but not now except for games/artist quality models.

23. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced23 Plasma Displays 3 florescent lights (red, green, blue) in each pixel position

24. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced24 Plasma Display Problems 3 florescent lights are big There is no likelihood of making them really small So plasma displays always have to be –big and –“low resolution” But for a home theatre wall display at 16x9 HDTV resolution they’re wonderful. Note that 16x9 aspect ratio Note that HDTV resolution is (in theory) up to 1920 x 1080 pixels at 16x9, but a current 50 inch – Sony – $9,500 plasma display has a 1280 x 768 resolution

25. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced25 Aspect Ratios When TV developed in 1940s, took the then standard motion picture ratio: 4x3 Movie industry responded to perceived threat from TV by inventing a new standard, 16x9 Other countries responded to US TV standard (NTIS) by specifying alternative incompatible standards to protect their electronics industries, e.g., the PAL standard used in Australia Different standards have different number of horizontal lines, different screen refresh rates Ain’t competition wonderful!

26. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced26 Getting Colour onto the Screen We can specify an intensity of each of the three colours in the range of 0 to 255. 255 by 255 by 255 = about 16,700,000. This takes 24 bits, 8 for each colour If we want pure red, we set: –Red = 255 –Green = 0 –Blue = 0 This translates to the intensity of the beams of electrons striking the different phosphors. What do we get when: –Red = 255 –Green = 255 –Blue = 255

27. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced27 But Wait, There’s More... “True colour” can be either 24 or 32 bits deep. Both of these have a palette of 16.7 million colours. If present, the 8 extra bit are called the alpha channel. This contains transparency information: How well can you see through this pixel, on a scale of 0-255 Primarily used as masks in animation where images are overlaid to build the final result. (Note: you can also call 32 bit colour “4 channel colour”)

28. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced28 The Horrors of the Colourful Life, Mark II Different monitors use different rare earths as phosphors. Electron guns wear out. Phosphors wear out. Ambient light plays an important role in colour as seen: –Natural light –Incandescent light –Fluorescent light. What if we are building coloured images on the screen for printing???? Pantone helps here too, but there are so many variables that the colours actually printed may be surprising.

29. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced29 But What if We Don’t Have 24/32 Bits We use either 8 bits (256 colours) or 16 bits (65,000 - 64k colours) We fake the colours in-between … BUT We can’t blend colours like when printing. Instead, we dither. We take a square of pixels (2x2, 3x3, 4x4) and put colours into them which (we hope) blend in the eye to look like the colour we want. Usually they don’t.

30. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced30 Or, We Can “Spend” Our CLUT on Greys – 2 5 8 256

31. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced31 Dithering and Anti-Aliasing (here’s the 16 colour picture the examples come from)

32. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced32 To Dither or Not To Dither We can carefully pick 256 colours from 16.7 million, creating a CLUT (colour look up table). The CLUT is specific to the page, not to a particular image on the page. So selecting the right CLUT is an art, not a science. Or we can dither. Or both. If you aren’t careful, dithering may happen automatically, and look horrible.

33. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced33 Dithering The Actual Pixels The Visual Effect

34. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced34 Anti Aliasing The screen is a very, very, very low resolution device: say 92x92 dots per inch as opposed to 600x600+ dpi for laser printers (10,000 dots versus 360,000: 36 times as many) Curved lines get the jaggedies You can make these look less bad by dithering in some pixels in colours between those of the line and the background where the jagged parts are. But, if this is done automatically, a number of your 256 colours will be used up without your noticing.

35. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced35 Anti Aliasing The Actual Pixels The Visual Effect

36. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced36 Anti Aliasing Used all the time Built into digital cameras Built into products like PhotoShop Built into products like Flash/Shockwave Done by hand with regularity, especially with GIFs –Remember, GIFs limited to 256 colours Even with PhotoShop, hand retouching is done with great frequency for high (perceived) quality images “Foxing” (fooling) the eye is the basis for much of the substance of computer graphics!

37. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced37 Compression – Static Images Because pictures take up so much space, we have to compress them –Reduce the number of colours –Store constant information only once –Reduce the size of images –Use vector graphics Used by Flash/Shockwave Used for animations Used for simple static images There are a number of different static image raster compression formats

38. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced38 Raster versus Vector Graphics Raster = The invisible dot that moves across the (TV, Monitor) screen, –Puts something in each pixel position. –Then does a flyback from the bottom right to top left positions. –Driven by storage of data (2-32 bits) for each pixel position. –Supports highly detailed images, e.g., photographs –File sizes are typically big, sometimes very big. Vector = A line (any line, not necessarily straight, but which can be described by a mathematical formula. –Formulas are converted into bit maps (raster images) –These are displayed on the screen –Supports low detail images, e.g., cartoons and drawings –File sizes are typically small, sometimes very small. Vector graphics trade computer processing for file size.

39. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced39 GIF/PNG GIF – Graphics Interchange Format is: –A lossless compression (so not very compressed) –Maximum of 8 bit colour (256 different colours) –One colour can be invisible (“transparent”) –Very fast to decompress –Universally used for headings, buttons, etc. –Works by recognizing areas of rows of pixels of the same colour and not storing actual pixel by pixel data but start/stop/colour information –That is, where possible works like a scanner and sends addresses of changes rather than all pixels. –Sends all pixels where necessary. –Useless for photographs

40. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced40 JPG/JPEG JPG - Joint Photographers Expert Group –Lossy compression What you get after decompression isn’t what you started with –Tuned to the human vision system: Starts by eliminating colour differences the eye can’t see –Degree of compression under your control You specify the degree of compression, from unnoticeable to horrible –Many colours (up to 32 bits) –No transparent colour –Used for photographs

41. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced41 Raster Graphics Data is stored for each pixel position. With an image that is 500 x 300 there are 150,000 pixel positions. –If it’s in two colours, we store either a 0 or a 1 for each position. So the number of bytes is 150,000 / 8 = 18,750 bytes –If there’s 4 colours, we store 00, 01, 10 or 11 for each pixel position -- 150,000 * 2 / 8 = 37,500 bytes –Note: They don’t have to be standard colours. We can say, using the CLUT, that 0 = puce, 1 = tangerine, 2 = maroon and 4 = black…or whatever colours we want. –With a palette of 256 colours, we need 8 bits for each pixel --- 150,000 * 8 / 8 = 150,000 bytes –With a palette of 16.7 million colours, we need 24 bits for each pixel -- 150,000 * 24 / 8 = 450,000 bytes –(With a 56 kb modem, how fast will this picture load?)

42. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced42 The Tragedy of Raster Graphics Remember, modems are serial (1 bit at a time) so that’s 56 * 1024 /8 bytes = 7,000 bytes per second at best so our 450,000 byte picture will take 65 seconds to load. Or 100 seconds with a 33k modem. Or 6.5 seconds with good broadband. Consider that animation/movies require 24 picture per second. Weep! Bash head against wall! Bewail fate! If a page contains 400 words @ 5 letters per word and 5 characters of extra HTML per word we get: (400 * 5) + (400 * 5) = 4,000 characters = 4,000 bytes =.6 seconds to load. It’s not called H T ML for nothing Drat, drat, drat. We’ll need to compress, use vector graphics, reduce the number of colours, compress again.

43. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced43 The Necessity of Raster Graphics HTML allows you to select size in terms of Heading 1 - Heading 8, not in terms of font, weight or point size. The user’s default font is used in the display. This is useless and hopeless from a design perspective. So headings and buttons have to be raster graphics (usually.GIF files with a limited colour palette). Build heading in Corel Draw or Illustrator (vector) Pass it to Photoshop (raster) Reduce the colour depth (bits per pixel) Dither and anti-alias and adjust both Save as.GIF/PNG

44. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced44 Palettes – Somewhat Historical – Relates to GIF Only We have 256 colours to play with (say), represented by the numbers 0 to 255. That is, we store 8 bits of colour information for each pixel. But the screen can display 64k different colours (16 bit colour) or 16.7 million colours (24 bit colour). So we pick our particular 256 colours out of all the available ones, using a CLUT (colour look up table). This shows our 256 colour numbers and the larger colour number to display for that number –0 = red of 12, green of 45 and blue of 210 –1 = red of 25, green of 90 and blue of 255 –and so on to 255 But we can have only one such CLUT for everything on the HTML page

45. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced45 Vector Graphics Like Corel Draw or Illustrator files –Lines and outline shapes are stored as mathematical formulas –Fills are stored as a few bytes. –A program reconstructs the always necessary raster image from the vectors, stores the results in the video card’s frame buffer, and shows the results on a raster device (your screen). –Image size can be increased or decreased without loss of detail or introduction of artefacts (e.g., moire effects) –So we have very small files but lots of processing work on the way to displaying the results. –Flash/Shockwave are currently the only standard ways of using vector graphics on the web. –But new standards are being developed to extend web use of vector graphics – their chance of success is ????? TrueType and PostScript fonts are vector graphics formats.

46. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced46 A Shift: Digital Sound Music on CD works as follows: –Each chunk of data represents the amplitude of the sound at a moment in time with a number ranging from 0 to 65,535 -- Whoopee, you now instantly say that each sample is a 16 bit number. –44,000 samples are taken per second per stereo channel. –So data storage per second is 44,000 samples * 2 bytes per sample * 2 channels = 176,000 bytes per second. –So a 3 and a half minute song takes 176,000 * 60 * 3.5 = 36,960,000 bytes of storage space –Hummmmm. We’re going to have problems with sound, too. –Your CD music is the equivalent to uncompressed raster graphics –MP3 is the equivalent to.JPG/.JPEG compression, as it is both lossy and the degree of loss (and compression) is under user control. Standard MP3 gets a standard length song down to, say 3- 4 meg. (Estimate using a 90% compression factor)

47. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced47 MP3 Assessed “MP3 -- MP3 was the disappointment I expected it to be. This does not mean it has no application - for the PC, MP3 is ideal. It has a good compression factor, allows easy transfer over the Internet, and sets new standards in audio compression (I said new - not better). MP3 is instantly recognised without an A/B switching test. It introduces some distortion to highs in particular. Also, in complex passages the instruments get lost - you cannot follow each one - it is as if the encoder could not work out what parts to keep and which to throw away. This hypothesis was confirmed when I went back and recorded the same songs at higher MP3 bit rates (128kbps was the one I used initially, as it is the most widely used today). At 256kbps and 320kbps there were some remarkable improvements. I do believe MP3 will stay mainly in the computing field…. For car applications, background music, or played over typical speakers found on most PC systems, MP3 is fine. As a serious audio storage medium to be used with quality equipment however, I cannot recommend it at all. “ Guy Churchill, 19 Mar 1999: http://www.minidisc.org/format_comparison.html http://www.minidisc.org/format_comparison.html

48. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced48 MIDI MIDI is the music equivalent to vector graphics. Music is stored as a set of instructions, a score, giving the note, the instrument to play the note, the way the note is to be played, etc. Many notes (voices) can be played at the same time. Sound is generated either via “FM Synthesis” or by using ROM or RAM based samples of how the instruments sound. Files are quite small but lots of processing is necessary to generate the sound.

49. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced49 Computing for the Disabled A Short Overview

50. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced50 Is This Trip Necessary? The WWW is an “information appliance” –It is available and used everywhere –People are expected to access and use it. –But what if you are spastic or blind? The contemporary ethic is that disabled people must not be denied access to public resources just because they are disabled. The WWW is a public resource So we must ….. –(Saying ‘It’s too hard’ isn’t acceptable!)

51. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced51 Good Sources UK supplier of products: http://www.keytools.com/default.asphttp://www.keytools.com/default.asp Solutions for Humans: http://www.sforh.com/http://www.sforh.com/ New Zealand/Canadian company specializing in hardware and software, esp. for the blind. Was Pulse Ware: http://www.humanware.com http://www.humanware.com Freedom Scientific, maker of JAWS, etc.: http://www.freedomscientific.com/ http://www.freedomscientific.com/ Australian supplier. Check out the Technology for Life section: http://www.quantech.com.au/index.html http://www.quantech.com.au/index.html Haptic interfaces. A good starting place is the Wikipedia entry, with lots of good links at the end: http://en.wikipedia.org/wiki/Haptichttp://en.wikipedia.org/wiki/Haptic

52. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced52 (?) Do we need to deal with all computer systems here, or just the WWW? Legally, it’s all publicly available systems. Practically (and economically), in some circumstances it may be better simply to supply a human intermediary. But doing so may be unacceptable to the disabled!

53. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced53 Interfaces for Spastics (The two bravest people I’ve ever known were badly spastic women who are productive members of society. And you didn’t dare try to help them.) Spastics obviously can’t use mice, so the interface must support non-mouse-driven use. A number of very helpful special keyboards are available.

54. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced54 Big Keyboards

55. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced55 “The Grid” On-Screen Touch Sensitive Screen Overlays Symbol Communication

56. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced56 “The Grid” On-Screen Touch Sensitive Overlays Typing -- Note suggestions on left. The user has entered ‘h’ and ‘o’

57. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced57 Intelikeys A special keyboard with overlays

58. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced58 For the Blind The leading screen reader is Jaws from Freedom Scientific –interactive talking installation –Supports Office and other standard products –Supports Internet Explorer including: links lists, frames lists, forms mode, reading HTML tables and graphic labels, etc. –Provides scripting language to extend to additional programs –Can output to refreshable Braille displays –$US895 to 1,295 per copy (ouch) + 120-200 maintenance per year.

59. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced59 Connect Outloud Also from Freedom Scientific –Cutdown, specialised version of the WWW part of Jaws –interactive talking installation –Supports Internet Explorer including: links lists, frames lists, forms mode, reading HTML tables and graphic labels, etc. –Can output to refreshable Braille displays –$US249 per copy (no ouch)

60. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced60 Refreshable Braille Displays Braille Star from Pulse Data (now Humandata) –40 or 80 Braille characters in line –Can take notes, annotate, store, work off line, etc –$UD6,000 or 11,000 (40 or 80 characters) (ouch, ouch)

61. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced61 One Hand Keyboards

62. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced62 Keyboard for Head/Mouth Pointer

63. Rev. 3 March, 2006CSE5900 Lecture 2: Colour and Sound Introduced63 Head Movement Mouse