1. Light and Sound, Seeing and Listening 2.1 Perception 2.2 Light 2.3 Color and Color Sensation 2.4 Sight and Visual Communication 2.5 Sound 2.6 Listening and Auditory Communication Chapter 2Theoretical Foundations of Multimedia

2. Perception and Cognition Perception — the consciousness or awareness of objects or other data through the medium of the senses. Cognition — high-order mental processes, such as complex representation, inference, and interpretation, that lead to perception and then to understanding. Chapter 2Theoretical Foundations of Multimedia

3. “Regions of Interest” for Studying Perception The EnvironmentThe Environment Incoming stimuliIncoming stimuli Sensory surfaces and peripheral neuronsSensory surfaces and peripheral neurons The BrainThe Brain Effector systemsEffector systems Motor responsesMotor responses Chapter 2Theoretical Foundations of Multimedia

4. Optical Illusions Chapter 2Theoretical Foundations of Multimedia Müller-Lyer Illusion Subjective contours Archimedes spiral

5. Auditory Illusions Bach sonatas and partitasBach sonatas and partitas Ravel’s BoleroRavel’s Bolero Chapter 2Theoretical Foundations of Multimedia

6. The Nature of Light A form of energyA form of energy Particle nature of lightParticle nature of light Waveform nature of lightWaveform nature of light Early Philosophers and Researchers: Plato Sir Isaac Newton Alhazen Thomas Young Christopher Scheiner Max PlanckEarly Philosophers and Researchers: Plato Sir Isaac Newton Alhazen Thomas Young Christopher Scheiner Max Planck Chapter 2Theoretical Foundations of Multimedia

7. The Behavior of Waves Chapter 2Theoretical Foundations of Multimedia A Sine Wave

8. The Behavior of Waves Chapter 2Theoretical Foundations of Multimedia 1 sec. 2 sec. 3 sec. One cycle per second, or 1 hertz (Hz)

9. The Behavior of Waves Chapter 2Theoretical Foundations of Multimedia Two waves of equal frequency and different amplitudes

10. The Behavior of Waves Chapter 2Theoretical Foundations of Multimedia Two waves and the wave resulting from their interaction

11. The Behavior of Waves Chapter 2Theoretical Foundations of Multimedia Two equivalent waves in phase cancellation

12. The Behavior of Waves Chapter 2Theoretical Foundations of Multimedia Two equivalent waves in phase and the resultant wave

13. The Behavior of Waves Chapter 2Theoretical Foundations of Multimedia A fundamental waveform and its second and third harmonics

14. The Elecromagnetic Spectrum Chapter 2Theoretical Foundations of Multimedia

15. The Elecromagnetic Spectrum Chapter 2Theoretical Foundations of Multimedia

16. The Speed of Light Chapter 2Theoretical Foundations of Multimedia 186,282.3959 miles per second Events within sight appear to be instantaneous Speed varies in different media Some materials absorb some of the light’s energy

17. The Measurement of Light Chapter 2Theoretical Foundations of Multimedia Intensity or “strength” of radiation Radiance — the total amount of energy (light and heat) measured in watts Luminance — light strength perceived by a human eye Brightness — subjective measure of how bright an object appears to a human Strength diminishes by inverse square proportion to distance from source

18. Reflection Chapter 2Theoretical Foundations of Multimedia Reflection — light strikes a surface and the waves bounce off May cause problems on a monitor under the wrong lighting conditions Allows fiber optic cables to function by causing the light being sent along the fiber to “bounce” off of the walls of the cable and continue toward the other end

19. Refraction Chapter 2Theoretical Foundations of Multimedia Refraction — light passes through a different medium and has its direction changed; sometimes called the “bending” of light

20. Refraction Chapter 2Theoretical Foundations of Multimedia Prism separating white light

21. Refraction Chapter 2Theoretical Foundations of Multimedia Lenses focus or spread light

22. Color Chapter 2Theoretical Foundations of Multimedia The sensation registered in the brain when different wavelengths of light impinge upon the retina of the eye, causing a message to be sent along the optic nerve to excite neurons in various portions of the brain.

23. Color Chapter 2Theoretical Foundations of Multimedia Reflected color occurs when a light illuminates an object and some of the wavelengths of the light are reflected off of the object, while others are absorbed by it. Transmitted or emitted color results from an atom or a molecule emitting light energy of a characteristic wave- length after being excited.

24. Measuring Color Chapter 2Theoretical Foundations of Multimedia Emission and Remission Curves

25. Measuring Color Chroma — hue, or a color’s pure expression Saturation — the amount of pure color (not black or white) in a chroma Value — subtle difference between shades and tints Lightness — amount of light reflected from an object Brightness — luminance of an object in the context of its surroundings Chapter 2Theoretical Foundations of Multimedia

26. Trichromatic Vision Subtractive principle of color — reflected color Additive principle of color — emitted color Thomas Young — proposed that the eye is composed of three different light- sensitive materials Hermann von Helmholtz — identified the retinal cells for the three colors Young-Helmholtz Theory — three– component or trichromatic vision theory Chapter 2Theoretical Foundations of Multimedia

27. Perception of Color The eye sees different wavelengths and intensities very differentlyThe eye sees different wavelengths and intensities very differently Violet is barely perceivedViolet is barely perceived Red is seen very wellRed is seen very well The cornea is yellow, and absorbs green, blue, and VioletThe cornea is yellow, and absorbs green, blue, and Violet As people age, the cornea becomes even more yellow so they see even less green, blue, and VioletAs people age, the cornea becomes even more yellow so they see even less green, blue, and Violet Chapter 2Theoretical Foundations of Multimedia

28. Psychological Aspects of Color Descriptive phrases — seeing Red; feeling blue; a black mood; a yellow cowardDescriptive phrases — seeing Red; feeling blue; a black mood; a yellow coward Color is highly subjectiveColor is highly subjective Colors are associated with emotionsColors are associated with emotions Warm colors (Red and yellow) appear nearWarm colors (Red and yellow) appear near Cool colors (blue and green) appear more distantCool colors (blue and green) appear more distant Pastels appear relaxed; dark colors seem more angryPastels appear relaxed; dark colors seem more angry Chapter 2Theoretical Foundations of Multimedia

29. Color Wheel Chapter 2Theoretical Foundations of Multimedia blue magenta red yellow green cyan

30. Colors on Computer Monitors Radiated vs. reflected light (recall the additive principle vs. the subtractive principle of light)Radiated vs. reflected light (recall the additive principle vs. the subtractive principle of light) RGB system — Red, green, blueRGB system — Red, green, blue CYM system — Cyan, yellow, magentaCYM system — Cyan, yellow, magenta HSV system — hue, saturation, & valueHSV system — hue, saturation, & value HLS system — hue, lightness, & saturationHLS system — hue, lightness, & saturation CIE system — Commission Internationale de l’EclairageCIE system — Commission Internationale de l’Eclairage Chapter 2Theoretical Foundations of Multimedia

31. RGB Color Cube Chapter 2Theoretical Foundations of Multimedia green G yellow blue red R B black gray line magenta cyan white

32. Sight and Visual Communication Physical process — light being trans- mitted and reflectedPhysical process — light being trans- mitted and reflected Physiological process — light enters the eye, falls on the retina, and is trans- mitted by the optic nervePhysiological process — light enters the eye, falls on the retina, and is trans- mitted by the optic nerve Neurological process — the brain reacts to the impulses from the optic nerveNeurological process — the brain reacts to the impulses from the optic nerve Psychological process — the mind interprets the information in the brainPsychological process — the mind interprets the information in the brain Chapter 2Theoretical Foundations of Multimedia

33. Sight and Visual Communication Chapter 2Theoretical Foundations of Multimedia The Camera Obscura

34. Sight and Visual Communication Chapter 2Theoretical Foundations of Multimedia Comparing a camera and the human eye

35. Aspects of Visual Literacy A prerequisite for the comprehension of visual mediaA prerequisite for the comprehension of visual media General cognitive consequencesGeneral cognitive consequences Awareness of visual manipulationAwareness of visual manipulation Aesthetic appreciationAesthetic appreciation Chapter 2Theoretical Foundations of Multimedia

36. Images and Meaning Sense — the external reality denoted by an imageSense — the external reality denoted by an image Feeling — the attitude conveyed by an image regarding the expressed realityFeeling — the attitude conveyed by an image regarding the expressed reality Tone — the attitude or message that the creator of an image wishes to transmitTone — the attitude or message that the creator of an image wishes to transmit Intention — the effect that an image is desired to produce in a viewerIntention — the effect that an image is desired to produce in a viewer Chapter 2Theoretical Foundations of Multimedia

37. Images and Meaning Sign — any physical object to which a community ascribes meaningSign — any physical object to which a community ascribes meaning Semiotics or Semiology — study of signsSemiotics or Semiology — study of signs Signifier — the physical entity that expresses the signSignifier — the physical entity that expresses the sign Signified — the concept or emotion that is conveyed by the signSignified — the concept or emotion that is conveyed by the sign Signification — the link between the expression and the conceptSignification — the link between the expression and the concept Chapter 2Theoretical Foundations of Multimedia

38. Images and Meaning Chapter 2Theoretical Foundations of Multimedia To refute the assertion that pictures are arbitrary in their connection to what they purport to represent, Messaris uses the real-life situation of the powerful and powerless camera angles

39. Lester’s pragmatic perspectives for analyzing and interpreting images Personal — gut reaction; subjectivePersonal — gut reaction; subjective Historical— importance of the image based on the medium’s time lineHistorical— importance of the image based on the medium’s time line Technical — relationship between light, the recording medium, and the presentationTechnical — relationship between light, the recording medium, and the presentation Ethical — producer’s, subject’s, and viewer’s moral and ethical responsibilities to the workEthical — producer’s, subject’s, and viewer’s moral and ethical responsibilities to the work Cultural — analysis of the work’s symbols that convey meaning in this society at this timeCultural — analysis of the work’s symbols that convey meaning in this society at this time Critical — issues transcending a particular image to shape a reasoned reactionCritical — issues transcending a particular image to shape a reasoned reaction Chapter 2Theoretical Foundations of Multimedia

40. Still Images Chapter 2Theoretical Foundations of Multimedia Concrete Abstract

41. Messaris’ ten major discrepancies between concrete images and reality Cannot reproduce full range of brightnessCannot reproduce full range of brightness Cannot reproduce full range of colorsCannot reproduce full range of colors Lack information about changes in brightnessLack information about changes in brightness Lack information about the color of objectsLack information about the color of objects Cannot reproduce a stereoscopic effectCannot reproduce a stereoscopic effect Cannot reproduce motion parallax effectsCannot reproduce motion parallax effects May not show diminution of size with increasing distance from the viewerMay not show diminution of size with increasing distance from the viewer May not be constrained to a single view at a timeMay not be constrained to a single view at a time May contain major distortions of featuresMay contain major distortions of features May entail omissions of features of the subjectMay entail omissions of features of the subject Chapter 2Theoretical Foundations of Multimedia

42. Messaris’ ten major discrepancies between concrete images and reality Messaris’ discrepancies do not present a serious problem. Human beings, even from a very young age, show a remarkable capacity to recognize and correctly identify even the most stylized or incomplete concrete representations, regardless of the presence or absence of color, depth cues, or a preponderance of easily recognizable objects. The significance of this observation for a multimedia developer is that the expense of creating and displaying extremely high-resolution, realistic graphics generally is not necessary. The lesson is not to waste resources on glitz but to devote them to substance. Chapter 2Theoretical Foundations of Multimedia

43. Components of Form Dot — a small circle that has been filled inDot — a small circle that has been filled in Line — a sequence of dots so close together that there appears to be no space between themLine — a sequence of dots so close together that there appears to be no space between them ShapesShapes ‰ Parallelogram — a four-sided figure with opposite sides parallel and equal in length ‰ Circle — the locus of all points equidistant from a given point ‰ Triangle — a three-sided figure – equilateral – isosceles Chapter 2Theoretical Foundations of Multimedia

44. Arrangements of Dots Chapter 2Theoretical Foundations of Multimedia Center of attention Tensionthroughasymmetry Tensionthroughdivision Fill in the dots

45. Arrangements of Lines Chapter 2Theoretical Foundations of Multimedia Openspace Constrictedspace Verticalhalt Dynamismanddirection

46. Arrangements of Lines (continued) Chapter 2Theoretical Foundations of Multimedia Nervousenergy Strengthandconfidence Grace and airiness Parallellines and text

47. Uses of Shapes — Parallelograms Chapter 2Theoretical Foundations of Multimedia Unbalancedappearance Balanced use of white space

48. Uses of Shapes — Circles Chapter 2Theoretical Foundations of Multimedia This circle dominates the field it is on. This circular device is an immediate attention-grabber

49. Uses of Shapes — Triangles Chapter 2Theoretical Foundations of Multimedia Pyramid — symbol of symbol of strength and stability Isosceles triangle pointing up evokes image of a church steeple Isosceles triangle stronglyindicatesdirection

50. Graphics Primitives Point — a pixel on a computer screenPoint — a pixel on a computer screen Line — two or more pixels connected (adjacent) to each otherLine — two or more pixels connected (adjacent) to each other Polygon — a closed figure of more than two sides existing in one planePolygon — a closed figure of more than two sides existing in one plane The computer domain of straight lines and geometrical figures cannot reproduce perfect renditions of the real worldThe computer domain of straight lines and geometrical figures cannot reproduce perfect renditions of the real world Chapter 2Theoretical Foundations of Multimedia

51. Factors Providing a Sense of Depth Space — the frame in which an image is setSpace — the frame in which an image is set Size — apparent sizes of objects provide cluesSize — apparent sizes of objects provide clues Color — warm colored objects seem closer than cool ones, as do more highly contrasted objectsColor — warm colored objects seem closer than cool ones, as do more highly contrasted objects Lighting — differences in light intensity and shadowing portray depthLighting — differences in light intensity and shadowing portray depth Textural Gradients — e.g., ripple effects become closer together in the distanceTextural Gradients — e.g., ripple effects become closer together in the distance Time — a culture’s perception of time influences the depiction of depthTime — a culture’s perception of time influences the depiction of depth Perspective — a drawing techniquePerspective — a drawing technique Chapter 2Theoretical Foundations of Multimedia

52. Three forms of Perspective Chapter 2Theoretical Foundations of Multimedia Illusionaryperspective Geometricalperspective Conceptualperspective

53. Motion in Images Real movement — animate and inanimate objects in motion in the real worldReal movement — animate and inanimate objects in motion in the real world Apparent movement —still images providing the appearance of motionApparent movement —still images providing the appearance of motion Persistence of vision — the retention of an image in the brain for a period of timePersistence of vision — the retention of an image in the brain for a period of time In MM, the number of images per second and the viewer’s persistence of vision de- termine the appearance of realistic motionIn MM, the number of images per second and the viewer’s persistence of vision de- termine the appearance of realistic motion Chapter 2Theoretical Foundations of Multimedia

54. Motion in Images Albert Michotte — Perception of physical causalityAlbert Michotte — Perception of physical causality ‰ launching effect ‰ entraining effect Graphic movement —the motion of the eyes in scanning an object (often left-to- right, top-to-bottom); culturally basedGraphic movement —the motion of the eyes in scanning an object (often left-to- right, top-to-bottom); culturally based Implied movement — the illusion of motion in a still image, without any movement of the image, its surroundings, or the viewerImplied movement — the illusion of motion in a still image, without any movement of the image, its surroundings, or the viewer Chapter 2Theoretical Foundations of Multimedia

55. Early Writing Chapter 2Theoretical Foundations of Multimedia Pictographs Ideographs

56. Early Writing Chapter 2Theoretical Foundations of Multimedia HieroglyphicsLogograms Early Alphabets Cuneiform

57. Typography Chapter 2Theoretical Foundations of Multimedia Movable Type A page from a Gutenberg Bible Courtesy of special collections, M. I. King Library, Uni- versity of Kentucky

58. Typography Chapter 2Theoretical Foundations of Multimedia Movable Type Same page from a Gutenberg Bible showing colored inks and illuminations Courtesy of special collections, M. I. King Library, Uni- versity of Kentucky

59. Typefaces — Serif Typefaces Chapter 2Theoretical Foundations of Multimedia AGaramondPalatino Bookman Old Style Courier Footlight MT Light New York

60. Typefaces — Sans Serif Typefaces Chapter 2Theoretical Foundations of Multimedia Arial Arial Narrow Century Gothic ChicagoHelveticaImpact

61. Typefaces — Special Typefaces Chapter 2Theoretical Foundations of Multimedia Braggadocio Brush Script MT DesdemonaMistralStencil(symbol)

62. Typefaces — Type Styles Chapter 2Theoretical Foundations of Multimedia Bold Italic Outline UnderlineShadow SMALL CAPS

63. Typefaces — Weight, Size, and x-height Chapter 2Theoretical Foundations of Multimedia Aa Aa Aa Different weights and stresses Aa Aa Aa Aa Font sizes from 12 points to 96 points hd hd hd Varying x-height

64. Typography — Alignment Chapter 2Theoretical Foundations of Multimedia Left Justified Right justifiedJustified Centered

65. Typography — Special Type Techniques Chapter 2Theoretical Foundations of Multimedia

66. Production of Sound Equilibrium— An object at rest; the mid- point of the motion of vibrationEquilibrium— An object at rest; the mid- point of the motion of vibration Independence of magnitude and period — accounts for the independence of volume and pitchIndependence of magnitude and period — accounts for the independence of volume and pitch Speed of a point in vibration varies — as vibration proceeds, a point reaches maxi- mum speed as it passes the equilibrium point, then slows until it reaches it maxi- mum swing, reverses itself and begins to speed up againSpeed of a point in vibration varies — as vibration proceeds, a point reaches maxi- mum speed as it passes the equilibrium point, then slows until it reaches it maxi- mum swing, reverses itself and begins to speed up again Chapter 2Theoretical Foundations of Multimedia

67. Sound Waves Frequency — number of vibrations per second; determines the pitchFrequency — number of vibrations per second; determines the pitch Amplitude—the intensity or volume of a heard sound, or the signal strength of an electronically transmitted soundAmplitude—the intensity or volume of a heard sound, or the signal strength of an electronically transmitted sound Harmonics — waves having frequencies related by arithmetic ratios; account for the different sounds of different instru- ments, even though they are playing the same noteHarmonics — waves having frequencies related by arithmetic ratios; account for the different sounds of different instru- ments, even though they are playing the same note Chapter 2Theoretical Foundations of Multimedia

68. Harmonic Motion Chapter 2Theoretical Foundations of Multimedia At rest Compressing Rarefying Wave motion

69. Fourier Spectrum Analysis Baron Jean Baptiste Joseph FourierBaron Jean Baptiste Joseph Fourier “Any vibration that repeats itself indef- initely can be built up from (or is analyz- able into) a set of pure sinusoid waves” Handel (1993)“Any vibration that repeats itself indef- initely can be built up from (or is analyz- able into) a set of pure sinusoid waves” Handel (1993) “Any periodic waveform, no matter how complex, can be analyzed, or decom- posed, into a set of simple sinusoid waves with calculated frequencies, amplitudes, and phase angles” Handel (1993)“Any periodic waveform, no matter how complex, can be analyzed, or decom- posed, into a set of simple sinusoid waves with calculated frequencies, amplitudes, and phase angles” Handel (1993) Chapter 2Theoretical Foundations of Multimedia

70. Fourier Spectrum Analyses Chapter 2Theoretical Foundations of Multimedia

71. Propagation of Sound Reflection— sound waves bounce off of surfaces and continue to travel in a new directionReflection— sound waves bounce off of surfaces and continue to travel in a new direction Direct sound —sound heard after no reflectionsDirect sound —sound heard after no reflections Early sound — sound heard after only one reflectionEarly sound — sound heard after only one reflection Reverberation — sound heard after several reflectionsReverberation — sound heard after several reflections Chapter 2Theoretical Foundations of Multimedia

72. Propagation of Sound Chapter 2Theoretical Foundations of Multimedia

73. Doppler Effect When either the source or the listener is in motion relative to the other, the wave- length of the soundwaves is affected, causing the pitch to seem to varyWhen either the source or the listener is in motion relative to the other, the wave- length of the soundwaves is affected, causing the pitch to seem to vary The pitch becomes increasingly higher as the source approaches the listenerThe pitch becomes increasingly higher as the source approaches the listener The pitch becomes increasingly lower as the source recedes from the listenerThe pitch becomes increasingly lower as the source recedes from the listener The relative motion must be at least 20 or 30 miles per hourThe relative motion must be at least 20 or 30 miles per hour Chapter 2Theoretical Foundations of Multimedia

74. Listening and Auditory Communication Hearing — merely receiving sound physi- cally, not consciouslyHearing — merely receiving sound physi- cally, not consciously Listening — “is perceiving sound with careful and responsive discrimination. It is thinking about sound — analyzing its quality, style, interpretation, and nuance. It is trying to understand what motivates a sound. It is engaging in new sonic exper- iences regardless of their strangeness. It is examining your reaction to sound in relation to your mood and feeling.” Alten, 1994Listening — “is perceiving sound with careful and responsive discrimination. It is thinking about sound — analyzing its quality, style, interpretation, and nuance. It is trying to understand what motivates a sound. It is engaging in new sonic exper- iences regardless of their strangeness. It is examining your reaction to sound in relation to your mood and feeling.” Alten, 1994 Chapter 2Theoretical Foundations of Multimedia

75. Reception of Sound Habituation — 85% of all primary auditory neurons become less sensitive to sounds if they persist unchanged. Musicians use changes in key and vibrato to counteract this phenomenonHabituation — 85% of all primary auditory neurons become less sensitive to sounds if they persist unchanged. Musicians use changes in key and vibrato to counteract this phenomenon Pitch — Most adults can detect pitches between 20 and 20,000 Hz, but pitches above 16,000 Hz are perceived as hiss.Pitch — Most adults can detect pitches between 20 and 20,000 Hz, but pitches above 16,000 Hz are perceived as hiss. Relationship to age — As age increases, range of hearing decreases.Relationship to age — As age increases, range of hearing decreases. Chapter 2Theoretical Foundations of Multimedia

77. Comparison of Changes in Decibels and Power Chapter 2Theoretical Foundations of Multimedia  dB  Power  dB  Power 01 15 32 11.3 18 64 21.6 20 100 32 30 1,000 42.5 40 10,000 53.2 50 100,000 64 60 1,000,000 75 70 10,000,000 86 80 100,000,000 98 90 1,000,000,000 10 10100 10,000,000,000 10 10100 10,000,000,000 11 12110 100,000,000,000 11 12110 100,000,000,000 12 16120 1,000,000,000,000 12 16120 1,000,000,000,000

78. Perception of Sound Hearing Levels — physical, abstract, etc.Hearing Levels — physical, abstract, etc. Meaning — is derived from relationships among the various levels on which humans hear soundsMeaning — is derived from relationships among the various levels on which humans hear sounds Localization — is the ability to locate a sound source in three-dimensional spaceLocalization — is the ability to locate a sound source in three-dimensional space Identification — is the ability to recognize voices, instruments, and other sources of soundsIdentification — is the ability to recognize voices, instruments, and other sources of sounds Chapter 2Theoretical Foundations of Multimedia

79. Handel’s Approaches to Conceptualizing Perception of Sound Prototype or template — compare what is heard with an idealized versionPrototype or template — compare what is heard with an idealized version Feature or attribute — abstract critical features to distinguish among possibilitiesFeature or attribute — abstract critical features to distinguish among possibilities Higher-order variable — time-varying, complex acoustic properties that uniquely identify an eventHigher-order variable — time-varying, complex acoustic properties that uniquely identify an event Innate systems — distinct brain structures that yield invariant preceptsInnate systems — distinct brain structures that yield invariant precepts Chapter 2Theoretical Foundations of Multimedia