Audio Scene Analysis and Music Cognitive Elements of Music Listening Kevin D. Donohue Databeam Professor Electrical and Computer Engineering University of Kentucky

What is Music? 1 a : the science or art of ordering tones or sounds in succession, in combination, and in temporal relationships to produce a composition having unity and continuity b : vocal, instrumental, or mechanical sounds having rhythm, melody, or harmony Merrian-Webster Online Dictionary:

Auditory Scene: Input  Sensory organs (ears) separate acoustic energy into frequency bands and convert band energy into neural firings  The auditory cortex receives the neural responses and abstracts an auditory scene. Time Frequency

Auditory Scene: Perception  Perception derives a useful representation of reality from sensory input.  Auditory Stream refers to a perceptual unit associated with a single happening (A.S. Bregman, 1990). Acoustic to Neural Conversion Organize into Auditory Streams Representation of Reality

Auditory Stream Experiment Bergman & Campbell (1971)  Streams tend to form by grouping notes close in time and frequency (similarity and proximity).

Circularity in Pitch Judgement  Shepard’s Scale (1964) (Auditory Demonstrations CD, from the Acoustical Society of America)

Perceptual Organization Organization properties:  Belongingness – a sensory element belongs to an organization (or stream) of which is a part.  Exclusive allocation – a sensory element cannot belong to more than one organization at a time.  Bregman & Rudnicky (1975)

Perceptual Organization Organization properties:  Closure – perceived continuity, a tendency to close strong perceptual forms, response to missing evidence.

Sequential and Spectral Integration  Sequential Integration  Grouping sensory elements over time or events at different times considered to be from the same source.  Melody, rhythm  Spectral Integration  Fusing simultaneous sensory elements over frequency into one  Timbre, harmony

Timbre and Spectral Integration  The time envelope and harmonic structure give rise the timbre of the sound.

Timbre and Spectral Integration  Simultaneous tones grouped by timbre Same Note (A)2 Notes (F and A)

Auditory Scene Organization  Primitive Stream Segregation  Inherent constraints in auditory scene analysis (perceptual organization demonstrated by infants/children)  Music: Organization of musical sensory units  Schema-based segregation  Learned constraints in auditory scene analysis (differences in perceptual organization resulting from training and culture)  Music: Differences between musicians and non-musicians  Music: Differences resulting from acculturation (A.S. Bregman, Auditory Scene Analysis, MIT Press 1990, pp. 1-45)

Music Related Terms  Pitch – Perceived frequency/fundamental tone (20Hz- 20kHz Range)  Melody – Pattern of tones identified by the intervals between consecutive pitches  Contour – Shape of the melody without regard to intervals  Loudness – Perceived intensity of sound (0dB to 120dB)  Timbre – Nature of a sound defined mostly by its harmonic structure and time envelope  Rhythm – Repeated pattern of strong and weak sounds  Tempo – Rate of the rhythm

Melody Invariance  A melody can typically be recognized over changes in pitch, loudness, timbre, tempo, spatial location, and reverberations.  Contours are typically recalled better than actual melodies (intervals) for unfamiliar tunes. (Massaro, Kallman, and Kelly 1980). (Daniel J. Levitin, Memory for Musical Attributes, in Music Cognition and Computerized Sound, ed. P.R. Cook, MIT Press, 1999, pp )

Primitive Musical Perception  Distinguish between cognitive components present at an early age and those resulting from acculturation.  Infant: Grasp of musical structures  Adult: Develop cognitive strategies for applying musical structures (W. Jay Dowling, The Development of Music Perception and Cognition, The Psychology of Music Academic Press, 1999, pp )

Summary  Innate organization for separating sounds from different sources. Grouping by pitch, contour, rhythm (phrasing), and timbre are exhibited by infants.  Acculturation refines melody distinctions and its relationship to harmonies and rhythms based on cultural scales and patterns.  Melodic memory is enhanced for melodies following note of a known scale.  Auditory scene analysis operations apply broadly to all sounds (speech, noise, music). Why some auditory streams become pleasurable/stimulating/interesting (music), and others are simply used to form a perception of reality is still not clear.

How many streams are there?

Interesting Websites  Mind, Music, and Machine  Auditory Scene Analysis  Joe Wolfe’s Web Page