1. It sure is smart but can it swing? (Digital audio and computer music)
COS 116: 2/22/2011
Sanjeev Arora
(slides from Prof. Rebecca Fiebrink) 

2. Concrete example of how different concepts
Today:
Concrete example of how different concepts
studied so far apply in one domain: Music.
Opening video: What are the performers doing? How are they controlling the music?

3. Overview Sound and music in the physical world and in human experience
Representations of music
Analyzing music with computers
Creating music with computers

4. 1. Sound and music

5. What is sound? Discussion Time “Pressure wave”
Vibrating objects -> sound pressure waves -> eardrum/inner ear -> brain
Demo sndpeek: Show waveform of tuning fork, speech
“Pressure wave”

6. What is music? Discussion Time “Organized sound”
Organized sound: Composers and performers manipulate pitch, rhythm, timbre, dynamics, structure in meaningful ways
Psychoacoustics: note, octave
Culture: banned tritone
People of a given culture and time period learn a lot about this organization just by exposure
e.g., even if you aren’t a musician, you expect V7-I (example)
At a very low level, when you expect that this pattern will occur, and your expectation is correct, your brain rewards you; this is one of the very low-level reasons for why we like music.
In fact, our brain responds to music a lot like it does to narcotics. Naloxone -- blocks opiate receptors for heroin
Psychoacoustics play an important role
Also dependence upon history, culture, experience
Engages listeners’ psychological mechanisms for expectation/reward

7. What do we hear? Frequency Pitch Loudness Timbre …
(start with chipmunk clip; pause it, and then let words
about frequency/pitch/loudness appear).

8. Psychoacoustics
Relationships between physical phenomenon of sound and our perception
Frequency <-> pitch
20-20,000Hz
Amplitude <-> loudness
Identities and strengths of frequencies present : timbre
Show sndpeek again with sounds
Mention that they will explore these issues in the lab. + =

9. Spectral representation
Every wave (sounds included) can be decomposed into “pure sine waves”, each of a single frequency.

10. 2. Representations of sound and music

11. Discussion Time How do you represent music? Score: Audio samples
Spectrum
Q: How do we represent music?
Show: score vs. audio file
Ask aloud “is score a faithful representation?”
Q: What is the relationship between the score representation and what you hear?
Explain spectrum -
Show adding waves in time
Adding waves in frequency
Show spectogram - 440, 880, both. (with sound)

12. Digital representation of music
Qs: How can computer halve the frequency of this signal?
Describe sampling, quantization
Mention that this is a “lossless” format, used for WAV, AIFF, CD, etc.
Just like we saw on the last slide, we can use less information to represent a sound and have a representation that is less faithful/complete: **downsampling** (play example of downsampled audio):
Q: What’s a benefit of using a lower sampling rate?

13. Compression Why? Saves bandwidth, memory etc. How?
Trivial but very lossy: sample less often
Cleverer: use psychoacoustic principles
MP3: Masking
Even cleverer: use physical principles of sound production (via models) of sound source)
By being smart about what information we leave out, we can get a better representation with fewer bits
Show sndpeek- just my voice: how spectrum changes w/ vowel
Sound example: phasor source, formant filter, source-filter

14. Choosing a representation
Representations are compromises
Standard representations are somewhat arbitrary (e.g. MIDI)
Best representation is task-dependent
Compromised: Frequency and dynamic range
Composer’s intention (what is notated)
Performance gestures
Aribtrariness: CD length, MP3 parameters (for popular music)

15. 3. Using technology to analyze sound and music

16. Analyzing speech Real-life apps: Customer service phone routing
Voice recognition software
Hard questions (for a computer): Identity, language, content

17. Computational Auditory Scene Analysis
Applications: Archival and retrieval, forensics, AI
e.g., forensic examination of tapes from RFK assasination reveal a second gunman.

18. Music information retrieval
Analyzing musical data
Query, recommend, visualize, transcribe, detect plagiarism, follow along with a score
Sites you can try
midomi.com
Themefinder.com
Pandora.com (human-driven), last.fm
Sound example: Hum melody (just me)
can a computer transcribe this? Figure out my tempo? Find a file on your hard drive that has this melody? Find melodies that are similar? Written by the same composer? Give me feedback on how well I sang it?

19. Machine learning for analysis
Given the types of music that the listerner likes; recommend new music that he or she might like.

20. (1) What does this pseudocode do?
Variable i;
for i=1 to 4 do { if (i < 3) then print( i*i)
else print (i);}
(2) Write 33 in binary.

21. 4. Using technology to create music and sound
A whirlwind tour of the 20th century, with a focus on computer technology

22. Creating music: Synthesis
Examples:
Stevie Wonder
Tina Turner (FM)
Kanye West/ Daft Punk (vocoder)

23. Three approaches to synthesis
Additive synthesis
Figure out which frequencies are present, and in what proportions
Synthesize a sine wave at each frequency, and superpose them.
Physical modeling
Start with knowledge about how physical systems oscillate
Simulate oscillation of an arbitrary system. (Recall Lecture 4) + + …=

24. Three approaches to synthesis
Cross-synthesis (eg vocoder)
Choose filter for speech (vowel)
Choose source to be another sound
Mention that this was generated with code written at Princeton

25. Some continua of computer music creation
low
Synthesis
Running a program
Computer follows performer
Live coding, special controllers
Interactivity
Controlling synthesis
Oboe demo!
Composition
high
Level of control

26. Performer-Computer Interaction
Augmented instruments
Software and hardware interfaces
Demo: PLOrk video, PBS
Demo: using a Wii-mote to control sound
New instruments
Perry’s Mug.
Live coding
Demo: Max’s drum machine
Chuck/miniaudicle
Download chuck and miniaudicle from

27. Questions: How can we…. develop new ways to synthesize sound?
give user control over synthesis parameters?
make machines interactive in a musical way?
augment human capabilities?
design new instruments that are easy to play? allow expert musicality?
create music that is emotionally and aesthetically compelling?

28. Final remarks Distinctions in this presentation are superficial
Analysis, representation, and creation interact
Technology draws on and contributes to our understanding of the physics and psychophysics of sound
Computer music is interdisciplinary
HCI, AI, programming languages, algorithms, systems building
Also psychology, music theory, acoustics, signal processing, engineering, physics, performance practice, library science, applied math & statistics, …
Technology is constantly complicating and changing the landscape of our musical experiences as creators, participants, listeners, and consumers.
Open questions span many of these fields

29.
Software to try:
Chuck,
Miniaudicle,