It sure is smart but can it swing? (Digital audio and computer music) COS 116: 2/22/2011 Sanjeev Arora (slides from Prof. Rebecca Fiebrink) 

Concrete example of how different concepts Today: Concrete example of how different concepts studied so far apply in one domain: Music. Opening video: http://www.music.princeton.edu/~nbritt/PLOrk-spring2010/video/G.mov What are the performers doing? How are they controlling the music?

Overview Sound and music in the physical world and in human experience Representations of music Analyzing music with computers Creating music with computers

1. Sound and music

What is sound? Discussion Time “Pressure wave” Vibrating objects -> sound pressure waves -> eardrum/inner ear -> brain Demo sndpeek: Show waveform of tuning fork, speech “Pressure wave”

What is music? Discussion Time “Organized sound” Organized sound: Composers and performers manipulate pitch, rhythm, timbre, dynamics, structure in meaningful ways Psychoacoustics: note, octave Culture: banned tritone http://www.lightbridgemusic.com/supp.htm People of a given culture and time period learn a lot about this organization just by exposure e.g., even if you aren’t a musician, you expect V7-I (example) At a very low level, when you expect that this pattern will occur, and your expectation is correct, your brain rewards you; this is one of the very low-level reasons for why we like music. In fact, our brain responds to music a lot like it does to narcotics. Naloxone -- blocks opiate receptors for heroin Psychoacoustics play an important role Also dependence upon history, culture, experience Engages listeners’ psychological mechanisms for expectation/reward

What do we hear? Frequency Pitch Loudness Timbre … (start with chipmunk clip; pause it, and then let words about frequency/pitch/loudness appear).

Psychoacoustics Relationships between physical phenomenon of sound and our perception Frequency <-> pitch 20-20,000Hz Amplitude <-> loudness Identities and strengths of frequencies present : timbre Show sndpeek again with sounds Mention that they will explore these issues in the lab. + =

Spectral representation Every wave (sounds included) can be decomposed into “pure sine waves”, each of a single frequency.

2. Representations of sound and music

Discussion Time How do you represent music? Score: Audio samples Spectrum Q: How do we represent music? Show: score vs. audio file Ask aloud “is score a faithful representation?” Q: What is the relationship between the score representation and what you hear? Explain spectrum - Show adding waves in time Adding waves in frequency Show spectogram - 440, 880, both. (with sound)

Digital representation of music Qs: How can computer halve the frequency of this signal? Describe sampling, quantization Mention that this is a “lossless” format, used for WAV, AIFF, CD, etc. Just like we saw on the last slide, we can use less information to represent a sound and have a representation that is less faithful/complete: **downsampling** (play example of downsampled audio): Q: What’s a benefit of using a lower sampling rate?

Compression Why? Saves bandwidth, memory etc. How? Trivial but very lossy: sample less often Cleverer: use psychoacoustic principles MP3: Masking Even cleverer: use physical principles of sound production (via models) of sound source) By being smart about what information we leave out, we can get a better representation with fewer bits Show sndpeek- just my voice: how spectrum changes w/ vowel Sound example: phasor source, formant filter, source-filter

Choosing a representation Representations are compromises Standard representations are somewhat arbitrary (e.g. MIDI) Best representation is task-dependent Compromised: Frequency and dynamic range Composer’s intention (what is notated) Performance gestures Aribtrariness: CD length, MP3 parameters (for popular music)

3. Using technology to analyze sound and music

Analyzing speech Real-life apps: Customer service phone routing Voice recognition software Hard questions (for a computer): Identity, language, content

Computational Auditory Scene Analysis Applications: Archival and retrieval, forensics, AI e.g., forensic examination of tapes from RFK assasination reveal a second gunman.

Music information retrieval Analyzing musical data Query, recommend, visualize, transcribe, detect plagiarism, follow along with a score Sites you can try midomi.com Themefinder.com Pandora.com (human-driven), last.fm Sound example: Hum melody (just me) can a computer transcribe this? Figure out my tempo? Find a file on your hard drive that has this melody? Find melodies that are similar? Written by the same composer? Give me feedback on how well I sang it?

Machine learning for analysis Given the types of music that the listerner likes; recommend new music that he or she might like.

(1) What does this pseudocode do? Variable i; for i=1 to 4 do { if (i < 3) then print( i*i) else print (i);} (2) Write 33 in binary.

4. Using technology to create music and sound A whirlwind tour of the 20th century, with a focus on computer technology

Creating music: Synthesis Examples: Stevie Wonder Tina Turner (FM) Kanye West/ Daft Punk (vocoder) http://www.youtube.com/watch?v=b2xBVbzPftA&feature=related

Three approaches to synthesis Additive synthesis Figure out which frequencies are present, and in what proportions Synthesize a sine wave at each frequency, and superpose them. Physical modeling Start with knowledge about how physical systems oscillate Simulate oscillation of an arbitrary system. (Recall Lecture 4) + + …=

Three approaches to synthesis Cross-synthesis (eg vocoder) Choose filter for speech (vowel) Choose source to be another sound Mention that this was generated with code written at Princeton

Some continua of computer music creation low Synthesis Running a program Computer follows performer Live coding, special controllers Interactivity Controlling synthesis Oboe demo! Composition high Level of control

Performer-Computer Interaction Augmented instruments Software and hardware interfaces Demo: PLOrk video, PBS Demo: using a Wii-mote to control sound New instruments Perry’s Mug. Live coding Demo: Max’s drum machine Chuck/miniaudicle http://sciencestage.com/v/29622/princeton-laptop-orchestra-princeton-university.html http://www.youtube.com/watch?v=B_waqIozoVw Download chuck and miniaudicle from http://soundlab.cs.princeton.edu/software/

Questions: How can we…. develop new ways to synthesize sound? give user control over synthesis parameters? make machines interactive in a musical way? augment human capabilities? design new instruments that are easy to play? allow expert musicality? create music that is emotionally and aesthetically compelling?

Final remarks Distinctions in this presentation are superficial Analysis, representation, and creation interact Technology draws on and contributes to our understanding of the physics and psychophysics of sound Computer music is interdisciplinary HCI, AI, programming languages, algorithms, systems building Also psychology, music theory, acoustics, signal processing, engineering, physics, performance practice, library science, applied math & statistics, … Technology is constantly complicating and changing the landscape of our musical experiences as creators, participants, listeners, and consumers. Open questions span many of these fields

http://soundlab.cs.princeton.edu/ Software to try: Chuck, Miniaudicle,