1. Advanced Multimedia Music Information Retrieval Tamara Berg

2. Announcements Still missing a few assignment 1’s Assignment 2 is online – due March 10 Assignment 2

3. Audio Indexing and Retrieval Motivation Features for representing audio: – Metadata – low level features – high level audio features Example usage cases: Audio classification Music retrieval

4. Howard Leung

5. Content Based Music Retrieval Extract music descriptions from a database of music documents. Extract music description from a query music document. Compute match between query and database descriptions. Retrieve similar music documents to query. Casey et al IEEE 2008

6. MIR tasks H: high level specificity – match specific instances of audio content. M: mid-level specificity – match high level audio features like melody, but do not match audio content. L: low specificity – match global (statistical) properties of the query Different usage cases require different descriptions and matching schema. Casey et al IEEE 2008

7. Metadata Most common method of accessing music Can be rich and expressive When catalogues become very large, difficult to maintain consistent metadata Useful for low specificity queries Casey et al IEEE 2008

8. Metadata Pandora.com – Uses metadata to estimate artist similarity and track similarity and creates personalized radio stations. Human entered metadata of musical- cultural properties (20-30 minutes per track of an expert’s time – 50 person-years for 1 million tracks). Pandora.com User contributed metadata repositories (gracenote, musicbrainz). Factual metadata (artist, album, year, title, duration). Cultural metadata (mood, emotion, genre, style).gracenote musicbrainz Automatic metadata methods – generate descriptions from community metadata automatically. Language analysis to associate noun and verb phrases with musical features (Whitman & Rifkin). Casey et al IEEE 2008

9. Content features Low level or high level Want features to be robust to certain changes in the audio signal (why?) – Noise – Volume – Sampling High level features will be more robust to changes, low level features will be less robust. Low level features will be easy to compute, high level difficult

10. Low level audio features Low level measurements of audio signal that contain information about a musical work. Can be computed periodically (10-1000 ms intervals) or beat synchronous. Casey et al IEEE 2008 In text analysis we had words, here we have to come up with our own set of features to compute from audio signal!

11. Example Low-Level Audio Features Howard Leung

13. Average number of times signal crosses zero amplitude value. 1 if true O o.w.

14. Howard Leung

15. Frequency Domain Reminder How much of each describes the frequency spectrum of a signal. Li & Drew Signals can be decomposed into a weighted sum of sinusoids

16. Frequency domain features How do we get to frequency domain? TimeFrequency

17. DFT Discrete Fourier Transform (DFT) of the audio Converts to a frequency representation DFT analysis occurs in terms of number of equally spaced ‘bins’ Each bin represents a particular frequency range DFT analysis gives the amount of energy in the audio signal that is present within the frequency range for each bin Inverse Discrete Fourier Transform (IDFT) Converts from frequency representation back to audio signal.

18. DFT Discrete Fourier Transform (DFT) of the audio Converts to a frequency representation DFT analysis occurs in terms of number of equally spaced ‘bins’ Each bin represents a particular frequency range DFT analysis gives the amount of energy in the audio signal that is present within the frequency range for each bin Inverse Discrete Fourier Transform (IDFT) Converts from frequency representation back to audio signal.

19. DFT Discrete Fourier Transform (DFT) of the audio Converts to a frequency representation DFT analysis occurs in terms of number of equally spaced ‘bins’ Each bin represents a particular frequency range DFT analysis gives the amount of energy in the audio signal that is present within the frequency range for each bin Inverse Discrete Fourier Transform (IDFT) Converts from frequency representation back to audio signal.

20. Howard Leung

21. Filtering Removes frequency components from some part of the spectrum Low pass filter – removes high frequency components from input and leaves only low in the output signal. High pass filter – removes low frequency components from input and leaves only high in the output signal. Band pass filter – removes some part of the frequency spectrum.

22. How could you do this using the FT and IFT? Compute FT spectrum of input. Zero out the part of the frequency spectrum that you want to filter out. Compute the IFT of this modified spectrum -> output will be input with some frequency components removed.

23. How could you do this using the FT and IFT? f = input

24. How could you do this using the FT and IFT? f = input FT(f)

25. How could you do this using the FT and IFT? 1 0.* f = input FT(f)

26. How could you do this using the FT and IFT? 1 0.* f = input FT(f) Zero out some freq components

27. How could you do this using the FT and IFT? 1 0.* = f = input FT(f) Zero out some freq components x xxx xxxxx xxxx x

28. How could you do this using the FT and IFT? 1 0.* = f = input FT(f) Zero out some freq components IFT o = Frequency limited output x xxx xxxxx xxxx x

29. How could you do this using the FT and IFT? 1 0.* = f = input FT(f) Zero out some freq components IFT o = Frequency limited output x xxx xxxxx xxxx x What kind of filter is this?

30. How could you do this using the FT and IFT? f = input

31. How could you do this using the FT and IFT? f = input FT(f)

32. How could you do this using the FT and IFT? 1 0.* f = input FT(f)

33. How could you do this using the FT and IFT? 1 0.* f = input FT(f) Zero out some freq components

34. How could you do this using the FT and IFT? 1 0.* = f = input FT(f) Zero out some freq components x x x x xx xxxxxx x

35. How could you do this using the FT and IFT? 1 0.* = f = input FT(f) Zero out some freq components IFT o = Frequency limited output x x x x xx xxxxxx x

36. How could you do this using the FT and IFT? 1 0.* = f = input FT(f) Zero out some freq components IFT o = Frequency limited output x x x x xx xxxxxx x What kind of filter is this?

37. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution (we’ll see this again for images): (convolution demo)demo 13253245 1/3 f = g = signal filter

38. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/3 *** = -2

39. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/3 *** = -210/3

40. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/3 *** = -210/3

41. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/3 *** = -210/3

42. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/3 *** = -210/3 3

43. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/3 *** = -210/3 311/3- What does this filter do?

44. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 f = g = signal filter

45. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.25

46. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.253

47. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.75

48. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.753.25

49. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.753.252.75

50. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo)demo 13253245 1/41/21/4 *** = -2.2533.753.252.753.75- In general filters will have a more complex effect on the output.

51. What is convolution doing?

52. Relationship f = input F = ft(f) g = filter G = ft(g) f f ★ g

53. Relationship f = input F = ft(f) g = filter G = ft(g) f f ★ g FF.*G

54. Relationship f = input F = ft(f) g = filter G = ft(g) f f ★ g FF.*G FT Theorem: Convolution in signal space is equivalent to point- wise multiplication in frequency space.

55. Relationship f = input F = ft(f) g = filter G = ft(g) f f ★ g FF.*G FT f ★ g = IFT(F.*G) F.*G = FT(f ★ g) Theorem: Convolution in signal space is equivalent to point- wise multiplication in frequency space.

56. Matlab demo soundFilt/demo.m

57. Howard Leung

59. Pitch-Class Profile (PCP) Represent the energy due to each pitch class Integrates the energy in all octaves into a single band There are 12 equally spaced pitch classes in western tonal music. So, typically 12 bands in the PCP.

60. Pitch-Class Profile (PCP) Represent the energy due to each pitch class Integrates the energy in all octaves into a single band There are 12 equally spaced pitch classes in western tonal music. So, typically 12 bands in the PCP. How might we calculate this using the DFT?

61. Howard Leung

62. High level music features High level intuitive information about a piece of music (melody, harmony etc). “It is melody that enables us to distinguish one work from another. It is melody that human beings are innately able to reproduce by singing, humming, and whistling. It is melody that makes music memorable: we are likely to recall a tune long after we have forgotten its text.” -Selfridge-Field Intuitive features, but hard to extract and ongoing areas of research. Casey et al IEEE 2008

63. Melody & Bass Estimation Melody and bass lines represented as continuous temporal trajectory of fundamental frequency, F0 (a series of musical notes). PreFEst (Predominant-F0 Estimation method – Goto 1999 ) – Estimate the F0 trajectory in mid-high freq range of input -> melody. – Estimate the F0 trajectory in low freq range-> bass. Casey et al IEEE 2008

64. Chord Recognition Musical performance is assumed to travel through a sequence of states. Hidden Markov Model (HMM – probabilistic model good for modeling sequences of data, here sequences of chords over time) is used to model these transitions and predict the best chord sequence given a set of observations (PCP). Transition model – Probability of transitioning from one chord to another Output model – Probability of a PCP given a chord. Casey et al IEEE 2008

65. Chord Recognition

67. Music Structure Segment into temporal regions with some internal consistency – Beat segmentation – Verse, chorus, bridge – Speech vs music Uses: – facilitate audio editing – Improve similarity measurements by removing irrelevant parts or selecting most representative parts (for recommender systems). Casey et al IEEE 2008

68. Music Structure Detect repeated structures and label them as being the same.

69. Music as vector of features Once again we represent (music) documents as a vector of numbers – Each entry (or set of entries) in this vector is a different feature

70. Music as vector of features Once again we represent (music) documents as a vector of numbers – Each entry (or set of entries) in this vector is a different feature To retrieve music documents given a query we can: – Find exact matches – Find nearest match – Find nearby matches – Train a classifier to recognize a given category (genre, style etc).

71. Audio Similarity We have a description of a music document based on some set of features, now how do we compare two descriptions? Casey et al IEEE 2008

72. Usage examples

73. Howard Leung

79. Query by humming Requires high level features because matches will not be exact Extract melody from dataset of songs Extract melody from hum Match by comparing similarities of melodies (nearby matches)

80. Copyright monitoring Compute fingerprints from database examples Compute fingerprint from query example Find exact matches

81. Best performing systems on MIREX 2007 Casey et al IEEE 2008

82. Music Browsing Musicream – UI for discovering and managing musical pieces. User can select a disc and listen to it. By dragging a disc in the flow, the user can easily pick out other similar pieces (attach similar discs). This interaction allows a user to unexpectedly come across various pieces similar to other pieces the user likes. Link to demo Casey et al IEEE 2008

83. Music Browsing Musicrainbow – UI for discovering unknown artists. Artists are mapped on a circular rainbow where colors represent different styles of music. Similar artists are mapped near each other. User rotates rainbow by turning a knob. Link to demo Casey et al IEEE 2008

84. Howard Leung