1. Music
Computer & New Media

2. Topics for Today Characterizing music Identifying music
Music recommendation
Music summarization
Interactive music
Composition support
Automatic music composition
Music management support

3. Characterizing Music From the audio signal From metadata Beat
Brightness
Pitch
Starting note
Potential key (music scale)
From metadata
Artist
Genre(s)
Album name, release date
On-line ratings/comments
Inclusion in playlists
Lyrics

4. Identifying Music
Music transcription
Query by humming

5. Music Recommendation For an individual For a playlist For a group
Cluster based on features and recommend music similar to what individual likes
For a playlist
Similar, but add constraints to fill particular amount of time
Can also use song-to-song similarity to determine order of music
For a group
Find music that is most liked, least disliked, or some combination thereof
For an activity
Use mood or similar information

6. Music Summarization
Most summaries in commercial sites are either the first phrase or a single selected musical phrase
Study of whether 22 second long multi-phrase music summaries would be better previews
Three algorithms compared that vary the selection of the components between phrases that are
sonically distinct and phrases that are
repeated more often
A comparative evaluation study showed that:
Users preferred multi-phrase previews in 87% of the cases over the preview representing the first 22 seconds of the song
90% of all of the multi-phrase summaries were considered at least as good representation of the song as the song’s starting segment

7. Interactive Music Music games MobiLenin Guitar Hero, Rock Band, …
Enable interaction with music in a public space
Not karaoke
Voting like in many pub/bar games
Audience can affect which version of music and video is shown

8. MobiLenin Interface and Process

9. Lessons Gave a focal point for interaction between members of a group
Content variety is necessary for continued engagement
Lottery for free beer motivated participation
Well Duh!

10. Music Composition Support
Lots of software tools that help composers generate new works
Rapid editing and visualization are common features
Some systems provide new models of composition
PaperTonnetz

11. Automatic Music Composition
Generation of a musical phrase by learning from examples
The Continuator adds phrases in real time using a Markov model based on the artist’s style
Example 1
Example 2
Automatic composition of details given higher-level structure of the desired music
RapidComposer generates the details given a chord progression

12. Managing Personal Music Collections
Music management is often based on:
explicit attributes (e.g. metadata values like the artist, album, and genre).
explicit feedback (e.g. ratings of preference and relevance)
Benefits
Easy to understand
Formal: consistent updating and access
Context-free
Question
How can music be accessed based on harder to express personal connections
The feelings or memories it triggers?
e.g. Music that sounds happy, makes us feel gloomy or reminds us of a person

13. Current Practices
Common metadata tags usually not sufficient to describe mood, feelings, memories, etc.
Effort/benefit trade-off
Personal reactions change
Explicit feedback and usage statistics helpful in retrieving music of preference
How would people organize music if there was a low-effort way of expressing their personalized interpretation of music?
Use of additional tags or customization of the existing ones can be tedious
Use of additional ratings associated to specific music attributes can be overwhelming for the user

14. Preliminary Study
12 participants asked to organize songs & create playlists in spatial hypertext
Visual attributes & spatial layout are changed to express associations
The majority found spatial hypertext helpful for organizing
Participants appreciated:
expressive power and freedom of the workspace
directly accessible metadata information of music
music previews for remembering music
Participants missed:
interactive hierarchical / tree views based on metadata
music previews for understanding music

15. Statistics of Artist Similarity Relatedness Assessment
MusicWiz Architecture
MusicWiz Interface
Music management environment that combines:
explicit information
implicit information
non-verbal expression of personal interpretation
Two basic components:
interface for interacting with the music collection
inference engine for assessing music relatedness
Metadata Module
Audio Signal Module
Lyrics Module
Worksp. Express. Module
Artist Module
Relatedness Table
Inference Engine
Workspace Status
Related Song Titles
Music Collection
Songs & Metadata
Songs
Lyrics
Statistics of Artist Similarity
Internet
Relatedness Assessment
Sim. Values

16. MusicWiz Interface Hierarchical Folder Tree View Workspace
Playlist Pane
Related Songs & Search Results View
Folder Tree View: Provides a location-based hierarchical views of the music collection.
Related Songs & Search Results View: Displays songs that are similar to the currently selected songs in the system tree view or the results of the search. Songs
then can be dragged and dropped from the list into the workspace and the playlist pane to update collections and playlists respectively.
Playback Controls
The MusicWiz interface

17. MusicWiz Music Similarity Assessment
5 modules for extracting, processing and comparing artists, metadata, audio content, lyrics, and workspace expression
Overall Similarity (S1, S2) =
= W1 * Overall Metadata Similarity(S1, S2) + + W2 * Overall Audio Signal Similarity(S1, S2) + + W3 * Overall Lyrics Similarity(S1, S2) + + W4 * Overall Workspace Expression Similarity(S1, S2)
where,
S1, S2 are the songs under comparison and Wn, n = 1..4 the user adjusted weights of the specialized similarity assessments
Each module produces an assessment of relatedness (a normalized value ranging from 0 – songs very dissimilar, to 1 – songs almost identical)

18. MusicWiz Similarity Assessment – Workspace Expression Module
Music objects can be related visually and spatially
Spatial parser identifies relations between the music objects
Recognizes three types of spatial structures: lists, stacks and composites
Composite
List
Stack

19. MusicWiz Evaluation 20 participants were asked to:
Task 1: organize 50 rock songs into sub-collections according to their preference
Task 2: form three, twenty-minute long playlists based on three different moods or occasions of their choice
Task 3: form three six-song long playlists, where each of them had to be related to a provided “seed”-song (not from the fifty of the original collection)

20. MusicWiz Evaluation Configuration No Suggestions Suggestions
Importance of the workspace
Importance of the music previews
Four groups of system use:
Group 1 (no workspace / no suggestions) had to complete the three tasks using MusicWiz’s browsing, searching, and playback functionality and Windows Explorer folders to form the collections and playlists
Group 2 (no workspace / with suggestions) used the same features as group 1 but also received suggestions from the similarity inference engine
Group 3 (with workspace / no suggestions) had to perform the tasks using the features available in group 1 but used the MusicWiz workspace to create the collections and playlists
Group 4 (with workspace / with suggestions) had all MusicWiz features
Configuration
No Suggestions
Suggestions
No Workspace
Group 1
Group 2
Workspace
Group 3
Group 4

21. Task 1 - Organization of Music
Statement
(1 – “I strongly disagree” to
7 – “I strongly agree”)
Group 1
Group 2
Group 3
Group 4
The system support in
organizing effortlessly /
quickly was enough
4.4
5.4
5.6
6.2
Enjoyed doing task
5.8
6.4 6
Organization will be
easily understood by
others
4.2
Configuration
No Suggestions
Suggestions
No Workspace
Group 1
Group 2
Workspace
Group 3
Group 4

22. Tasks 2 & 3 – Playlist Creation
Statement
(1 – “I strongly disagree” to
7 – “I strongly agree”)
Task
Group 1
Group 2
Group 3
Group 4
System support for
quick selection was
enough
Two
4.8
6.2
5.8
Three
4.4
6.8
5.6
finding music 6
5.4
4.6
6.4
Enjoyed doing task
5.2
6.6
Configuration
No Suggestions
Suggestions
No Workspace
Group 1
Group 2
Workspace
Group 3
Group 4

23. Music Overview Music processing can support a variety of activities
Composition
From traditional to interactive
Selection & Recommendation
Example: iTunes, Pandora,
Use for shared spaces
Playback
Example: MobinLenin
Management & Summarization
Example: MusicWiz
Games
Guitar Hero, Rockband, etc.