1. M ND: Music Improvisation and Narrative Design
Peter Khooshabeh, Ethan Smith, John Thompson
MIND, a human-centered musical improvisation system supports the lifestyles of the elderly. MIND benefits the elderly by providing entertainment and usefulness through a music therapy paradigm
Motivation
Create artificial companionship by allowing elderly to interact with a music synthesis system
Segmentation
CAMERA
DELAY +
n = 100
BACKGROUND MODEL A B
THRESHOLD -
CENTROID CALCULATION
For the first 100 frames, motion is detected via temporal differencing while a background model is constructed. After 100 frames, background subtraction is used to detect ‘foreign’ objects.
Current Frame
Background Model
Background Subtraction
Thresholded
Background
Subtraction
Temporal
Difference
Previous Frame
Hardware & Software
Three Unibrain fire-I cameras, four loud speakers, Windows XP PC, Mac and G4.
OpenCV library, OSC, SuperCollider
LeftCam
CenterCam
RightCam
Generative Nodes
Chord Nodes
Impulse Nodes
Sample Playback
Transformative Nodes
Convolution
Pitch/Time Warping
Sonic Nodes:
Nodes with various sonic functions
are distributed in the virtual space.
Center of Mass
Center of Mass
Center of Mass
Adaptive Space:
The system adapts its state and musical structures based on the activity of the space.
Four activity levels are determined:
High activity state
Moderate activity state
Low activity state
Sparse activity state
Each state determines what synthesis algorithms will be implemented in that state as well as whether the nodes should adjust their position in relation to the tracked object
Camera calibration
Tracking results
2.09 fps
OSC Messages
World_X
3D Position Calculation:
Epipolar Geometry
Least Square
World_Y
Xcamera =MextXworld
Ximage =MintXcamera
World_Z
A captured frame from the visualization
Performance: Estimation Error (cm)
EPP-avg
EPP-std
LS-avg
LS-std
X-direction
3.6
7.0
5.3
8.7
Z-direction
5.9
33.5
5.7
30.0
The Data Manager distributes the tracking results to the visualization system and the synthesis servers
Music Synthesis
Calibration/Tracking
Object paths create phrase level material in the mobile form