Presentation on theme: "Collaborative Annotation, Archival and Visualization in a Biofeedback Rehabilitation system Hari Sundaram Arts Media and Engineering Arizona State University."— Presentation transcript:
Collaborative Annotation, Archival and Visualization in a Biofeedback Rehabilitation system Hari Sundaram Arts Media and Engineering Arizona State University
Memex Seminar, January 29, 2014 2 Introduction Motivation: Every 45 seconds, someone in the United States suffers a stroke. It results in functional deficits of neuropsychological and physical functions in post-stroke survivors. Up to 85% of patients have a sensorimotor deficit in the arm, such as muscle weakness, abnormal muscle tone, abnormal movement synergies, and lack of coordination during voluntary movement Goal Design a real time multimodal biofeedback system for stroke patient rehabilitation. Archival / annotation and information visualization to provide insight.
The Biofeedback system [To appear in acm mm 2006]
Memex Seminar, January 29, 2014 4 System Overview The Biofeedback system system situates participants in a multi-sensory engaging environment, where physical actions of the right arm are closely coupled with digital feedback. The Biofeedback system integrates five computational subsystems. Motion capture Motion analysis Audio feedback Visual feedback Database for archival and annotation All five subsystems are synchronized with respect to a universal time clock.
Memex Seminar, January 29, 2014 5 Action Analysis Arm Representation 11 labeled markers on arm and torso 3 labeled markers on the back of chair Feature Extraction 3D hand trajectory / 3D hand trajectory relative to the predefined straight line Shoulder / Elbow extension Hand Orientation Shoulder rotation / abduction/elevation Trunk flexion / rotation / lean and shoulder trajectory Wrist extension Multi-goal Framework Reaching Opening Flow
Memex Seminar, January 29, 2014 6 Coupling Action to Feedback Engagement Aesthetically attractive, easy to use and intuitive. Message and Mapping Reaching - visual target, an image completion/reassembly task, and an accompanying musical progression. Flow - pointalistic sound clouds in the main musical line, flowing particles in the visuals Opening - a rich, resonant musical accompaniment. Environment Introduction ( visual ) Abstract I (visual+audio) Abstract II (visual+audio), more variation
Memex Seminar, January 29, 2014 7 Audio Feedback Dynamic mapping of the normalized distance to target along the z coordinate to harmonic progression. Map the hand trajectory velocity in the z direction to event density. Joint Synchrony and Harmonic Progression. Shoulder - woodwind sounds (flute, clarinet, bassoon) through the progression Elbow - string sounds (a violin section of tremolo, a violin section, and a pizzicato violincello section). Mapping of Shoulder and Elbow Extensions Midi velocity (M v ) Duration (t d ) The probability of an octave doubling (P d )
Memex Seminar, January 29, 2014 8 Visual Feedback Transition Environment 3D virtual environment Physical movement will control the virtual environment. Abstract Environment A picture in a frame Explosion Turbulence Horizontal and Vertical Pull
Memex Seminar, January 29, 2014 9 Validation Offline Segmentation Reaction Reaching Grasping Returning Spatial Error Target-Hand Distance Hand Orientation Arm Openness Should Openness Elbow Openness Reaching Duration Flow Error Zero crossing number Polynomial curve fitting error Consistency
Memex Seminar, January 29, 2014 10 Results Reaching - our visual-audio feedback design can guide the normal subject to do the reaching as accurately as they did in real world. Openness - our audio feedback design for the abstract environment can help subjects with more openness. Flow - the smooth of speed curve means three things: Subjects are clearer the goal and they need not hesitant what will happen. Subjects are clearer about the feedback cue. Based on the current feedback and their memory, they can easily find the way to reach the target. Subjects start following the rhythm, that is mapped in the audio feedback.
Memex Seminar, January 29, 2014 12 Overview Challenges: Continuous data streams and large datasets Real-time annotation has high cognitive load We are integrating an archival subsystem into a team with different domain experts. Our Approach: Continuous multimodal archival Real-time collaborative annotation Offline information visualization
Memex Seminar, January 29, 2014 13 Archival Subsystem Design Part of our overall Biofeedback system Manage multimodal data streams Different data transport rates (total: 1.89MBps) Scalable Multicast Network raises synchronization problem
Memex Seminar, January 29, 2014 14 Continuous Multimodal Archival We split computational and storage resources into two archival subsystems 1.Archiving parametric system models Raw motion capture data Motion analysis parameters Audio-visual synthesis parameters Data was multicast 2.Contextual media capture Seven channels Actual audio-visual feedback data Three microphones Video camera Hardware: soundboard, microphones, VGA monitor scan- converter, video camera, mpeg hardware encoder, due-core server max / msp graphical program
Memex Seminar, January 29, 2014 15 Database design Indexing Scheme The patient / session / set / trial hierarchy Universal time stamp of synchronized subsystems Structural DB tables Motion capture and analysis parameters categorization Group audio-visual data by feedback semantics We first stream parametric data into a multi- buffered queue, then write to DB using bulk insert in parallel. We keep reference of multimedia data. Privacy issue
Memex Seminar, January 29, 2014 16 Real-Time Collaborative Annotation Why emphasize real-time collaborative? Annotations are time critical Each trial is short (~5 sec.) – there can be many unexpected events in this period – can cause cognitive overload. Team is focused on the experiment! Design goals of the annotation tool Distributive Personalized (Multi-disciplinary team) Collaborative
Memex Seminar, January 29, 2014 17 Annotation Interface Design Elements Dynamic experiment progression indicators Domain specific checklist Collaborative annotation sharing We multicast annotations from one client to others Random query, retrieval and modification User feedback is very positive
Memex Seminar, January 29, 2014 18 Information Visualization Offline visualization for review / annotation of archived data. Our design goals Hierarchical and selectable motion parameters / evaluation metrics navigation Synchronized contextual information playback Facilitates annotation modification Helps domain experts share information and improve their subsystems
Memex Seminar, January 29, 2014 19 Visualization Prototype Features: Allows navigation through our trial hierarchy on motion analysis results Contextual media playback with parametric motion analysis visualization Provides offline annotation facilities
Memex Seminar, January 29, 2014 20 Open Issues Event Model Event definition by domain experts Event detection Event Network Modeling Pre-emptive Annotation Show events with high priority Event log SenseCam Integration SenseCam pictures can be integrated into our visualization framework
thanks Team: Weiwei Xu, Yinpeng Chen, Richard Wallis, Thanassis Rikakis, Hari Sundaram, Todd Ingalls, Loren Olson, Jiping He, Sharon Liu