1. Can you see what I hear? The Design and Evaluation of a Peripheral Sound Display for the Deaf F. Wai-ling Ho-Ching, Jennifer Mankoff, James A. Landay Presented by: Eshita Sharmin

2. Introduction  People use sound in many subtle ways to gain awareness of the state of the world around them  In 1997 there were 3.4 millions Americans with difficulty hearing, of which 227,000 were deaf  Explored the ways in which hearing people use sound in their everyday lives  Conceptual drawings of sound by hearing participants and exploration with a deaf participant using paper prototypes formed the basis for the design

3. Categories  Visualizing Audio Employ waveforms and spectrographs as visualizations, and are targeted at expert users. Not meant to display non-speech sounds  Monitoring and Notification Supports monitoring of peripheral information without grabbing the users attention Displays lie on the boundary between background and foreground awareness Must notify the user of interesting information without impeding the performance of a primary task Requires careful design

4. Sound Awareness Tools TechniqueApplicationCharacteristicsCons Vibration sensingAwareness of sounds that create vibration, e.g. sensing footsteps, feeling that a computer is on Does not require focus of attention Supports some ambient and some notification sounds Depends on infrastructure (e.g. having hardwood floors). Flashing lightsAwareness of telephones, doorbells (Notification sounds) Supports notification soundsFixed visual attention Must hook up each device Hearing DogsAwareness of all soundsSupports notification sounds onlyRequires ongoing maintenance Requires a priori training per sound Visual InspectionMultiple applications, e.g. Steam for a kettle, Looking out the window for the arrival of a guest Sometimes is the only alternative Some ambient, some notification Different for each sound Polling rather than interrupt based Hearing Aids, Cochlear Implants Enhancing existing hearing, but not at the fidelity of “normal” hearing Enhances awareness of all soundsRequires training for interpretation of sounds Results vary by case

5. Use of Sound in the Home and at Work

6. Needs of the Deaf  Awareness of the presence of others  Interaction with sound based appliances  Leaving the home environment

7. Intuitive Visualization of Sound

8. Two Prototypes

9. Experiment- Participants and Apparatus

10. Procedure and Design

11. Results (1/2)  Correct detection

12. Results (2/2)  Distraction Measured by counting the rate at which a participant could select 0’s in the primary task. The performance of primary task was compared across three conditions Monitoring of the Spectrograph prototype Monitoring of the Ripple prototype Absence of a secondary task The rings visualization was perceived as less distracting than the Spectrograph  Learning Ripples visualization was easier to learn than Spectrograph

13. Qualitative Observations  Field Study Deployed the Spectrograph visualization for a week with a deaf graduate student Findings are  Robustness is extremely important  Participant was able to detect a number of sounds  Pilot Study “This is great! … I’m learning to hear again after 30 years”

14. F. Wai-ling Ho-Ching  Working as a research intern in the Adaptive Systems and Interaction group at Microsoft Research (Feb 9, 2003).  Graduated in December 2002 from UC Berkeley with a Masters in Computer Science. During study at Berkeley, she worked on a visual display of sound for the deaf with Jennifer Mankoff and James Landay.  Recieved a Bachelors of Computer Science degree with an English minor at Simon Fraser University.  born and raised in British Columbia, Canada

15. Publications  F. Wai-ling Ho-Ching, Jennifer Mankoff, James A. Landay. (2003) Using peripheral displays to provide the deaf with awareness of environmental audio. Workshop paper accepted to the Elegant Peripheral Awarness Workshop at CHI 2003. 4 pages.  F. Wai-ling Ho-Ching, Jennifer Mankoff, James A. Landay, (2003). From Data to Display: the Design and Evaluation of a Peripheral Sound Display for the Deaf. In Proceedings of CHI 2003. 8 pages. In press.  Edward Deguzman, F. Wai-ling Ho-Ching, Tara Matthews, Tye Rattenbury, Maribeth Back, Steve Harrison, (2003). EEWWW!: Tangible Instruments for Navigating into the Human Body, In Extended Abstracts of CHI 2003. 2 pages. In press.  Inkpen, K, Ho-Ching, W., Kuederle, O. Scott, S and Shoemaker, G. (1999), This is Fun! We're All Best Friends and We're all Playing": Supporting Children's Synchronous Collaboration. CSCL'99 Proceedings. 252-259.  Ho-Ching, W., Inkpen, K. (1999), Multiple Mice: Realizing Single Display Groupware, Extended Abstract and Interactive Poster Demonstration presented at ASI 1999 (Vancouver, BC), CSCL 2000 (Palo Alto CA).

16. Jennifer Mankoff  EECS Department University of California, Berkeley  Received Ph.D. as a member of the Future Computing Environments research group in the College of Computing at Georgia Tech

17. Publications  Mankoff, J., Dey, A.K., Hsieh, G., Kientz, J., Ames, M., Lederer, S. Heuristic evaluation of ambient displays. CHI 2003, ACM Conference on Human Factors in Computing Systems, CHI Letters 5(1): 169-176. 2003.  J. Heer, N. Good, A. Ramirez, M. Davis and J. Mankoff. Presiding Over Accidents: System Mediation of Human Action. In Proceedings of CHI 2004, ACM Conference on Human Factors in Computing Systems, CHI Letters 6(1):To Appear.  Melody Y. Ivory and Jennifer Mankoff and Audrey Le. "Using Automated Tools to Improve Web Site Usage by Users with Diverse Abilities". In Information Technology and Society. 3(1), pp. 195-236.Information Technology and Society.  Jennifer Mankoff and Gary Hsieh and Ho Chak Hung and Sharon Lee and Elizabeth Nitao. "Using Low-Cost Sensing to Support Nutritional Awareness". In Proceedings of Ubicomp 2002. G. Borriello & L.E. Holmquist (Eds.), October, 2002. LNCS 2498. Springer-Verlag. pp. 371-378.  A. Dey, and J. Mankoff, "Designing Mediation for Context-Aware Applications," ACM Transactions on Computer-Human Interaction, special issue on Sensing- Based Interactions. To Appear.

18. James A. Landay  Associate Professor, Computer Science Division, University of California, BerkeleyComputer Science DivisionUniversity of California, Berkeley  Carnegie Mellon University, Pittsburgh, PA (8/90-12/96) Ph.D. in Computer Science, 1996 Thesis: Interactive Sketching for the Early Stages of User Interface Design M.S. in Computer Science, 1993  University of California, Berkeley, CA (8/85- 5/90) B.S. in Electrical Engineering/Computer Science with High Honors, 1990

19. Publications  James A. Landay and Brad A. Myers, "Sketching Interfaces: Toward More Human Interface Design." IEEE Computer, vol. 34, no. 3, March 2001, pp. 56-64.  Jason Hong and James A. Landay, "WebQuilt: A Framework for Capturing and Visualizing the Web Experience." To appear in Proceedings of The Tenth International World Wide Web Conference, Hong Kong, May 2001.  Jason Hong and James A. Landay. "A Context / Communication Information Agent." In Personal and Ubiquitous Computing, Special Issue on Situated Interaction and Context-Aware Computing. 5(1): Springer-Verlag. 2001, pp. 78-81.  Oviatt, S.L., Cohen, P.R., Wu, L.,Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J. & Ferro, D. "Designing the user interface for multimodal speech and gesture applications: State-of- the-art systems and research directions," Human Computer Interaction, 2000, vol. 15, no. 4, 263-322 (to be reprinted in Human-Computer Interaction in the New Millennium, ed. by J. Carroll), Reading, MA.: Addison- Wesley, in press).  Hesham M. Kamel and James A. Landay. "A Study of Blind Drawing Practice: Creating Graphical Information Without the Visual Channel." In Assets 2000: Proceedings of the Fourth ACM Conference on Assistive Technologies, Arlington, VA, Nov. 2000, pp. 34-41.  Scott R. Klemmer, Anoop K. Sinha, Jack Chen, James A. Landay, Nadeem Aboobaker, Annie Wang, "SUEDE: A Wizard of Oz Prototyping Tool for Speech User Interfaces." CHI Letters: ACM Symposium on User Interface Software and Technology: UIST '00, 2000. 2(2): p. 1-10.  Jason I. Hong and James A. Landay, "SATIN: A Toolkit for Informal Ink-based Applications." CHI Letters: ACM Symposium on User Interface Software and Technology: UIST '00, 2000. 2(2): p. 63-72.  James Lin, Mark W. Newman, Jason I. Hong, and James A. Landay. "DENIM: Finding a tighter fit between tools and practice for web site design." CHI Letters: Human Factors in Computing Systems, CHI 2000, 2000. 2(1): p. 510-517.  Allan C. Long, James A. Landay, and Lawrence A. Rowe. "Visual Similarity of Pen Gestures." CHI Letters: Human Factors in Computing Systems, CHI 2000, 2000. 2(1): p. 360-367.  "Using Note-Taking Appliances for Student to Student Collaboration." James A. Landay, in Frontiers in Education '99, San Juan, Puerto Rico, November 1999.

20. ?