1. Mapping GUIs to Auditory Interfaces
Mercator
Mapping GUIs to Auditory Interfaces

2. Goals
Provide transparent access to X Windows applications for computer users who are blind or severely visually impaired.
Build a framework to monitor, model, and translate graphical interfaces of X Window applications without modifying the application.
Develop a methodology for translating graphical interfaces into nonvisual interfaces.

3. Mercator Design Considerations
Modality for nonvisual interface -- auditory or tactile.
Substantial research in auditory interfaces (Gaver, Bly, Blattner et al)
Users’ ability to monitor multiple auditory signals - the “cocktail party effect” (Cherry)
Active versus passive interaction
Low cost, standard audio devices
Users’ possible lack of tactile sensitivity due to diabetes

4. Modeling Visual Interfaces

5. Auditory Widgets Correct level of abstraction Convey objects
… and attributes

6. Navigation Support two activities Mouse navigation is unsuitable
Allow users to “scan” interface
Allow users to operate interface
Mouse navigation is unsuitable
Map interface structure into hierarchical tree structure
Based on widget hierarchy
Users walk tree structure to navigate
Works with existing keyboard short-cuts

9. Mercator Architecture
Three Main Goals:
Capture -- high-level semantically meaningful information from X applications.
Store -- a semantic, off-screen model of the graphical interface.
Translate -- the stored information to present a consistent alternative interface.

10. Spectrum of solutions Modify the applications Modify the toolkit
Interpose between application and window server

11. First Architecture

12. Second Architecture

13. How this worked Xt hooks, RAP, Xlib safety net Control flow
Interface modeling
Interpreted interfaces
Simulating input

14. How to do this now...