1. Vocabularies for Description of Accessibility Issues in MMUI Željko Obrenović, Raphaël Troncy, Lynda Hardman Semantic Media Interfaces, CWI, Amsterdam zeljko.obrenovic@cwi.nl http://www.cwi.nl/~obrenovi/

2. Introduction What (output) modalities are most suitable in which situation? How should different (output) modalities be combined? Using rich accessibility description of MM UIs to answer these two questions

3. Motivation Expressing human functionality and anatomical structures required by modalities –Capturing existing knowledge –Reasoning over modalities Limited scope, small non-standard vocabularies –Card, Mackinlay, Robertson: Morphological analysis of input device; –Modality theory, Niels Ole Bernsen: Based on taxonomy of output modalities; Linguistic, analog, arbitrary, static, media…

4. Describing Accessibility Issues in Multimodal User Interfaces MM user interfaces - systems that communicate a message, an effect, –Stimulating a particular human functionality or anatomical structures Interaction constraints –Influence of various factors on human anatomical structures and functionalities.

5. Describing Accessibility Issues in Multimodal User Interfaces Modality User constraints Social constraints Environment constraints Device constraints Accessibility issues (interaction constraints) Multimodal issues Computer Human Sensing Perception Cognition Motor skills Linguistic skills

6. Interaction Modalities Interaction Constraints Interaction Context described in terms of Reasoning Framework define queryresult Applications described in terms of Vocabularies

7. WHO Resources Bioinformatics Additional concepts Reasoning Framework queryresult Applications Interaction Modalities Interaction Constraints Interaction Context described in terms of define described in terms of

8. Vocabularies: ICF 1/2 WHO International Classification of Functioning, Disability and Health (ICF): –2001, 9 years revision, widely used in health community Describing “the person in his/her world“ –Applicable to all people, whatever their health condition –Carefully designed to be relevant across cultures as well as age groups and genders –Uses neutral terms (function vs. disease) –Allows for an assessment of the degree of disability

9. Vocabularies: ICF 2/2 Around 1500 concepts: –Body functions and structure –Activities (tasks and actions by individual) and participation (involvement in a life situation) –Environmental factors Problems with formalization –Not defined by knowledge engineering experts Formalized ICF Checklist: 180 core concepts Possibility to reuse millions of profiles and statistical data expressed by ICF!

10. Vocabularies: Anatomy Foundational Model of Anatomy (FMA) –Detailed description of human anatomy –Open source and available for general use –Used in bioinformatic community –~ 100 000 concepts –Available in OWL format –Problems: size

11. Vocabularies: Interaction Effects Some examples: –Gestalt visual grouping By similarity, motion, texture, symmetry, proximity, parallelism, closure, good continuation –Gestalt visual highlighting By color, polarity, brightness, orientation, size, motion, flicker, depth, shape –3D cues Visual: stereo vision, motion parallax, linear perspective, rel. size, shadow, familiar size, interposition, horizon Audio: inter-aural time/intensity difference, HRTF, head movement, echo, attenuation of high frequencies

12. Vocabularies WHO Resources Bioinformatics Additional concepts Reasoning Framework queryresult Applications define described in terms of Interaction Context described in terms of Interaction Modalities Interaction Constraints

13. Example Modality: Speech

14. Example Constraint: Noise

15. Interaction Constraints Interaction Context Device profile User profile Environment profile User interface described in terms of Vocabularies WHO Resources Bioinformatics Additional concepts Reasoning Framework queryresult Interaction Modalities define described in terms of Applications

16. Reasoning: Using Descriptions Possible with rich and explicit descriptions of (implicit) accessibility requirements What modalities are suitable in which situation? –Combining descriptions of constraints and modalities (speech in noisy environment?) How should different modalities be combined? –Combining modality descriptions to identify conflicting requirements (speech & short term memory);

17. Implementation In early stage Using Semantic Web technologies: –OWL, RDF standards –Tools and database to support reasoning Sesame, Jena…; –Reusing some of our and existing tools for data visualization and exploration Facet browsing

18. Conclusions Expressing human functionality and anatomical structures required by modalities Relations with other projects: –W3C Multimodal Interaction Framework - content –W3C Accessibility Initiative (WAI) - guidelines Future work: –Full implementation of reasoning framework –Solving problems with vocabularies: No relations among concepts, overlapping –New vocabularies are coming: The Human Brain Project (with FMA) DARPA Digital Human – unifying medical ontologies