1. Université catholique de Louvain (UCL) Belgian Laboratory of Computer-Human Interaction (BCHI) Place des Doyens, 1 B-1348 Louvain-la-Neuve (Belgium) Presented by University of Castilla-La Mancha - Albacete, November 15, 2004 1 Extraordinary User Interfaces Jean Vanderdonckt vanderdonckt@isys.ucl.ac.be Head of BCHI Lab, http://www.isys.ucl.ac.be/bchi

2. University of Castilla-La Mancha - Albacete, November 15, 2004 2 What is an extraordinary user interface? An extraordinary User Interface (UI) –may exhibit untraditional interaction Techniques: drag & drop, fold & drop, pick & drop Styles: mixed reality Devices: head mounted display –but should not necessarily be limited to sophisticated applications You can do that too on existing UIs You can also create extraordinary things on ordinary UIs

3. University of Castilla-La Mancha - Albacete, November 15, 2004 3 Target Applications, Domains Notations and toolsUser Interface Interaction Techniques 2004 TODAY Adapted from [Palanque,2002] No Interaction TechniqueAutomated, batch systemsNothing Character UIsBusiness applicationsScreen definitions Graphical User Interfaces Information systems Entity-relationship Attribute model State-transition diagrams

4. University of Castilla-La Mancha - Albacete, November 15, 2004 4 Graphical User Interfaces Design once, use many UsiXML = USer Interface eXtensible Markup Language –See http://www.usixml.orghttp://www.usixml.org –FormiXML: forms editor for Java applications –FlashiXML: rendering engine in SWF

5. University of Castilla-La Mancha - Albacete, November 15, 2004 5 Graphical User Interfaces Usual interaction technique –Drag & drop New interaction techniques –Fold & drop –Pick & drop [Dragicevic, 2004] [Rekimoto, 2001] http://liihs.irit.fr/dragice/foldndrop/index.html

6. University of Castilla-La Mancha - Albacete, November 15, 2004 6 Target Applications, Domains Notations and toolsUser Interface Interaction Technique 2004 TODAY Adapted from [Palanque,2002] 2005 No Interaction TechniqueAutomated, batch systemsNothing Character UIsBusiness applicationsScreen definitions Graphical User Interfaces Information systems Entity-relationship Attribute model State-transition diagrams Multi-platform User Interfaces Web, Java applications Task model, context model, UML,…

7. University of Castilla-La Mancha - Albacete, November 15, 2004 7 The situation Multiple computing platforms –Until 2005, the average user will be confronted with at least 5 computing platforms [Forrester,1999]

8. University of Castilla-La Mancha - Albacete, November 15, 2004 8 The situation Home Bookmarks Yes No Sports Options Back ----Home Page-----

9. University of Castilla-La Mancha - Albacete, November 15, 2004 9 The problem To develop user interfaces (UIs) simulta- neously for multiple contexts of use A context of use = triple –User –Computing platform –Surrounding environment Organisation Socio-psychological factors

10. University of Castilla-La Mancha - Albacete, November 15, 2004 10 The problem Why is it difficult? –For the designer: Multiplicity of contexts of use No systematic design method –For the user: Poor usability of resulting UI UI not adapted to the genuine context of use –For the developer: Increase of development and maintenance costs Increasing development complexity Little of no factoring out of common elements

11. University of Castilla-La Mancha - Albacete, November 15, 2004 11 The problem Why should it be systematic? –Some consider it noble to have a method –Other consider it noble not to have a method –Not to have a method is bad –But to stop entirely at any method is worse still –One should at first observe rules severely, then change them in an intelligent way –The aim of possessing method is to seem finally as if one had no method [Lao Tch’ai Tche, many many years ago]

12. University of Castilla-La Mancha - Albacete, November 15, 2004 12 Mono-platform UI Traditional approach –Visual development

13. University of Castilla-La Mancha - Albacete, November 15, 2004 13 Mono-platform UI Advantages of traditional approach –UI is graphical by nature –A UI prototype can be Rapidly produced Easily modified Visually demonstrated

14. University of Castilla-La Mancha - Albacete, November 15, 2004 14 Mono-platform UI Shortcomings of traditional approach –No structured method for developing a UI –All the knowledge remain implicit –Modification can lead to unstructured design –Little or no tool support for iterative design Selection of widgets can be inappropriate UI Layout can be tedious, repetitive Problem of the spaghetti of callbacks –Early prototyping is hard to achieve and time consuming –Limited reusability (throw away)

15. University of Castilla-La Mancha - Albacete, November 15, 2004 15 Mono-platform UI Shortcomings of traditional approach –Limited use of UI builder Table with dynamic data Gantt chart Direct manipulation Desired interfaceObtained interface Menu bar and pull-down menus Toolbar Scrollbars [Szekely,1994]

16. University of Castilla-La Mancha - Albacete, November 15, 2004 16 Mono-platform UI Interface builders cannot build their own UI [Szekely,1994]

17. University of Castilla-La Mancha - Albacete, November 15, 2004 17 Mono-platform UI Model-based approach –Models: explicitly capture knowledge about UI and Interactive Applications with appropriate abstractions –Method: structures the definition and use of underlying models in a stage-wise approach –Supporting tools: support the use of the method by providing tools for models and their related operations. Ideally, one model should be supported by at least one tool

18. University of Castilla-La Mancha - Albacete, November 15, 2004 18 Mono-platform UI Goal: to integrate all three facets Method Model 1 Model … Model n Models Model Interface 1 Tools

19. University of Castilla-La Mancha - Albacete, November 15, 2004 19 Multi-platform Plastic UI: Virtual keyboard [Grolaux, 2001]

20. University of Castilla-La Mancha - Albacete, November 15, 2004 20 Multi-platform Plastic UI: FlexClock [Grolaux, Van Roy, Vanderdonckt, 2002] See http://www.info.ucl.ac.be/people/ned/flexclock/

21. University of Castilla-La Mancha - Albacete, November 15, 2004 21 Multi-platform Plastic UI: adaptable bounded value See http://www.isys.ucl.ac.be/bchi/research/adaptable/sizedemo.htm

22. University of Castilla-La Mancha - Albacete, November 15, 2004 22 Multi-platform for Emergency [Amouh et al.,2004]

23. University of Castilla-La Mancha - Albacete, November 15, 2004 23 Multi-platform for Emergency Three platforms –Pocket PC –Desktop PC –Wall Screens See http://www.tele.ucl.ac.be/PROJECTS/ARTHUR/

24. University of Castilla-La Mancha - Albacete, November 15, 2004 24 Multi-platform for Emergency Model and method –Design the reference screen first –Refine the others screens later By applying graceful degradation By applying transformation techniques See http://www.polymedis.com

25. University of Castilla-La Mancha - Albacete, November 15, 2004 25 Graceful degradation [Florins & Vanderdonckt,2004]

26. University of Castilla-La Mancha - Albacete, November 15, 2004 26 Transformation rules

27. University of Castilla-La Mancha - Albacete, November 15, 2004 27 Transformation rules See http://www.isys.ucl.ac.be/bchi/publications/2004/Florins-IUI2004.pdf

28. University of Castilla-La Mancha - Albacete, November 15, 2004 28 Transformation rules

29. University of Castilla-La Mancha - Albacete, November 15, 2004 29 Target Applications, Domains Notations and toolsUser Interface Interaction Technique 2004 TODAY Adapted from [Palanque,2002] 2005 No Interaction TechniqueAutomated, batch systemsNothing Character UIsBusiness applicationsScreen definitions Graphical User Interfaces Information systems Entity-relationship Attribute model State-transition diagrams Multi-platform User Interfaces Web, Java applications Task model, context model, UML,… Virtual Reality User Interfaces 3D Applications Scene model

30. University of Castilla-La Mancha - Albacete, November 15, 2004 30 Virtualisation of UIs Going from 2D to 3D –Mapping widgets in 2D to 3D [Vanderdonckt, Bouillon, Chow, 2004] See http://www.isys.ucl.ac.be/bchi/publications/2004/Vanderdonckt-W3D2004.pdf

31. University of Castilla-La Mancha - Albacete, November 15, 2004 31 Virtualisation of UIs Graphical rendering in 3D of UsiXML –Virtual world [Molina, 2004]

32. University of Castilla-La Mancha - Albacete, November 15, 2004 32 Target Applications, Domains Notations and toolsUser Interface Interaction Technique 2004 TODAY Adapted from [Palanque,2002] 2005 No Interaction TechniqueAutomated, batch systemsNothing Character UIsBusiness applicationsScreen definitions Graphical User Interfaces Information systems Entity-relationship Attribute model State-transition diagrams Multi-platform User Interfaces Web, Java applications Task model, context model, UML,… Virtual Reality User Interfaces 3D Applications Scene model Mixed Reality User Interfaces Command and control systems, games World models

33. University of Castilla-La Mancha - Albacete, November 15, 2004 33 Mixed reality Mixed reality = real world + digital world See http://urbicande.tele.ucl.ac.be

34. University of Castilla-La Mancha - Albacete, November 15, 2004 34 Mixed reality for games See http://www.alterface.com

35. University of Castilla-La Mancha - Albacete, November 15, 2004 35 No representation for tasks in real or virtual world tasks independent of user interaction e.g. tracking object of interest various contexts of use in Mixed Reality Systems i.e. changing point-of-view Adaptive IS composition based on focus of interest Mixed reality in surgery [Trevisan, Gemo, Vanderdonckt, Macq, 2004]

36. University of Castilla-La Mancha - Albacete, November 15, 2004 36 Composing the MIS Real CIO Patient head Virtual CIOs Anatomic organs Virtual CIOs Annotations Primary object of interest

37. University of Castilla-La Mancha - Albacete, November 15, 2004 37 No overlapping annotations for readability Highest priority of virtual object (3) Medium priority of real object (2) Low priority of background (1) Annotations placed from peripheral to central zones Other objects as part of Background Annotation placement strategy

38. University of Castilla-La Mancha - Albacete, November 15, 2004 38 Context-aware Multimodal Annotation Rendering (a) Background context (b) Background/Real Object context (c) Virtual Object context - what information to display - annotation positions - modality switch User focus controls adaptive presentation management

39. University of Castilla-La Mancha - Albacete, November 15, 2004 39 A context model describes all the entities that may influence carrying out the interactive task of user with the intended UI Environment model - such attributes may be physical (e.g., lighting conditions), psychological (e.g., level of stress), and organization (e.g., location and role definition in the organization chart). Platform model – couple of software/hardware, such atributes may be screen resolution/size User model – stereotypes, sub-stereotypes (profiles) Context model in mixed reality [Trevisan, Gemo, Vanderdonckt, Macq, 2004]

40. University of Castilla-La Mancha - Albacete, November 15, 2004 40 Plasticity Domains in the MIS Plasticity thresholds and context boundary values Inter-context continuity Perceptive, cognitive, functional intra-context continuity and inter- context continuity Intra-context / inter-context frequency and frequency switch Refined Metrics for Plastic MR systems Plasticity analysis and usability evaluation [Trevisan, Gemo, Vanderdonckt, Macq, 2004]

41. Target Applications, Domains Notations and toolsUser Interface Interaction Technique The Invisible UI No more programming? No more bugs? All Applications 2020 2004 TODAY 2006 Adapted from [Palanque,2002] 2005 No Interaction TechniqueAutomated, batch systemsNothing Character UIsBusiness applicationsScreen definitions Graphical User Interfaces Information systems Entity-relationship Attribute model State-transition diagrams Multi-platform User Interfaces Web, Java applications Task model, context model, UML,… Virtual Reality User Interfaces 3D Applications Scene model Mixed Reality User Interfaces Command and control systems, games World models Tangible UIs Embodied UIs Physical models Embarked syst

42. University of Castilla-La Mancha - Albacete, November 15, 2004 42 Thank you very muchfor your attention and see you in 2020 For more information and free downloading, visit our web site at: http://www.isys.ucl.ac.be/bchi

43. University of Castilla-La Mancha - Albacete, November 15, 2004 43 Special thanks to Pascual González López María Dolores Lozano Pérez Víctor Manuel López Jaquero José Pascual Molina Massó Francisco Montero Simarro