1 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 An Intelligent Editor for Multi-Presentation User Interfaces Benoît Collignon 1, Jean Vanderdonckt 1, Gaëlle Calvary 2 1 Université catholique de Louvain (UCL) Louvain School of Management (LSM) Belgian Laboratory of Computer-Human Interaction (BCHI) 2 Université Joseph Fourier – Grenoble I, Laboratoire LIG 385, rue de la Bibliothèque BP 53 - F Grenoble Cedex 9 (France)

2 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Introduction The problem of designing multi-target user interfaces –Context of use = {user, computing platform, physical environment} –Multiple contexts of use => multiple variations of users, platforms, and environments Independently Simultaneously Notion of plasticity –Plasticity = user interface ability to change according to a change of the context of use while preserving predefined usability properties Context-aware adaptation = mere adaptation depending on context changes Plasticity = more than context-aware adaptation since usability should be guaranteed at a minimum level of satisfaction To address the problem, we combine –Model-driven engineering of user interfaces –Plasticity as a mean to address context-aware adaptation

3 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Multiplicity of contexts of use LocationRoleDeviceExperience SportingMultimedia Travel programme WorkingTravel booking site Powerful interface for complex operations TravellingBooking notification Everywhere connectivity for simple data exchange FamilyTV is multi-media family device #1

4 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Example of a Plastic User interface Property of plasticity = adaptation to the context of use while preserving predefined usability properties of interest [Grolaux et al.,2002]

5 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Example of a multi-target UI Traditional approach: Atomica HTML/Java UI code Target 1: Web terminal Target 2: Personal computer Visual C++ UI code Target 3: Pocket PC Mobile Visual Basic UI

6 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Related work Some solutions to address this problem: –The mediator solution (model-based approach) [Eisenstein et al.,2001]

7 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Related work Some solutions to address this problem: –Multiple variations based on a model (here, a platform model) [Märtin et al.,1990]

8 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Related work Some solutions to address this problem: –Adaptive UI layout: widgets are (un)displayed and laid out according to resolution [Plomp & Keranen,2002]

9 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Related work Some solutions to address this problem: –Artistic resizing : widgets have different variations according to resolution [Dragicevic et al.,2005] w h wLwL hLhL yLyL xLxL r hBhB wBwB  x L = (w-w L ) / 2  y L = (h-h L ) / 2  w L = 20  h L = 10  w B = 5  h B = 5  r = 20

10 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Our solution Combine model-driven engineering and plasticity domains

11 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Case study Case study: multi-target plastic UI for date & time (FlexClock) [Grolaux et al.,2002]

12 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Case study Dialogue modeled through a Moore machine –16 alternative variations –6 resizing operations Observations –Visualization is hard to achieve

13 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

14 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

15 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

16 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

17 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

18 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

19 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR W1 HH:MM W2 HH:MM:SS W3 HH:MM JJ/MM W4 HH:MM:SS JJ/MM/AAAA W5 HH:MM:SS DAY JJ/MM/AAAA W6 HH:MM:SS DAY JJ MONTH AAAA W7  W8 HH:MM  JJ/MM W9 HH:MM:SS  JJ/MM/AAAA W10 HH:MM:SS  DAY JJ/MM/AAAA W11 HH:MM:SS  DAY JJ MONTH AAAA W13  HH:MM:SS DAY JJ/MM/AAAA W12  HH:MM:SS JJ/MM/AAAA W14  HH:MM:SS DAY JJ MONTH AAAA W15  HH:MM:SS DAY JJ MONTH AAAA CALENDAR W16 HH:MM:SS  DAY JJ MONTH AAAA CALENDAR

20 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Case study Dialogue modeled through a Mealy machine W2W1 W2 W1 2 / / / / / / / / 16 8

21 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Case study Main idea: design one UI variation for one context at a time, connect related variations afterwards Horizontal screen resolution Vertical screen resolution Plasticity domain

22 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Case study Main idea: design one UI variation for one context at a time, connect related variations afterwards

23 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Tool support: PlastiXML Graphical editor for plasticity domains –Direct manipulation of plasticity domains (here, rectangles) –Assignment of a UI variation to each plasticity domain –Automated generation of UsiXML for Layout Navigation between variations <ScreenSizeAspect id="scrd1" shape="rectangle" corners="{(80,10);(80,30);(ScreenSizeXLimit,10); (ScreenSizeXLimit,30)}" allowedOperations="{vertical shrinkage}"/> <ScreenSizeAspect id="scrd2" shape="polygon" corners="{(80,30);(ScreenSizeXLimit,30); (ScreenSizeXLimit,60);(100,60); (100,ScreenSizeYLimit); (80,ScreenSizeYLimit)}" allowedOperations="{vertical shrinkage}"/>

24 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Scenario First plasticity domain

25 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Scenario Second plasticity domain

26 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Scenario Third plasticity domain

27 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Running applications Example #1: country selector

28 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Running applications Example #2: multi-presentation medical teaching application

29 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Running applications Example #3: multi-presentation simple information systems

30 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Conclusion (1/2) Advantages –One UI variation is assigned to one context (here, a resolution) at a time –Connect related variations afterwards –Visual design of connected variations insted of separate (unrelated) design –Automated generation of UsiXML specifications Presentation Navigation –Mealy machine –Moore machine Limitations –Apart from copy/paste, no direct reuse of components from one UI variation to another –No factoring out of common components Possible extension –Generate a default UI variation to feed the various resolutions to be supported –Edit afterwards Generalization needed

31 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Conclusion (2/2) Feedback from designers –Like the visual, colourful design aspects –Dislike the modelling aspects (fortunately generated by software) –Limited to GUIs for the moment Feedback from users –Adaptation always induce some user disruption –No explanation of the adaptation But would it be better is explanation is provided

32 ACM SAC’2008 – Fortaleza, Ceará, Brazil, March 16-20, 2008 Thank you very much for your attention For more information and downloading, User Interface eXtensible Markup Language European network on Multimodal UIs Special thanks to all members of the team!