2. SOFTWARE INTERFACES Marek Milosz PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 1 HMI - 2

3. Agenda HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 2  Types of software interfaces  Material for interacting  Interaction mechanisms

4. Text-based Interface, Text User Interface Graphical User Interface Types of software interfaces PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 3 HMI - 2

5. Text User Interface PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR)  TUI  The characters displayed on the monitor in line or lines (text)  Semigraphics or pseudographics  Until today: emoticons (SMS, Skype, e-mail,...)  Encoding: ASCII  ASCII art 4 HMI - 2

6. TUI - Emoticons PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR)  :) :-) ;-)  :D :* 5 HMI - 2

7. TUI - ASCII art (1) PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 6 HMI - 2

8. TUI - ASCII art (2) PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 7 HMI - 2

9. Graphical User Interface PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR)  GUI  Uses: graphics objects  User interaction with graphical objects  Methods:  Graphics cursor + Keyboard  Mouse  Touch screen  Light pen ... 8 HMI - 2

10. Interface objects Material for interacting PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 9 HMI - 2

11. Line of text PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR)  Type using a keyboard (physical or virtual)  Command line or text box  Elements of I/O:  Forms  Questions/answers  Command line 10 HMI - 2

12. Menus HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 11  Set of options displayed on the screen  Options visible  less recall - easier to use  rely on recognition so names should be meaningful  Selection by:  numbers, letters, arrow keys  mouse  combination (e.g. mouse plus accelerators)  Options often hierarchically grouped  sensible grouping is needed

13. Forms to fill HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 12  Primarily for data entry or data retrieval  Screen like paper form  Data put in relevant place  Requires  good design  obvious correction facilities

14. Windows HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 13  Areas of the screen that behave as if they were independent:  can contain text or graphics  can be moved or resized  can overlap and obscure each other, or can be laid out next to one another (tiled)  Scrollbars – allow the user to move the contents of the window up and down or from side to side  Title bar – describes the name of the window

15. Icons HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 14  Small picture or image  Represents some object in the interface – often a window or action  Windows can be closed down (iconised)  Icons can be many and various  highly stylized  realistic representations  standarisation in a particular environment  Hard to develop in many industrial cases

16. Icons – standards HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 15

17. Pointers – cursors HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 16  Shows point of input device activity  Texts vs. graphics  Important component WIMP style relies on pointing and selecting things  Uses mouse, trackpad, joystick, trackball, cursor keys or keyboard shortcuts  Wide variety of graphical images

18. Menus – kinds HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 17  Menu Bar at top of screen (normally), menu drags down  pull-down menu - mouse hold and drag down menu  drop-down menu - mouse click reveals menu  fall-down menus - mouse just moves over bar!  Contextual menu appears where you are  pop-up menus - actions for selected object  pie menus - arranged in a circle easier to select item (larger target area) quicker (same distance to any option) … but not widely used!

19. Menus - extras HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 18  Cascading menus  hierarchical menu structure  menu selection opens new menu  and so in ad infinitum  Keyboard accelerators  key combinations - same effect as menu item  two kinds active when menu open – usually first letter active when menu closed – usually Ctrl + letter  usually different in different applications

20. Menus – design issues HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 19  Which kind to use?  What to include in menus?  Words to use (action or description)  How to group items?  How to order items?  Choice of keyboard accelerators (possible conflicts)

21. Buttons HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 20  Individual and isolated regions within a display that can be selected to invoke an action  Special kinds  radio buttons – set of mutually exclusive choices  check boxes – set of non-exclusive choices  Parts of windows/forms

22. Toolbars HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 21  Long lines of icons  Fast access to common actions  Often customizable:  choose which toolbars to see  choose what options are on it

23. Ribbons HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 22  Kind of toolbars  Form of a set of toolbars placed on several tabs  Group and speed using of icons  Mainly used in MS Office

24. Palettes and tear-off menus  Problem: menu not there when you want it!  Solution:  tear-off and pin-up menus – menu ‘tears off’ to become palette  palettes – little windows of actions – shown/hidden via menu option e.g. available shapes in drawing package HMI - 2 23 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR)

25. Dialogue boxes HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 24  Information windows that pop up to inform of an important event or to request information  Dirrerent types: alert, error, informational  Different icons used  Modal windows

26. Dialogue boxes - symbols HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 25

27. Other window objects HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 26  Text field  Choice field  Label  List  …

28. Physical objects for interacting HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 27  Buttons  Devices – controllers  Accelerometer  Cameras  …..

29. Interaction mechanisms PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 28 HMI - 2

30. Direct and indirect interaction HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 29 system interface real system immediate feedback real feedback

31. Initiative HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 30  Who has the initiative? old question–answer–> computer WIMP interface–> user  WIMP exceptions … pre-emptive parts of the interface  Modal dialog boxes  come and won’t go away!  good for errors, essential steps  but use with care

32. Type of interaction mechanisms HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 31  Instructions  Talk, conversation (usually with avatars)  Direct manipulation of objects (virtual or physical, visual objects)  Exploration (virtual or physical environment)

33. Type of interfaces PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR)  Command line (Command Line Interface, CLI)  WIMP (Windows - Icons - Menus - Pointer)  Natural User Interface (NUI)  Voice  Touch  Gesture  Multimodal  Augmented reality 32 HMI - 2

34. Interface: CLI (1) PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR)  Command line  Character cursor - keyboard (including special keys F…)  Specific residues: enter the text fields or CAPCHA 33 HMI - 2

35. Interface: CLI (2) PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 34 HMI - 2

36. Interface: WIMP PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR)  Windows – Icons – Menus – Pointer 35 HMI - 2

37. Interface WIMP – precursor  Xerox Alto  1973  One of the first PC using the Desktop Metaphor PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 36 HMI - 2

38. Xerox Alto – file manager PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 37 HMI - 2

39. Interface WIMP – Windows HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 38  W1.0 – 1985; W3.0 – 1990, W95, W2000  Graphical, 16-bit multi-tasking shell on top of an existing MS-DOS installation

40. Windows 3.0 HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 39  1990

41. Evolution of interfaces HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 40 Codified Strict CLI Metaphor Exploratory GUI Direct Intuitive NUI

42. NUI – Natural User Interface HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 41

43. MUI – Modern User Interface HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 42

44. Voice interface HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 43  Hands-free and Eyes-free  Disadvantage:  Noise  Errors in identifying the command  Snooping (lack of privacy of communication)  User Requirements:  Hands free  I do not need to look  Close to customers  Problems

45. Voice interface - Avatars HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 44

46. Touch interface HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 45  It uses the sense of touch  It can work two ways: input and output  Input:  Touchpad  Touchscreen  Output:  Gloves and clothes making pressure  Special I/O devices using the Braille alphabet

47. Touchpad HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 46  Mouse replacement device  The advantage - built-in and requires no additional work surface as a mouse  The alternative - trackball (inverted mouse) or the trackpoint (joystick)

48. Touchscreen HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 47

49. Touchscreen – gestures (1) HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 48

50. Touchscreen – gestures (2) HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 49

51. Braille devices HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 50

52. Gesture interface HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 51  It uses the human body to transmit commands  Devices:  Gloves with sensors  2D cameras  3D cameras  Cameras with structured light  Motion controllers

53. Cameras with structured light HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 52  Use structural lighting (e.g. points or grid) to obtain information about the environment (3D)  Examples:  Kinect (2010)  EyeTracker  Kinect:  Cloud of points (infrared LED + infrared camera) -> image depth  The image -> an ordinary 2D camera  Microphones (4)  Head rotating by motor (vertical)  Accelerometer

54. Kinect – cloud of points HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 53

55. Gesture interface – applications HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 54  Entertainment  Medical (no touch)  Processing of large number of visual data / graphics / multimedia

56. Augmented reality HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 55  Overlaping the real and virtual images

57. Augmented reality - welding technology training HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 56

58. Groupware Interfaces HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 57  Provide the simultaneous interaction of a large number of participants

59. Brain-Computer interface HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 58  BCI  Recognition signals the brain working and using them to transfer commands to a computer  Electroencephalograph  Threats:  The accuracy of reading and interpreting  The variety of people  Variability of states of a single man  Ethics: reading thoughts, control his thoughts and so on

60. BCI – first experiments HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 59

61. Physical object design HMI - 2 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR) 60  Many constraints:  ergonomic – minimum button size  physical – high-voltage switches are big  legal and safety – high cooker controls  context and environment – easy to clean  aesthetic – must look good  economic – … and not cost too much!  Physical vs. logical state (button: on/off)

62. Thank you for attention Questions? HMI - 2 61 PROfessional network of Master’s degrees in Informatics as a Second Competence – PROMIS (544319-TEMPUS-1-2013-1-FR-TEMPUS-JPCR)