1. Lecture 12 implementation support

2. Today’s Lecture programming tools  levels of services for programmers windowing systems  core support for separate and simultaneous user-system activity programming the application and control of dialogue interaction toolkits  bring programming closer to level of user perception user interface management systems  controls relationship between presentation and functionality

3. Introduction We have spent much effort up to this point considering design and analysis of interactive systems from a relatively abstract perspective. Now we will focus the task of coding the interactive application and how it is structured.

4. Introduction The detailed specification gives the programmer instructions as to what the interactive application must do.  The programmer must translate that into machine executable instructions to say how that will be achieved on the available hardware devices. The objective of the programmer then is to translate down to the level of the software that runs the hardware devices.

5. Quest of this lecture How does HCI affect of the programmer? Advances in coding have elevated programming hardware specific  interaction-technique specific Layers of development tools  windowing systems  interaction toolkits  user interface management systems

6. Elements of windowing systems The first important feature of a windowing system is its ability to provide programmer independence from the specifics of the hardware devices.  A typical workstation will involve some visual display screen, a keyboard and some pointing device, such as a mouse.

7. Windowing Systems.. It is imperative to be able to program an application that will run on a wide range of devices.  The programmer wants to direct commands to an abstract terminal, which understands a more generic language and can be translated to the language of many other specific devices.

8. Windowing Systems.. Besides making the programming task easier, the abstract terminal makes portability of application programs possible.  Only one translation program – or device driver – needs to be written for a particular hardware device and then any application program can access it.

9. Windowing Systems.. A given windowing system will have a fixed generic language for the abstract terminal which is called its imaging model.  The imaging models are sufficient to describe very arbitrary images. For efficiency reasons, specific primitives are used to handle text images, either as specific pixel images or as more generic font definitions.

10. Examples of Imaging Model Pixels  The display screen is represented as a series of columns and rows of points – or pixels – which can be explicitly turned on or off, or given a color. This is a common imaging model for personal computers and is also used by the X windowing system.

11. Examples of Imaging Model Graphical kernel system (GKS)  An international standard which models the screen as a collection of connected segments, each of which is a macro of elementary graphics commands.

12. Examples of Imaging Model Programmer’s hierarchical interface to graphics (PHIGS)  Another international standard, based on GKS but with an extension to model the screen as editable segments.

13. Examples of Imaging Model PostScript  A programming language developed by Adobe Corporation which models the screen as a collection of paths which serve as infinitely thin boundaries or stencils which can be filled in with various colors or textured patterns and images.

14. Imaging Models Though these imaging models were initially defined to provide abstract languages for output only, they can serve at least a limited role for input as well.  For example, the pixel model can be used to interpret input from a mouse in terms of the pixel coordinate system.

15. Elements of Windowing System It would then be the job of the application to process the input event further once it knows where in the image it occurred. The other models above can provide even more expressiveness for the input language.  They can relate the input events to structures that are identifiable by the application program.

16. Sharing Windowing support several separate user tasks simultaneously. Sharing the resources of a single hardware configuration with several copies of an abstract terminal. Each abstract terminal will behave as an independent process and the windowing system will coordinate the control of the concurrent processes.

17. Sharing To ease the programming task again, this coordination of simultaneously active processes can be factored out of the individual applications, so that they can be programmed as if they were to operate in isolation.

18. Sharing The window system must also provide a means of displaying the separate applications.  accomplished by dedicating a region of the display screen to each active abstract terminal. The coordination task then involves resolving display conflicts when the visible screen regions of two abstract terminals overlap.

19. roles of a windowing system

20. Architectures of windowing systems three possible software architectures  all assume device driver is separate  differ in how multiple application management is implemented 1. each application manages all processes  everyone worries about synchronization  reduces portability of applications 2. management role within kernel of operating system  applications tied to operating system 3. management role as separate application maximum portability

21. The client-server architecture

22. X Window System A classic example of a window system based on the client–server architecture is the industry-standard X Window System (Release 11), developed at the Massachusetts Institute of Technology (MIT) in the mid-1980s.  It is based on a pixel-based imaging model and assumes that there is some pointing mechanism available.

23. X Windows X adoption is due to a standard.  X is based on a network protocol which clearly defines the server–client communication. The X Protocol can be implemented on different computers and operating systems, making X more device independent.

24. X Windows It also means that client and server need not even be on the same system in order to communicate to the server.

25. X Windows The X server performs the following tasks:  allows (or denies) access to the display from multiple client applications;  interprets requests from clients to perform screen operations or provide other information;  De-multiplexes the stream of physical input events from the user and passes them to the appropriate client;

26. Tasks by X server  minimizes the traffic along the network by relieving the clients from having to keep track of certain display information, like fonts, in complex data structures that the clients can access by ID numbers.

27. X Windows architecture

28. X Windows architecture (ctd) pixel imaging model with some pointing mechanism X protocol defines server-client communication separate window manager client enforces policies for input/output:  how to change input focus  tiled vs. overlapping windows  inter-client data transfer

29. Programming the Application We now concentrate our attention on programming the actual interactive application, which would correspond to a client in the client–server architecture. Interactive applications are generally user driven in the sense that the action the application takes is determined by the input received from the user.

30. Programming paradigms Two paradigms  read–evaluation loop  Notification based

31. Read Evaluation loop The server sends user inputs as structured events to the client application. As far as the server is concerned, the only importance of the event is the client to which it must be directed. The client application is programmed to read any event passed to it and determine all of the application-specific behavior that results as a response to it

32. Programming the application - 1 read-evaluation loop repeat read-event(myevent) case myevent.type type_1: do type_1 processing type_2: do type_2 processing... type_n: do type_n processing end case end repeat

33. Notification based The main control loop for the event processing does not reside within the application.  A centralized notifier receives events from the window system and filters them to the application program in a way declared by the program.

34. Programming the application - 1 notification-based void main(String[] args) { Menu menu = new Menu(); menu.setOption(“Save”); menu.setOption(“Quit”); menu.setAction(“Save”,mySave) menu.setAction(“Quit”,myQuit)... } int mySave(Event e) { //save the current file } int myQuit(Event e) { //close down }

35. Going with the grain– Design Focus system style affects the interfaces  modal dialogue box easy with event-loop (just have extra read-event loop) hard with notification (need lots of mode flags)  non-modal dialogue box hard with event-loop (very complicated main loop) easy with notification (just add extra handler) beware! if you don’t explicitly design it will just happen implementation should not drive design

36. Using Toolkits A key feature of WIMP interfaces from the user’s perspective is that input and output behaviors are intrinsically linked to independent entities on the display screen.  This creates the illusion that the entities on the screen are the objects of interest – interaction objects we have called them – and that is necessary for the action world of a direct manipulation interface.

37. Using Toolkits  A classic example is the mouse as a pointing device.  The input coming from the hardware device is separate from the output of the mouse cursor on the display screen.  However, since the visual movement of the screen cursor is linked with the physical movement of the mouse device, the user feels as if he is actually moving the visual cursor.

38. Using toolkits Interaction objects  input and output intrinsically linked Toolkits provide this level of abstraction  programming with interaction objects (or  techniques, widgets, gadgets)  promote consistency and generalizability  through similar look and feel  amenable to object-oriented programming movepressreleasemove

39. Interfaces in Java – design Focus Java toolkit – AWT (abstract windowing toolkit) Java classes for buttons, menus, etc. Notification based;  AWT 1.0 – need to subclass basic widgets  AWT 1.1 and beyond -– callback objects Swing toolkit  built on top of AWT – higher level features  uses MVC architecture (see later)

40. User Interface Management Systems (UIMS) UIMS add another level above toolkits  toolkits too difficult for non-programmers concerns of UIMS  conceptual architecture  implementation techniques  support infrastructure non-UIMS terms:  UI development system (UIDS)  UI development environment (UIDE) e.g. Visual Basic

41. UIMS as conceptual architecture separation between application semantics and presentation improves:  portability – runs on different systems  reusability – components reused cutting costs  multiple interfaces – accessing same functionality  customizability – by designer and user

42. UIMS tradition – interface layers / logical components linguistic:lexical/syntactic/semantic Seeheim: Arch/Slinky

43. Seeheim model Presentation Dialogue Control Functionality (application interface) USER APPLICATION switch lexical syntacticsemantic

44. conceptual vs. implementation Seeheim  arose out of implementation experience  but principal contribution is conceptual  concepts part of ‘normal’ UI language … because of Seeheim … … we think differently! e.g. the lower box, the switch needed for implementation but not conceptual

45. Semantic feedback different kinds of feedback:  lexical – movement of mouse  syntactic – menu highlights  semantic – sum of numbers changes semantic feedback often slower  use rapid lexical/syntactic feedback but may need rapid semantic feedback  freehand drawing  highlight trash can or folder when file dragged

46. What’s this?

47. the bypass/switch rapid semantic feedback direct communication between application and presentation but regulated by dialogue control

48. more layers!

49. Arch/Slinky more layers! – distinguishes lexical/physical like a ‘slinky’ spring different layers may be thicker (more important) in different systems or in different components

50. monolithic vs. components Seeheim has big components often easier to use smaller ones  esp. if using object-oriented toolkits Smalltalk used MVC – model–view–controller  model – internal logical state of component  view – how it is rendered on screen  controller – processes user input

51. MVC model - view - controller model view controller

52. MVC issues MVC is largely pipeline model: input  control  model  view  output but in graphical interface  input only has meaning in relation to output e.g. mouse click  need to know what was clicked  controller has to decide what to do with click  but view knows what is shown where! in practice controller ‘talks’ to view  separation not complete

53. PAC model PAC model closer to Seeheim  presentation – manages input and output  abstraction – logical state of component  control – mediates between them manages hierarchy and multiple views  control part of PAC objects communicate PAC cleaner in many ways … but MVC used more in practice (e.g. Java Swing)

54. PAC presentation - abstraction - control abstractionpresentation control AP C AP C AP C AP C

55. Implementation of UIMS Techniques for dialogue controller menu networks state transition diagrams grammar notations event languages declarative languages constraints graphical specification  for most of these see chapter 16 N.B. constraints  instead of what happens say what should be true  used in groupware as well as single user interfaces (ALV - abstraction–link–view) see chapter 16 for more details on several of these

56. Graphical Specification what it is  draw components on screen  set actions with script or links to program in use  with raw programming most popular technique  e.g. Visual Basic, Dreamweaver, Flash local vs. global  hard to ‘see’ the paths through system  focus on what can be seen on one screen

57. The drift of dialogue control internal control (e.g., read-evaluation loop) external control (independent of application semantics or presentation) presentation control (e.g., graphical specification)

58. Implementation Consideration How components in a conceptual architecture can be realized?  Implementations based on the Seeheim model must determine how the separate components of presentation, dialog controller and application interface are realized.

59. Implementation Considerations  Window systems and toolkits provide the separation between application and presentation. The use of callback procedures in notification-based programming is one way to implement the application interface as a notifier.  In the standard X toolkit, these callbacks are directional as it is the duty of the application to register itself with the notifier.

60. Implementation considerations Myers has outlined the various implementation techniques used to specify the dialog controller separately. Which are, Menu networks Grammar notations State transition diagrams Event languages Declarative languages Constraints and Graphical specification

61. Implementation Techniques Menu networks: The communication between application and presentation is modeled as a network of menus and submenus.  The menu is used to embody all possible user inputs at any one point in time.

62. Implementation Techniques Grammar notations: The dialog between application and presentation can be treated as a grammar of actions and responses, and, therefore, described by means of a formal context-free grammar notation, such as BNF (Backus–Naur form)

63. Implementation Techniques State transition diagrams: State transition diagrams can be used as a graphical means of expressing dialog. Event languages: Event languages are similar to grammar notations, except that they can be modified to express directionality and support some semantic feedback.

64. Implementation Techniques Declarative Languages: A declarative approach concentrates more on describing how presentation and application are related. Constraints: Constraints embody dependencies between different values that must always be maintained.

65. Implementation Techniques Graphical specification: These techniques allow the dialog specification to be programmed graphically in terms of the presentation language itself.

66. Summary Levels of programming support tools Windowing systems  device independence  multiple tasks Paradigms for programming the application  read-evaluation loop  notification-based Toolkits  programming interaction objects UIMS  conceptual architectures for separation  techniques for expressing dialogue