1. BIWST 20091 Formal Approaches to Modelling HCI David Duce Oxford Brookes University daduce@brookes.ac.uk

2. BIWST 20092 Contents Syndetic modelling (Duke, Barnard, Duce, May) Process algebraic modelling (Bowman, Barnard et al) Verification-guided modelling … (Curzon, Blandford et al)

3. BIWST 20093 Syndetic Modelling David Duke, David Duce (1999). The Formalization of a Cognitive Architecture and its Application to Reasoning About Human Computer Interaction, Formal Aspects of Computing, 11, pp. 665-689 Philip Barnard, Jon May, David Duke, David Duce (2000). Systems, Interactions, and Macrotheory, ACM ToCHI, 7(2), 222-262 David Duce and David Duke (2001), Syndetic Modelling: Computer Science Meets Cognitive Psychology, Electronic Notes in Theoretical Computer Science, Volume 43, May 2001, Pages 50-74. Available in ScienceDirect (invited talk at workshop FM-Elsewhere)

4. BIWST 20094 Research problems How does interaction work? How can we build better interactive systems? Interaction = computation + cognition State Interface Interpretation Perception / Action

5. BIWST 20095 Implicit modelling of cognition –Design rationale approaches –Why does a problem occur? –How can it be addressed? Explicit models –Which cognitive model? –How can it be represented? –How can we work with it? Mind the gap

6. BIWST 20096 Bridging the gap Syndetic modelling –Duke, Barnard, Duce and May –Faconti, Bowman and Massink –mathematical model as common language Cognitive model –Interacting Cognitive Subsystems –Barnard, 1979 -

7. BIWST 20097 Building a syndetic model Mathematical model of ICS Device model Cognitive model of domain user Syndetic model Domain or technology model?

8. BIWST 20098 Syndesis in practice MATIS example –Laurence Nigay, Joelle Coutaz (IMAG - Grenoble) –experimental platform for multi-modal interaction –deictic reference

9. BIWST 20099 The device... interactor MATIS attributes visfields: qnr name value- query content vis current: qnr- current query mouse: seq data- data stream from mouse speech: seq slot- data (& holes) from speech result: name data- outcome of resolving deixis actions artspeak : name value- articulate a data value limselect: data- select a data value fuse- fuse input streams fill- fill in slots on a query form axioms 1speech = X [speak(nm,d)] speech = X^ (nm, d) 2mouse = M [select(d)] mouse = M^ d

10. BIWST 200910 ICS Interacting Cognitive Subsystems, Barnard et al, MRC, Cambridge Highly parallel architecture Control of system wide interactions is decentralised Rich behaviour arises from interaction among multiple subsystems Computational models must capture interactions between mental subsystems at an abstract level

11. BIWST 200911 ICS architecture OBJVISACMPL ARTBS LIM IMPLIC PROP

12. BIWST 200912 Subsystem operation Incoming representations Blending at input array Transformed into output representations Copied into episodic memory... … and revived

13. BIWST 200913 ICS configuration

14. BIWST 200914 Formalising ICS interactor ICS attributes sources: tr P tr stable: P tr input: tr repr _on_: repr tr _@_: repr sys coherent: tr tr B buffered: tr config: Config flows: P Flow actions engage: tr tr disengage : tr tr buffer: tr trans

15. BIWST 200915 From principles to axioms axioms 1coherent(t 1, t 2 ) dest(t 1 ) = dest(t 2 ) p, q : repr p on t 1 q on t 2 p q 2t stable s 1, s 2 : sources(t) coherent(s 1, s 2 ) (t = buffered sources(t) stable) 3t config (t stable src(t) {art, lim} s: tr t sources(s)) Etc.

16. BIWST 200916 … the user... interactor MATIS-User MATIS- include the MATIS spec ICS- and the ICS framework actions read : data- observe the MATIS presentation axioms 1 per (read(d)) d in MATIS vis obj:, :obj mpl:, :mpl prop: flows 2 per (select(d)) d on-flow word search d in MATIS 3 per (speak(s)) s on-flow speech

17. BIWST 200917 wordsearch and speech

18. BIWST 200918 … and the consequences. reasoning about interaction MATIS-User per (speak(s)&select(d)) { proof steps } s d specification points to cause of difficulties –inspectable argument critical assumptions –inspectable model theoretical grounds –inspectable theory freedom for change

19. BIWST 200919 Achievements … Syndesis allows reasoning about properties of interaction Chain of reasoning is explicit Points to theoretically grounded reasons Approach is independent of particular cognitive theory –ICS has breadth and depth –wide range of applications –ICS expert system –commensurate level of abstraction to FDTs

20. BIWST 200920 …. limitations Representations are abstract, uninterpreted –properties addressed only through axioms –insight ceases at level of axioms –why might representations be coherent? ICS captured at one level of granularity –but ICS deals with multiple levels Flows and configurations Representations and transformations –duration calculus, time-based process calculi might address but, require more detail than cognitive science can supply require more engagement with mathematical structures

21. BIWST 200921 Prospects - dynamics Dynamic aspects of ICS include –overall configuration and stability of data streams from moment to moment –mental representations as inputs to processes –process (if any) that can draw on contents of image record (buffered) Issues –what representations are in memory? –how are they used in tasks? –how is product of revival related to contents of memory? –how to deal with multiple levels of detail, variable levels of temporal granularity?

22. BIWST 200922 Prospects – mental representations Acknowledge existence of representations carried in flows, but not structure –cognitive theory unclear (modelling perspective) –lack the right mathematics Human visual, auditory systems extensively studied –signal processing theory –ICS assumes that all subsystems based on common architecture –is there a common mathematical model for all subsystems?

23. BIWST 200923 … directions ??? Interaction between representations (blending) suggests wave-like model Analogies with quantum mechanics Electric circuit models (voltage-controlled oscillators, etc) used to model low-level neural behaviours Multiple frequencies processed in parallel in visual system – scale- space models Can different levels be linked by refinement?

24. BIWST 200924 Prospects – social context Safety-critical systems involve human as well as computer agents; need to reason about both Systems increasingly involve groups of users, not just single users Distant future: treatment of emergent group behaviour

25. BIWST 200925 Conclusions (Syndetic modelling) Crystallise our understanding of human information processing through mathematical structures Reveals where understanding incomplete or vague Moderate success in reasoning about flows Need deeper understanding of representations Opening up levels of detail may shed light on utilisation of information in the external world Can we generalise to higher levels of interaction between humans in groups?

26. BIWST 200926 Contents Syndetic modelling (Duke, Barnard, Duce, May) Process algebraic modelling (Bowman, Barnard et al) Verification-guided modelling … (Curzon, Blandford et al)

27. BIWST 200927 Bowman, Barnard et al Li Su, Howard Bowman, Philip Barnard, Brad Wyble (2008). Process algebraic modelling of attentional capture and human electrophysiology in interactive systems, Formal Aspects of Computing, DOI 10.1007/s00165-008-0094-3 Latest in a series of papers Explore process algebra formalism; alternative formalism for describing ICS

28. BIWST 200928 Process algebra Processes connected by communication channels Interact by message exchange along channels Processes can be hierarchically nested LOTOS is one common process algebra formalism; there are many others

29. BIWST 200929 Variant of Attentional Blink (Barnard et al) Subjects were asked to report a word if it referred to a job or profession for which people get paid, e.g. waitress Embedded amongst distractors (background words) that all belonged to same category, e.g. nature words But also stream contained a key distractor item, semantically related to target category, e.g. tourist, vegetarian Serial position that target appeared after key distractor was varied in experiments

30. BIWST 200930 Results Target report accuracy w.r.t lag of target relative to key distractor Key distractor drew attention away from target with a clear temporal profile Diagram from Su et al (2008)

31. BIWST 200931 Two subsystem model Su et al, figure 5. (CLOCK process and control channels omitted) Subsystems perform salience assessments as items pass through pipeline Word composed of six constitutent representations (not letters) – meaning builds up over time

32. BIWST 200932 Attention allocation Can only assess salience at a subsystem when attention is engaged Attention can only be engaged at one subsystem at a time Cannot glance at one item while looking at and scrutinising another When attention engaged, the subsystem is buffered Buffer ensures serial allocation of resources, while items pass concurrently

33. BIWST 200933 Salience assignment Subsystems assign salience based on constituent representations entering Word composed of six constituent representations – ca. 110 ms presentation time used in experiments Semantic meaning builds up over time, not letter by letter Assume 3 constituent time slots (60 ms) required for extraction of useful meaning Numbers consistent with earlier experiments

34. BIWST 200934 LOTOS specification (extract) process SOURCE[device_source,source_implic,tick,debug] : noexit := (* Initialise the delay line with 4 empty constituent representations. *) let vis_dl : Dline = (* Specification omitted *) in Vis[device_source,source_implic,tick,debug](vis_dl) where process Vis[device_source,source_implic,tick,debug](dl:Dline) :noexit := tick?y:Nat!false; (* Synchronise with the global clock at the start of each cycle. *) (( (device_source?ia:Rep; (* Recieve visual stimuli from device and store it in ia (input array). *) exit(ia)) ||| (source_implic!Get(dl); (* Get() returns the last item in the delay line. *) exit(any Rep)) ) (* The above processes are performed in parallel, and when both finish, the sequential operator >> ensures that stimuli in the input array are passed to the next step. *) >> accept ia:Rep in ( [(y mod 20) eq 0] -> (* Update delay line with the new stimuli when time is a multiple of 20. *) (let dl:Dline = Put(ia,Push(dl)) inVis[device_source,source_implic,tick,debug](dl)) [] [(y mod 20) ne 0] -> (* Otherwise, do noting. *) (Vis[device_source,source_implic,tick,debug](dl)) )) endproc

35. BIWST 200935 Contents Syndetic modelling (Duke, Barnard, Duce, May) Process algebraic modelling (Bowman, Barnard et al) Verification-guided modelling … (Curzon, Blandford et al)

36. BIWST 200936 Approach Rimvydas Ruksenas, Jonathan Back, Paul Curzon, Ann Blandford (2009), Verification-guided modelling of salience and cognitive load, Formal Aspects of Computing. DIO 10.1007/s00165-009-0102-7 Higher order formalisation of properties of cognitive architecture (Not ICS) Formalisation of salience and dependency on cognitive load Original formalisation in HOL; most recent paper used SAL (Symbolic Analysis Laboratory)

37. BIWST 200937 Aims Well-defined interfaces use procedural and sensory cues, to increase salience of appropriate actions But cognitive load can influence strength of the cues Formalises relationship between salience and cognitive load revealed by empirical data Remembering to collect original document after making photocopies Remembering to take bank card after balance enquiry Well-designed interfaces increase sensory salience of signals used to cue actions that are frequently forgotten, or performed in wrong sequence Evidence that sensory cues are not always noticed under high workload senarios

38. BIWST 200938 Cognitive principles Non-determinism – any one of several cognitively plausible behaviours might be taken Relevance – given several options, person chooses one that seems relevant to task goals Salience – affects user choices, even though non-deterministic Mental versus physical actions – delay between moment person commits mentally to action and moment when physical action is taken; each physical action modelled is associated with an internal mental action committing to it Pre-determined goals – user engages an interaction with knowledge of task, and task-dependent sub-goals Reactive behaviour – user may react to external stimulus, e.g. flashing light, insert coins in adjacent slot Voluntary task completion – person may decide to terminate interaction Forced task termination – if no apparent action person can take that will help complete task, e.g. ticket machine doesnt sell the right kind of ticket

39. BIWST 200939 SAL Symbolic Analysis Laboratory The heart of SAL is a language, developed in collaboration with Stanford and Berkeley, for specifying concurrent systems in a compositional way. It is supported by a tool suite that includes state of the art symbolic (BDD-based) and bounded (SAT-based) model checkers, an experimental "Witness" model checker, and a unique "infinite" bounded model checker based on SMT solving. Auxiliary tools include a simulator, deadlock checker and an automated test generator. Higher-order language SAL specifications are transition systems Constructs include functions, update, non-deterministic choice and guarded commands

40. BIWST 200940 Cognitive architecture in SAL Main concept is user goals Organised as hierarchical tree, nodes are compound or atomic Atomic goals map to an action A goal, g, is modelled as –guard predicate, specifies when goal enabled, e.g. device prompts –choice models high-level ordering of goals by specifying when goal can be chosen –achieved – predicate specifying main task goal –salience – specifies sensory salience of g –cueing – function, for each goal g, returns strength of atomic goal h as procedural cue for g –load – specifies intrinsic load associated with execution of g –subgoals – data structure specifying subgoals of goal State space consists of input variable in, output variable out and global variables (memory) mem, and env (environment)

41. BIWST 200941 Fire engine dispatch task (from FAC paper)

42. BIWST 200942 Experiment Modelled task and compared to experimental results Errors investigated –Initialisation – clicking on Start next call without prioritising calls –Confirmation 1 – Start next call should only be clicked when both new call ID and confirm priority change have been clicked. Forgetting to click confirm priority change was confirmation error 1. –Mode – constructing route using wrong mechanism – user behaves as if system in the other mode –Termination – clicking get/send route information without selecting a backup unit was considered to be erroneous –Confirmation 2 – required to click on route complete after selecting backup unit

43. BIWST 200943 Error rates (FAC paper)

44. BIWST 200944 Results Refined architecture; taking account of procedural and sensory cues

45. BIWST 200945 Ruksenas et al conclusions Model was refined based on empirical data; needs further work to confirm that the refinements are generic Irrespective of that, led to deeper understanding of the empirical data –Model allowed team to probe data more deeply than would otherwise have been possible –Refinements followed from discussions between cognitive science and formal modelling specialists in the team; relied on cognitive mechanisms and factors known from the literature Formal development of salience and load rules suggested new experimental hypotheses –Salience hierarchy We hypothesize that participants executing a routine procedure will make significantly less errors than participants executing a non-routine procedure (where procedural cues are absent), regardless of whether they possess a high or low level knowledge of the domain. –Sensory salience We hypothesize that the sensory salience of an action only captures attention if the semantic meaning of the object has been encoded or the procedure has been learnt by following spatial mappings. –Cognitive cueing We hypothesize that cognitive cueing is sensitive to both intrinsic and extraneous load providing that the semantic meaning of objects are interpreted correctly... Provides a good example of the cyclic nature of our interdisciplinary research methodology based on the mutual benefits of bridging cognitive science and formal methods in computer science

46. BIWST 200946 Acknowledgements (Syndetic Modelling) Phil Barnard Jon May Giorgio Faconti Howard Bowman Mieke Massink Ann Blandford Amodeus-2 TACIT

47. BIWST 200947 Thank You! Any Questions?

48. BIWST 200948 Overall architecture vis-obj lim-hand copy mpl-art art-speech prop-mpl prop-obj obj-lim obj-prop