BIWST Formal Approaches to Modelling HCI David Duce Oxford Brookes University

BIWST Contents Syndetic modelling (Duke, Barnard, Duce, May) Process algebraic modelling (Bowman, Barnard et al) Verification-guided modelling … (Curzon, Blandford et al)

BIWST Syndetic Modelling David Duke, David Duce (1999). The Formalization of a Cognitive Architecture and its Application to Reasoning About Human Computer Interaction, Formal Aspects of Computing, 11, pp Philip Barnard, Jon May, David Duke, David Duce (2000). Systems, Interactions, and Macrotheory, ACM ToCHI, 7(2), David Duce and David Duke (2001), Syndetic Modelling: Computer Science Meets Cognitive Psychology, Electronic Notes in Theoretical Computer Science, Volume 43, May 2001, Pages Available in ScienceDirect (invited talk at workshop FM-Elsewhere)

BIWST Research problems How does interaction work? How can we build better interactive systems? Interaction = computation + cognition State Interface Interpretation Perception / Action

BIWST Implicit modelling of cognition –Design rationale approaches –Why does a problem occur? –How can it be addressed? Explicit models –Which cognitive model? –How can it be represented? –How can we work with it? Mind the gap

BIWST Bridging the gap Syndetic modelling –Duke, Barnard, Duce and May –Faconti, Bowman and Massink –mathematical model as common language Cognitive model –Interacting Cognitive Subsystems –Barnard,

BIWST Building a syndetic model Mathematical model of ICS Device model Cognitive model of domain user Syndetic model Domain or technology model?

BIWST Syndesis in practice MATIS example –Laurence Nigay, Joelle Coutaz (IMAG - Grenoble) –experimental platform for multi-modal interaction –deictic reference

BIWST The device... interactor MATIS attributes visfields: qnr name value- query content vis current: qnr- current query mouse: seq data- data stream from mouse speech: seq slot- data (& holes) from speech result: name data- outcome of resolving deixis actions artspeak : name value- articulate a data value limselect: data- select a data value fuse- fuse input streams fill- fill in slots on a query form axioms 1speech = X [speak(nm,d)] speech = X^ (nm, d) 2mouse = M [select(d)] mouse = M^ d

BIWST ICS Interacting Cognitive Subsystems, Barnard et al, MRC, Cambridge Highly parallel architecture Control of system wide interactions is decentralised Rich behaviour arises from interaction among multiple subsystems Computational models must capture interactions between mental subsystems at an abstract level

BIWST ICS architecture OBJVISACMPL ARTBS LIM IMPLIC PROP

BIWST Subsystem operation Incoming representations Blending at input array Transformed into output representations Copied into episodic memory... … and revived

BIWST ICS configuration

BIWST Formalising ICS interactor ICS attributes sources: tr P tr stable: P tr input: tr repr _on_: repr tr repr sys coherent: tr tr B buffered: tr config: Config flows: P Flow actions engage: tr tr disengage : tr tr buffer: tr trans

BIWST From principles to axioms axioms 1coherent(t 1, t 2 ) dest(t 1 ) = dest(t 2 ) p, q : repr p on t 1 q on t 2 p q 2t stable s 1, s 2 : sources(t) coherent(s 1, s 2 ) (t = buffered sources(t) stable) 3t config (t stable src(t) {art, lim} s: tr t sources(s)) Etc.

BIWST … the user... interactor MATIS-User MATIS- include the MATIS spec ICS- and the ICS framework actions read : data- observe the MATIS presentation axioms 1 per (read(d)) d in MATIS vis obj:, :obj mpl:, :mpl prop: flows 2 per (select(d)) d on-flow word search d in MATIS 3 per (speak(s)) s on-flow speech

BIWST wordsearch and speech

BIWST … and the consequences. reasoning about interaction MATIS-User per (speak(s)&select(d)) { proof steps } s d specification points to cause of difficulties –inspectable argument critical assumptions –inspectable model theoretical grounds –inspectable theory freedom for change

BIWST Achievements … Syndesis allows reasoning about properties of interaction Chain of reasoning is explicit Points to theoretically grounded reasons Approach is independent of particular cognitive theory –ICS has breadth and depth –wide range of applications –ICS expert system –commensurate level of abstraction to FDTs

BIWST …. limitations Representations are abstract, uninterpreted –properties addressed only through axioms –insight ceases at level of axioms –why might representations be coherent? ICS captured at one level of granularity –but ICS deals with multiple levels Flows and configurations Representations and transformations –duration calculus, time-based process calculi might address but, require more detail than cognitive science can supply require more engagement with mathematical structures

BIWST Prospects - dynamics Dynamic aspects of ICS include –overall configuration and stability of data streams from moment to moment –mental representations as inputs to processes –process (if any) that can draw on contents of image record (buffered) Issues –what representations are in memory? –how are they used in tasks? –how is product of revival related to contents of memory? –how to deal with multiple levels of detail, variable levels of temporal granularity?

BIWST Prospects – mental representations Acknowledge existence of representations carried in flows, but not structure –cognitive theory unclear (modelling perspective) –lack the right mathematics Human visual, auditory systems extensively studied –signal processing theory –ICS assumes that all subsystems based on common architecture –is there a common mathematical model for all subsystems?

BIWST … directions ??? Interaction between representations (blending) suggests wave-like model Analogies with quantum mechanics Electric circuit models (voltage-controlled oscillators, etc) used to model low-level neural behaviours Multiple frequencies processed in parallel in visual system – scale- space models Can different levels be linked by refinement?

BIWST Prospects – social context Safety-critical systems involve human as well as computer agents; need to reason about both Systems increasingly involve groups of users, not just single users Distant future: treatment of emergent group behaviour

BIWST Conclusions (Syndetic modelling) Crystallise our understanding of human information processing through mathematical structures Reveals where understanding incomplete or vague Moderate success in reasoning about flows Need deeper understanding of representations Opening up levels of detail may shed light on utilisation of information in the external world Can we generalise to higher levels of interaction between humans in groups?

BIWST Contents Syndetic modelling (Duke, Barnard, Duce, May) Process algebraic modelling (Bowman, Barnard et al) Verification-guided modelling … (Curzon, Blandford et al)

BIWST Bowman, Barnard et al Li Su, Howard Bowman, Philip Barnard, Brad Wyble (2008). Process algebraic modelling of attentional capture and human electrophysiology in interactive systems, Formal Aspects of Computing, DOI /s Latest in a series of papers Explore process algebra formalism; alternative formalism for describing ICS

BIWST Process algebra Processes connected by communication channels Interact by message exchange along channels Processes can be hierarchically nested LOTOS is one common process algebra formalism; there are many others

BIWST Variant of Attentional Blink (Barnard et al) Subjects were asked to report a word if it referred to a job or profession for which people get paid, e.g. waitress Embedded amongst distractors (background words) that all belonged to same category, e.g. nature words But also stream contained a key distractor item, semantically related to target category, e.g. tourist, vegetarian Serial position that target appeared after key distractor was varied in experiments

BIWST Results Target report accuracy w.r.t lag of target relative to key distractor Key distractor drew attention away from target with a clear temporal profile Diagram from Su et al (2008)

BIWST Two subsystem model Su et al, figure 5. (CLOCK process and control channels omitted) Subsystems perform salience assessments as items pass through pipeline Word composed of six constitutent representations (not letters) – meaning builds up over time

BIWST Attention allocation Can only assess salience at a subsystem when attention is engaged Attention can only be engaged at one subsystem at a time Cannot glance at one item while looking at and scrutinising another When attention engaged, the subsystem is buffered Buffer ensures serial allocation of resources, while items pass concurrently

BIWST Salience assignment Subsystems assign salience based on constituent representations entering Word composed of six constituent representations – ca. 110 ms presentation time used in experiments Semantic meaning builds up over time, not letter by letter Assume 3 constituent time slots (60 ms) required for extraction of useful meaning Numbers consistent with earlier experiments

BIWST LOTOS specification (extract) process SOURCE[device_source,source_implic,tick,debug] : noexit := (* Initialise the delay line with 4 empty constituent representations. *) let vis_dl : Dline = (* Specification omitted *) in Vis[device_source,source_implic,tick,debug](vis_dl) where process Vis[device_source,source_implic,tick,debug](dl:Dline) :noexit := tick?y:Nat!false; (* Synchronise with the global clock at the start of each cycle. *) (( (device_source?ia:Rep; (* Recieve visual stimuli from device and store it in ia (input array). *) exit(ia)) ||| (source_implic!Get(dl); (* Get() returns the last item in the delay line. *) exit(any Rep)) ) (* The above processes are performed in parallel, and when both finish, the sequential operator >> ensures that stimuli in the input array are passed to the next step. *) >> accept ia:Rep in ( [(y mod 20) eq 0] -> (* Update delay line with the new stimuli when time is a multiple of 20. *) (let dl:Dline = Put(ia,Push(dl)) inVis[device_source,source_implic,tick,debug](dl)) [] [(y mod 20) ne 0] -> (* Otherwise, do noting. *) (Vis[device_source,source_implic,tick,debug](dl)) )) endproc

BIWST Contents Syndetic modelling (Duke, Barnard, Duce, May) Process algebraic modelling (Bowman, Barnard et al) Verification-guided modelling … (Curzon, Blandford et al)

BIWST Approach Rimvydas Ruksenas, Jonathan Back, Paul Curzon, Ann Blandford (2009), Verification-guided modelling of salience and cognitive load, Formal Aspects of Computing. DIO /s Higher order formalisation of properties of cognitive architecture (Not ICS) Formalisation of salience and dependency on cognitive load Original formalisation in HOL; most recent paper used SAL (Symbolic Analysis Laboratory)

BIWST Aims Well-defined interfaces use procedural and sensory cues, to increase salience of appropriate actions But cognitive load can influence strength of the cues Formalises relationship between salience and cognitive load revealed by empirical data Remembering to collect original document after making photocopies Remembering to take bank card after balance enquiry Well-designed interfaces increase sensory salience of signals used to cue actions that are frequently forgotten, or performed in wrong sequence Evidence that sensory cues are not always noticed under high workload senarios

BIWST Cognitive principles Non-determinism – any one of several cognitively plausible behaviours might be taken Relevance – given several options, person chooses one that seems relevant to task goals Salience – affects user choices, even though non-deterministic Mental versus physical actions – delay between moment person commits mentally to action and moment when physical action is taken; each physical action modelled is associated with an internal mental action committing to it Pre-determined goals – user engages an interaction with knowledge of task, and task-dependent sub-goals Reactive behaviour – user may react to external stimulus, e.g. flashing light, insert coins in adjacent slot Voluntary task completion – person may decide to terminate interaction Forced task termination – if no apparent action person can take that will help complete task, e.g. ticket machine doesnt sell the right kind of ticket

BIWST SAL Symbolic Analysis Laboratory The heart of SAL is a language, developed in collaboration with Stanford and Berkeley, for specifying concurrent systems in a compositional way. It is supported by a tool suite that includes state of the art symbolic (BDD-based) and bounded (SAT-based) model checkers, an experimental "Witness" model checker, and a unique "infinite" bounded model checker based on SMT solving. Auxiliary tools include a simulator, deadlock checker and an automated test generator. Higher-order language SAL specifications are transition systems Constructs include functions, update, non-deterministic choice and guarded commands

BIWST Cognitive architecture in SAL Main concept is user goals Organised as hierarchical tree, nodes are compound or atomic Atomic goals map to an action A goal, g, is modelled as –guard predicate, specifies when goal enabled, e.g. device prompts –choice models high-level ordering of goals by specifying when goal can be chosen –achieved – predicate specifying main task goal –salience – specifies sensory salience of g –cueing – function, for each goal g, returns strength of atomic goal h as procedural cue for g –load – specifies intrinsic load associated with execution of g –subgoals – data structure specifying subgoals of goal State space consists of input variable in, output variable out and global variables (memory) mem, and env (environment)

BIWST Fire engine dispatch task (from FAC paper)

BIWST Experiment Modelled task and compared to experimental results Errors investigated –Initialisation – clicking on Start next call without prioritising calls –Confirmation 1 – Start next call should only be clicked when both new call ID and confirm priority change have been clicked. Forgetting to click confirm priority change was confirmation error 1. –Mode – constructing route using wrong mechanism – user behaves as if system in the other mode –Termination – clicking get/send route information without selecting a backup unit was considered to be erroneous –Confirmation 2 – required to click on route complete after selecting backup unit

BIWST Error rates (FAC paper)

BIWST Results Refined architecture; taking account of procedural and sensory cues

BIWST Ruksenas et al conclusions Model was refined based on empirical data; needs further work to confirm that the refinements are generic Irrespective of that, led to deeper understanding of the empirical data –Model allowed team to probe data more deeply than would otherwise have been possible –Refinements followed from discussions between cognitive science and formal modelling specialists in the team; relied on cognitive mechanisms and factors known from the literature Formal development of salience and load rules suggested new experimental hypotheses –Salience hierarchy We hypothesize that participants executing a routine procedure will make significantly less errors than participants executing a non-routine procedure (where procedural cues are absent), regardless of whether they possess a high or low level knowledge of the domain. –Sensory salience We hypothesize that the sensory salience of an action only captures attention if the semantic meaning of the object has been encoded or the procedure has been learnt by following spatial mappings. –Cognitive cueing We hypothesize that cognitive cueing is sensitive to both intrinsic and extraneous load providing that the semantic meaning of objects are interpreted correctly... Provides a good example of the cyclic nature of our interdisciplinary research methodology based on the mutual benefits of bridging cognitive science and formal methods in computer science

BIWST Acknowledgements (Syndetic Modelling) Phil Barnard Jon May Giorgio Faconti Howard Bowman Mieke Massink Ann Blandford Amodeus-2 TACIT

BIWST Thank You! Any Questions?

BIWST Overall architecture vis-obj lim-hand copy mpl-art art-speech prop-mpl prop-obj obj-lim obj-prop