1. Method and Application of Protocol Analysis Anguo Dong Computer Science Department, University of Calgary

2. Topics covered What are protocol and protocol analysis History of protocol analysis Three types of protocol analysis and their applications A comprehensive protocol analysis software and WBG (Web Behavior Graph)

3. What are protocol and protocol analysis? Definition 1: Protocols are participants' accounts of their thought processes during the performance of a specific task [4]. Definition 2: Protocol analysis is a method of inferring how humans reason and solve problems, using the protocols collected as data[1].

4. Topics Covered What are protocol and protocol analysis History of protocol analysis Three types of protocol analysis and their applications A comprehensive protocol analysis software and WBG (Web Behavior Graph)

5. Topics Covered What are protocol and protocol analysis History of protocol analysis Three types of protocol analysis and their applications A comprehensive protocol analysis software and WBG (Web Behavior Graph)

6. Three types of protocol analysis and their applications Verbal protocol analysis Discrete action protocol analysis Eye-movement protocol analysis

7. Verbal Protocol Analysis (I) Think out aloud -- obtaining data from verbal utterances Two basic approaches -- the concurrent or “think aloud” method and the retrospective method Time delay occurring in retrospective protocol

8. Verbal Protocol Analysis (II) Summary of the steps of verbal protocol analysis 1. Taping 2. Transcription 3. Code development or encoding vocabulary 4. Segmenting and encoding process 5. Analysis

9. Verbal Protocol Analysis (III) … I want to store the important phone number 210-1234 in memory. Em, First, I need to press the mem function key. Then press the number 2101234 in sequence. Ok, next press the mem function key again. Finally, enter the memory location. I will choose 6 because the memory location has to be between 0-9 and the first 6 have been occupied. …” After censoring the tape, the following transcriptions are produced.

10. Verbal Protocol Analysis (IV) 11. … I want to store the important phone number 210-1234 in memory. Store(2101234, memory) 12. Em, First, I need to press the mem function key. Press(mem) 13. Then press the number 2101234 in sequence. Press(1), Press(2), …, Press(4) 14. Ok, next press the mem function key again. Press(mem) 15. Finally, enter the memory location. I will choose 6 Goal(2101234, memLoc) 16. because the memory location has to be between 0-9 and the first 6 have been occupied Because[Between(memL oc; 0, 9) & Occupied(0, 5)]  Enter(6)

11. Three types of protocol analysis and their applications Verbal protocol analysis Discrete action protocol analysis Eye-movement protocol analysis

12. Discrete Action Protocol Analysis Discrete Action Protocol Analysis collects the keystrokes, mouse clicks, or other actions that people exhibits as they engage in interactive behavior [3]. ACT-PRO -- A tool for analyzing discrete action protocols

13. Basic Structure of ACT-PRO [3] Discrete Action Protocol ACT-PRO Grouping program Tracing program Grouping program Grammars of action patterns Hierarchical structure Manual modification based on results The grouping and tracing programs of ACT-PRO. Solid lines represent direct input; dotted lines represent manual modification based on the results of the program outputs.

14. Task-relevant Hierarchical Goal Structure of a VCR Interface [3] PROGRAM-A-SHOW SET-START-TIMESET-END-TIMESET- CHANNEL SET-DAY-OF- WEEK SET- STAR T- HOU R SET- START - 10MIN SET- STAR T- MIN SET- END- HOU R SET- END- 10MI N SET- END- MIN Start- hour button Start- 10min button Start- min button End- hour button End- 10min button End- min button Channe l button Day- of- week button

15. A Syntactic Representation of Action Pattern – Grammar SET-CHANNEL: [Object1][Object2][Object3 ](1) [Object1] → channel(2) [Object2] → up-arrow(3) [Object2] → down-arrow(4) [Object2] → up-arrow [Object2](5) [Object2] → down-arrow [Object2](6) [Object2] → enter [Object2](7) [Object3] → enter(8)

16. Parsing the Action Protocol by Using the Grammars When a sequence of actions matches all the grammar rules in a grammar in the specified order, the sequence of actions is “captured” by the grammar [3]. Example: if the sequence of actions channel, up-arrow, down-arrow, enter is matched to the grammars.

17. Three types of protocol analysis and their applications Verbal protocol analysis Discrete action protocol analysis Eye-movement protocol analysis

18. Eye-movement Protocol Analysis (I) By studying eye movements, researchers can understand user behavior in basic interface tasks to reveal how users encode and process information and to infer user intent in real- time interfaces [4]

19. Eye-movement Protocol Analysis (II) Pros – Can provide a wealth of information – Convenient – Subjects need little instruction and training Cons – Generate enormous sets of eye-movement data – Include a great deal of equipment noise – High degree of individual variability

20. Topics covered What are protocol and protocol analysis History of protocol analysis Three types of protocol analysis and their applications A comprehensive protocol analysis software and WBG (Web Behavior Graph)

21. A Comprehensive Protocol Analysis System -- SNIF-ACT [5]

22. WBG -- Web Behavior Graph [5]

23. References [1] Fu, W.-T., & Gray, W. D. (2000). Implications of rational analysis for interface design: Observations in a natural learning environment. Manuscript submitted for publication. [2] D.Anders Ericsson & Herbert A. Simon (1993). Protocol Analysis: Verbal Report as Data. Massachusetts Institute of Technology [3] WAI-TAT FU ACT-PRO action protocol analyzer: A tool for analyzing discrete action protocols. Behavior Research Methods, Instruments, & Computers 2001, 33 (2), 149-158 [4] Dario D. Salvucci & John R. Anderson Automated Eye-Movement Protocol Analysis. Carnegie Mellon University, Cambridge Basic Research [5] User Interface Research at PARC http://www2.parc.com/istl/projects/uir/projects/snif-act/ http://www2.parc.com/istl/projects/uir/projects/snif-act/ [6] Ericsson Protocol Analysis and Verbal Reports on Thinking – An Updated and Extracted Version From Ericsson(2002).

24. Question and Answer