1. Presented by Arseniy Mstislavskiy

2. Debriefing & Performance
Performance is «What?»
Debriefing is «Why?»
D. is for reviewing participant's actions at performance part of test
done after every test session
to understand each error, difficulty, omission occured during session
D. is the last chance for it
Participants explain things you can't see (thoughts, preferences, rationale behind actions)
Observers give diff. point of view, fill gaps in memory, may feel engaged and help with resolution later,
the moderator can take from interactions with observers some notion of the
priorities of the stakeholders on the design team, a useful thing to know when
it comes time to report results.

3. I. Debriefing with participants
Don't make p. feel defensive about actions, opinions
Don't react to p.'s answers
If that happens p.'s answers become suspect
«Thanks for answering that. I appreciate your feedback» - and go further

4. 1. Place
Same room: reference to product needed, p. became comfortable in a room
Another room: p. is shy, distracted or tired (testing in a large observed room)
Move if there is no choice (observers can sill be involved)

5. 2. Basic guidelines Right after the test
«We're getting to the end»
P. fills any post-test questionnaires
You collect thoughts (simultaneously)
How the session went (in general)
Unresolved issues left
Unnoticed things
Missed crucial instructions
Possible reasons for unclear actions
Review debriefing topics guide (sort and remove issues)

6. 2. Basic guidelines Continuation
Review questionnaire (p. rests)
Unexpected interesting answers
Behaviour and answer don't match Or
skim through notes
gather questions from observers

7. 2. Basic guidelines Let p. express any ideas «What do you think?»
Start with high-level issues (use debr. topics guide) general research questions
Move to specific issues (found)
Review marked places of the post-test questionnaire — understand the basis of a problem
D. is not for resolving problems (e.g. asking p. what he/she thinks)
Finish questioning before letting observers get in
O. write their questions down (even in same room)
O. ask themselves (if p. is comfortable)

8. 3. Advanced guidelines Replay the test
Manual method (reexperiencing emotions, reactions)
p. can visualise what he/she experienced at specific point
You ask questions you had
Video method
Requires more time
Is better then memorising
Seeing themselves may help p.
Removes the need to interrupt p. during the test performance

9. 3. Advanced guidelines audio record d. Transcribe it later
Good for pulling remarks
Helps in content analysis
Review alternate designs
Collecting p.'s preferences during early stages of developement
Comparing prototypes is easier for p. than generating ideas

10. 3. Advanced guidelines «What did you remember»
Can't be asked during the test («Do you see this …?» - «Of course I do!»)
Cover/remove particular element
! Be aware of dangerous situations
«Devil's advocate»
P. doesn't want to be critical
Very important feature (need to be 100% sure)
Be careful
Say you're surprised
LIE! Say that others say the opposite.
Doesn't work on
Very young/old persons
Ingratiating persons
Testing => possible job
Warn observers and team beforehand (!)
Is only used to get real feelings, not to bias

11. II Debriefing with observers
Make o. co-researchers
O. are other people to talk about what happened
O. become more engaged later
You see important things to include in report
You make team agree on issues => remedies

12. II Debriefing with observers
No break after test session
«How did that go?»
Gather ideas in the rolling issues list (will help for top 10 type reports)
«What surprised you?»
Group ideas
Go through task by task
Let everybody
understand the issue
agree on the reasons
Document all items (in a way accessible to o.)
Stop when all o.'s insights voiced

13. III At the end Make last debriefing session with o.
Step throught the rolling issues list
Helps
organising findings in the final report
Be sure noone is surprised with report
Ask o.
Any revisions
Sort items
Voting
Call out top 10
Card sorts
Don't talk about remedies
Remember you will have to review collected issues
Those are just accumulated impressons and personally biased recalls
You then see subtle differencie possibly conflicting with debriefing with o.
==> ==> Ch. 11 «Analyze Data and Observations»