1. Role of technology

2. Applying Grounding Theory To Technology
Clark & Brennan (1991): “People should ground with those techniques available in a medium that lead to the least collaborative effort.”
Hypothesis: Objective characteristics of different communication media change the costs of conversational grounding and strategies people use.
Some key types of costs:
Production/Reception costs: costs of producing/receiving messages
Start-up costs: costs of initiating conversation
Asynchrony costs: costs of timing utterances
Speaker change costs: costs of turn-taking
Repair costs: costs of correcting misunderstandings
Should allow us to predict in advance what features new technologies should have to meet different collaborative purposes

3. Affordances of Communication Media (Clark & Brennan, 1991)
Co-Presence
Participants share physical environment, including a view of what each other is doing and looking at
Visibility
Participants can see one another but not what each is doing or looking at
Audibility
Participants can hear one another
Cotemporality
Messages are received close to the time that they are produced, permitting fine-grained interactivity
Simultaneity
Multiple participants can send/receive messages at the same time, allowing backchannel communication
Sequentiality
Participants take turns in an orderly fashion in a single conversation
Reviewability
Messages do not fade over time
Revisability
Messages can be revised before being sent

4. Technology changes strategies and costs of grounding
Exactly how conversationalist achieve common ground depends up the details of the technology available
Features of communication setting
Change
Needs for & costs of
formulation
production
reception
understanding
start-up
delay
asynchrony
speaker change
display
fault
repair
co-presence
visibility
audibility
co-temporality (no lag)
simultaneity (full duplex)
sequentiality
reviewability
revisability

5. Affordances of Conventional Media
Face-to-Face
Video Conf.
Phone
Copresence ++ ? --
Visibility +
Audibility
Cotemporality
Simultaneity
Sequentiality
Reviewability
Revisability

6. Interactivity improves communication
encoding
Message
decoding
Sender
Receiver
Response/
Backchannel
What does the traditional model leave out? 2 headed arrows.
Want to briefly demonstrate that interactivity has major effects on communication outcome
Early research I did shows that if 2 people are hearing the same story fm a speaker, but only one can provide back channel feedback & other has feedback blocked, that the active listener understand the story better, agrees more with the speaker.
Speaker's story is more tailored to what the active listener's knowledge. E.g., efficient where the active listener was previously briefed, and more repetitive where active listener made confused.
decoding
encoding
Feedback and interactivity is one effective way of achieving common ground
Feedback tailors communication to an audience, making it more effective

7. Effects of technology on partner-specific learning
Need feedback to learn from each other
Some technology can disrupt the feedback
Half-duplex (speakerphone) vs full-duplex audio (telephone)
Even msecs reduces coordination
Video that desynchronizes audio & video channels disrupts lip-reading

8. Role of Visual Information

9. Referential Communication Results
Multiple 2-person referential communication tasks
E.g., Find nearest doctor on map
E.g., Build trash cart
Common results:
Voice speeds solutions compared to typing & writing
Faster times
More turns
More words
Visual channel doesn't help (where video is for seeing partner)
(fm Chapanis, 1972)

10. "Talking-heads" video doesn't improve referential communication
Tacit assumption in using video
Since face-to-face communication is successful, communication media that are more like face-to- face are better
Most of the content is in the words
Gestures may be pre-verbal, rather than illustration
For emotion, video and audio channel can be redundant
Rich media are useful for handling ambiguous and controversial topics
E.g., Images change lie-detection, but help liar over the lie-detector
Seeing your partner doesn't improve ability to communicate about objects in the world:

11. Cisco Telepresence Application
Will this technology improve distributed work?

12. Physical Co-presence Provides Visual Cues for Grounding
Physical co-presence: People are present at the same time in the same place
Provides several types of visual information that can be used to ground utterances.
Task-oriented video systems vs. “talking heads” video:
Advantages: View of others’ hands and of task objects allows monitoring of partner’s attention and comprehension; shared view of objects allows for efficient reference.
Disadvantages: No feedback from others’ facial expressions

13. Bicycle Studies: Method
Bicycle repair task
25 pairs of participants
“Helper” provides instructions
“Worker” repairs the bike
Communications Media
Side-by-Side
Audio/Video: head-mounted camera showing workers’ hands and field of view
Audio-Only
Within-subjects design
Dependent measures
Task completion times, observer ratings of work and communication quality, surveys, content-coded transcripts

14. In the Video Condition Pairs Used Visual Space
Grounding: Used visual space to disambiguate reference, establish common ground, & identify task elements
Situational awareness: To know when worker was ready for next instruction
Example(219_s_v.mov)
But technical flaws limited its utility
Example(Clip230_s_v.mov)

15. Bicycle Study: Key Findings
Performance is best with full physical co-presence (Side-by-Side)
Communication with video system was more efficient than audio-only.
Workers use deictic pronouns for task objects/locations when they think the helper can see them
The head-mounted video system used in this study did not adequately support shared visual space.
The top figure shows performance times in minutes by media condition. As can be seen, the side-by-side condition (left bar) is significantly faster than the other two conditions, which did not differ from one another.
The second figure shows a portion of our conversational analysis, focuses on deictic reference.
Reference: A small but critical proportion of overall messages. Objects had to be identified before instructions for working with them could be given. Ability to see participants’ behaviors and task objects should lead to more efficient reference. The results show that helpers used deictic references very little unless they were side-by-side with the worker. Workers, however, used deictic references almost as much when they knew the helper could see them via video as when they were side by side.

16. New Research Paradigm to Test Those Questions
Stylized referential communication task
To increase control
To systematically vary task attributes
Construct artificial shared visual environment
Allows independent manipulation of features of a shared space that co-occur in the real world => identify which are important

17. Cooperative Jigsaw Puzzle Task
Helper has picture of target and gives instructions to worker, who moves pieces to match target
Subjects communicate via audio & shared computer screens
Target
Shared view
Work area
Staging area

18. Exploring the Role of Shared Visual Information
What features of physical space influence its value?
Fidelity of views
Hypotheses: Delay, small windows, rotation & other factors that make views dissimilar will degrade collaborative performance
When is shared visual context most important?
Visual complexity
Hypothesis: When task is complex enough that language itself is insufficient to efficiently describe events

19. Task visual complexity
Manipulations
Task visual complexity
Visual fidelity
None: Audio only
Partial
Shared screen with a 3-second delay
Shared screen with rotation
Shared screen with a small view port
Immediate: Full shared screens with no delay & no rotation
Simple
Complex
Primary colors
Tartan plaids
Static colors
Changing colors
Pieces abutted
Pieces overlapped

20. Summary of Multiple Experiments
Task performance
Shared visual space improved task performance (speed & accuracy) in all experiments
Improved performance most for visually complex tasks
Shifted conversational strategies
Shared visual space improved improved efficiency of reference (e.g., words/reference)
Lack of shared visual space forced many workarounds

21. Experimental Manipulations
Fidelity of the Visual Space
Immediate
Delayed (3 seconds)
None
Other studies
Rotation of the spatial perspectives
Discontinuous, “push to see” images
Visual difficulty:
Static vs. Dynamic Tasks
Spatially easy vs. difficult puzzles
Easy versus difficult to name objects
Same vs. different visual perspective
Immediate condition
TALK ABOUT THE ADVANTAGES TO EACH OF THESE!!!
While Hypothesis 1 might seem straightforward, remember that in the previous study Fussell et al. found no difference between those performing the task with the head-mounted camera in comparison to those in the audio-only condition
Hypothesis 2 and 3 In cases were the scene can easily be described with language (e.g., simple objects that are easily discernable) there is less value to the SVS, however, when the objects or environment are changing or difficult to describe there should be an added value to having the SVS
Hypothesis 4 By seeing the partner perform some task, the Helper gets immediate feedback about whether the partner understood a directive. If the visual feedback were delayed (e.g., as caused by video compression or network lags), the value of the visual information may be diminished. Delay in updating the display may diminish the value of the visual information both for the ability to compare the current work state to the goal state (as described in the earlier scenario) and for the team’s ability to coordinate language (as discussed here).
Hypothesis 5 As suggested by Clark’s theory, we should expect to see structural and content changes in the discourse as the pairs attempt to communicate using the least collaborative effort
Visual feedback, however, may be less necessary if the task is simple enough (e.g., a game of tic-tac-toe) or if the pair has an efficient, well-practiced vocabulary to describe events (e.g., routine communication between pilot and air traffic controllers). In these cases, a shared visual display provides little new information.
No SVS condition

22. How Does SVS Change Communication
How are pairs communicating differently when they have a shared visual space?
Communication is more efficient with a shared visual space
Helper uses actions to assess worker’s comprehension
Both helper and worker use more efficient referring expressions and diexus (e.g., this/that/pointing)
Helper can precisely time interruptions and corrections
SVS facilitates an awareness of the task state
Without shared visual space
Worker is responsible for updating helper on state of the task
Worker verbally indicates understanding of an instruction, and is sometimes wrong
Examined Details of Discourse to Explore Predictions
We examined word rate (the number of words, controlling for time)
Explored details of Helper and Worker utterances separately to understand how overall communication was changing

23. Shared Visual Space Is More Important in Changing Environments
Shared visual space improved performance
Immediate significantly faster than delay and no SVS (p<.0001)
3 second delay led to significant decrease in the value of the shared space
Shared visual space was more important when the objects were changing (i.e., hard to describe)
Immediate affected significantly less than delay (p=.05) and no SVS (p=.0002)
Shared visual space is less important when words can easily describe the objects and environment
Puzzle Difficulty
Easy vs. Difficult (LS Means: 61.9 sec vs sec, p=0.002)
Color Drift
Stable vs. Drifting (LS Means: 54.3 sec vs sec, p<0.001)
Interaction:
Visual Space * Puzzle Difficulty Interaction
Not significant (F (2,256) = 1.06, p > .35)
Instructive in that visual complexity itself did not raise the value of a shared view of the work.
It was primarily when the task was dynamic and the environment was changing that having the display was most beneficial

24. Rate of Word Production: Workers Increase Their Speech Rate
Pairs increased speech rate when fidelity decreased (all p<.02)
The fidelity of the SVS influenced Workers more than the Helpers (F(2,110) = 10.80, p < .0001)
The pairs adjusted their use of language to accommodate for lack of shared space
Workers increased their speech rate to compensate
The accommodation was insufficient in comparison to when the pairs made use of the shared visual space
Fidelity of Shared Space
Immediate vs. Delay (t=-2.55, p=.01)
Immediate vs. No SVS (t=-4.84, p<.0001)
Delayed vs. No SVS (p<.017)
(LS Means: Immed=19.4; Delay=30.1; None=45.0 words per unit time)
Fidelity of Shared Space * Speaker Role Interaction
F (2,110) = 10.80, p < .0001
The fidelity of the SVS influenced the Worker’s efficiency much more than the Helper’s
Changes in Worker behavior reflect their accommodation to differences in the Helper’s view of the workspace
Adjust use of language, but still not as good as when they have SVS
*** NOTE ***
THIS IS VERY INTERESTING!!
The Worker always sees the same screen. They have their staging area on the right, and the work area on the left.
However, they adjust their speech pattern (they begin to speak more, while the Helpers stay approximately the same) in order to facilitate the efficiency of the group as whole. This provides support for Clark’s theory of least collaborative effort.

25. Conversation Issuing Acknowledgements
Workers took a more active role in ensuring messages were understood when there was no shared visual space (i.e., when the helper could not see them).
Immediate SVS
No SVS
H: The, the right hand, the top right hand corner of the blue block touches the bottom left hand corner of the first orange block.
W: [Positioned piece correctly]
W: Like that?
H: Yeah.
H: All right that's good.
H: And that's gonna be on top of the red one but only the right side of the red is going to be showing.
H: You know what I mean?
W: OK, so it's like...
H: Oh, like, put it on the left side of the red.
W: ...side of it and you see half of the red block.
H: Right, of the red, Yeah.
W: OK.

26. Typically the Helper gave directives and the Worker moved pieces
Acknowledgements of Understanding: Pairs Use SVS to Monitor Comprehension
Pairs were most explicit in stating their understanding when they had no shared visual space
Workers were more explicit in stating their understanding when there was no shared visual space available
Typically the Helper gave directives and the Worker moved pieces
Used SVS to monitor understanding
Reserved language for breakdowns
Without the SVS the pairs substitute language to confirm understanding
Shared Visual Space
Immediate vs. No SVS (p<.0001)
Immediate vs. Delay (ns)
The pairs were most explicit in stating their understanding when they had No SVS
Shared Visual Space * Speaker Role Interaction
F(2,107)=8.752, p=.0003
Use SVS to monitor comprehension
Follow cycle of Helper giving directives and reserve language for breakdowns
When No SVS it is important for Workers to explicitly state their understanding
When SVS Available
More efficient to follow cycle of Helper giving Worker instructions and reserving speech for when things go wrong
When SVS Fidelity Decreases
Workers must be more explicit in communicating their level of understanding
Bigger difference for the No SVS (t=4.05, p<.0001) than for the Delayed (t=1.36, p=.1777) in comparison to the Immediate
The interaction between the shared visual space and the speaker demonstrates that it was more important for the Workers to explicitly state their understanding when the shared visual space was of lower fidelity (for the interaction F(2,107)=8.752, p=.0003).
Closer examination reveals that this is even more evident in the No Shared Visual Space pairs than it is for the Immediate (t=4.05, p<.0001), while there appeared to be less difference between the Immediate and Delayed Shared Visual Space (t=1.36, p=.1777).
The Helpers typically produce about the same portion of acknowledgements regardless of the degree of the shared visual space. However, the Workers significantly increase their rate of production.

27. Using visual info for grounding vs situational awareness
ImmedDiffStable PM-B3.mpg
Grounding: Visual info helps parties to assess whether the language used is mutually understood or needs to be elaborated
D:: Do the red first
W: Which red?
D: Dark red (W drags out a red block).
D: Darker (W drags out a red block)
D: Is there a lighter one? (W drags out a red block)I
D: Yeah, that’s it
Situational awareness: Visual info allows parties to assess the current state of a task and plan future actions
D: Now you want a darker blue on top of the lighter blue.
W: (moves the correct tile into position)
D: (realizes W finished the last instruction & D can offer a new instruction)
D: OK, now an orange touching the top right hand corner …

28. Feelings of connection may be more important than information flow

29. What About Feelings of Connection? (Garau, Slater, Bee, Sasse, 2001)
Study comparing video with human, avatar w/ realistic gaze, avatar w/ random gaze & audio communication
Negotiation task – avoid a scandal
Garau, M., et al. (2001). The Impact of Avatar Realism and Eye Gaze Control on Perceived Quality of Communication in a Shared Immersive Virtual Environment. Proceedings of the Conference on Computer-Human Interaction (pp ). New York: ACM

30. Like video & realistic avatars best
Sample questions:
Face-to-face: This felt like a natural conversation
Involvement: I was absorbed in the converation
Co-presence: was very aware of my conversation partner
Partner evaluation: I trusted my partner.

31. Yee et al Meta-Analysis
Meta-analysis of 25 experiments
People interact via text or voice
Presence of avatar
None
Unrealistic (cartoon)
Photorealistic
Outcomes
Performance
Subjective evaluations of experience or partner
Performance & subjective evaluations improved with avatar
Realism only influence subjective evaluations
Note: Minor difference in performance outcome when people interact with vi text or voice partner with no visual representation (no avatar) or with an avatar representation. The presence of the avatar made no difference for object performance measures. If avatar, is it a cartoon representation or photorealistic? In terms of subjective outcomes, people feel much closer to a photorealistic vs cartoon avatar.
Yee, N., Bailenson, J., & Rickertsen, K. (2007). A meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in interfaces CHI '07 Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 1-10). New Yor: ACM