1. Role of Visual Information

2. Multiple 2-person referential communication tasks E.g., Find nearest doctor on map E.g., Build trash cart Common results: Voice speeds solutions compared to typing Faster times More turns More words Visual channel doesn't help (in a talking head set-up) (fm Chapanis, 1972) Referential Communication Results

3. "Talking-heads" video doesn't improve referential communication Tacit assumption in using video Since face-to-face communication is successful, communication media that are more like face-to-face are better Most of the content is in the words Gestures may be pre-verbal, rather than illustration For emotion, video and audio channel can be redundant Rich media are useful for handling ambiguous and conflictful topics E.g., Images change lie-detection, but help liar over the lie-detector Seeing your partner doesn't improve ability to communicate about objects in the world: https://www.youtube.com/watch?v=0kd2SO1_kSA

4. Physical Co-presence Provides Visual Cues for Grounding Physical co-presence: People are present at the same time in the same place Provides several types of visual information that can be used to ground utterances. Task-oriented video systems vs. “talking heads” video: –Advantages: View of others’ hands and of task objects allows monitoring of partner’s attention and comprehension; shared view of objects allows for efficient reference. –Disadvantages: No feedback from others’ facial expressions

5. Bicycle Studies: Method Bicycle repair task 25 pairs of participants –“Helper” provides instructions –“Worker” repairs the bike Communications Media –Side-by-Side –Audio/Video: head-mounted camera showing workers’ hands and field of view –Audio-Only Within-subjects design Dependent measures –Task completion times, observer ratings of work and communication quality, surveys, content-coded transcripts

6. In the Video Condition Pairs Used Visual Space Grounding: Used visual space to disambiguate reference, establish common ground, & identify task elements Situational awareness: To know when worker was ready for next instruction –Example(219_s_v.mov)Example(219_s_v.mov) But technical flaws limited its utility

7. Confirming referents H: The next thing we got to do uh, ok, hook the brake cable up. H: The brake cable’s unhooked right now. Uhm... if you look up uh... ju uh... to the top part of the brakes here. W: U-huh H: You see uh this, this loose cable that kind of goes across from this one arm to this other arm. W. Uh H: Called the straddle cable W: The little tiny one. It’s got a little kindy, tiny horse shoe thingy hanging from it H: No, that’s called, that’s the actual cable coming from above. W: Right H: Ok, there’s also a second part, which is this cable that just kind of loops around from here to here. W: Ok. I see it. It's hooked between the two right now. H: Right. Right. It's attached on both ends. W: Right. H: Ok. Uhm, what you want is, you want to hook that uh... that anchor plate - that little horseshoe thingy. It... W: Uh huh. H:... hooks on the straddle cable

8. Bicycle Study: Key Findings Performance is best with full physical co-presence (Side-by- Side) Communication with video system was more efficient than audio-only. –Workers use deictic pronouns for task objects/locations when they think the helper can see them The head-mounted video system used in this study did not adequately support shared visual space.

9. Exploring the Role of Shared Visual Information What features of physical space influence its value? –Fidelity of views –Hypotheses: Delay, rotation & host of other factors that make views dissimilar will degrade collaborative performance When is shared visual context most important? –Visual complexity –Hypothesis: When task is complex enough that language itself is insufficient to efficiently describe events

10. New Research Paradigm to Test Those Questions Stylized referential communication task –To increase control –To systematically vary task attributes Construct artificial shared visual environment –Allows independent manipulation of features of a shared space that co-occur in the real world => identify which are important

11. Cooperative Jigsaw Puzzle Task Helper has picture of target and gives instructions to worker, who moves pieces to match target Subjects communicate via audio & shared computer screens Target Shared view Work areaStaging area

12. Manipulations Task visual complexity Visual fidelity –None: Audio only –Partial Shared screen with a 3-second delay Shared screen with rotation Shared screen with a small view port –Immediate: Full shared screens with no delay & no rotation SimpleComplex Primary colorsTartan plaids Static colorsChanging colors Pieces abuttedPieces overlapped

13. Summary of Multiple Experiments Task performance –Shared visual space improved task performance (speed & accuracy) in all experiments –Improved performance most for visually complex tasks Shifted conversational strategies –Shared visual space improved improved efficiency of reference (e.g., words/reference) –Lack of shared visual space forced many workarounds

14. Experimental Manipulations Fidelity of the Visual Space Immediate Delayed (3 seconds) None Other studies –Rotation of the spatial perspectives –Discontinuous, “push to see” images Visual difficulty: Static vs. Dynamic Tasks Other studies –Spatially easy vs. difficult puzzles –Easy versus difficult to name objects –Same vs. different visual perspective Immediate condition No SVS condition

15. How Does SVS Change Communication How are pairs communicating differently when they have a shared visual space? Communication is more efficient with a shared visual space Helper uses actions to assess worker’s comprehension Both helper and worker use more efficient referring expressions and diexus Helper can precisely time interruptions and corrections SVS facilitates an awareness of the task state Without shared visual space Worker becomes responsible for updating helper on state of the task

16. Shared Visual Space Is More Important in Changing Environments Shared visual space improved performance Immediate significantly faster than delay and no SVS (p<.0001) 3 second delay led to significant decrease in the value of the shared space Shared visual space was more important when the objects were changing (i.e., hard to describe) Immediate affected significantly less than delay (p=.05) and no SVS (p=.0002) Shared visual space is less important when words can easily describe the objects and environment

17. Rate of Word Production: Workers Increase Their Speech Rate Pairs increased speech rate when fidelity decreased (all p<.02) The fidelity of the SVS influenced Workers more than the Helpers (F(2,110) = 10.80, p <.0001) The pairs adjusted their use of language to accommodate for lack of shared space Workers increased their speech rate to compensate The accommodation was insufficient in comparison to when the pairs made use of the shared visual space

18. Conversation Issuing Acknowledgements Workers took a more active role in ensuring messages were understood when there was no shared visual space (i.e., when the helper could see them). Immediate SVSNo SVS H: The, the right hand, the top right hand corner of the blue block touches the bottom left hand corner of the first orange block. W: [Positioned piece correctly] W: Like that? H: Yeah. H: All right that's good. H: And that's gonna be on top of the red one but only the right side of the red is going to be showing. W: [Positioned piece correctly] H: You know what I mean? W: OK, so it's like... H: Oh, like, put it on the left side of the red. W:...side of it and you see half of the red block. H: Right, of the red, Yeah. W: OK.

19. Acknowledgements of Understanding: Pairs Use SVS to Monitor Comprehension Pairs were most explicit in stating their understanding when they had no shared visual space Workers were more explicit in stating their understanding when there was no shared visual space available Typically the Helper gave directives and the Worker moved pieces Used SVS to monitor understanding Reserved language for breakdowns Without the SVS the pairs substitute language to confirm understanding

20. What About Feelings of Connection? (Garau, Slater, Bee, Sasse, 2001) Study comparing f/f, avatar w/ realistic gaze, avatar w/ random gaze & audio communication Negotiation task – avoid a scandal DVs –Face-to-face feel, Involvement, co-presence, partner evaluation:

21. Yee et al (2007) Meta-Analysis Comparison of 25 experiments –People interact via text or voice –Presence of avatar None Unrealistic (cartoon) Photorealistic –Outcomes Performance Subjective evaluations of experience or partner Performance & subjective evaluations improved with avatar Realism only influence subjective evaluations Yee, N., Bailenson, J., & Rickertsen, K. (2007). A meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in interfaces CHI '07 Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 1-10). New Yor: ACM

22. Apple Facetime https://www.youtube.com/watch?v=iu1jHtf_oUc