1. Discourse and Pragmatics
Lecture 11
Multimodal Discourse Analysis

2. The changing face of texts

3. Multimodal Discourse Analysis
Two Strands
Analysis of Interaction
Influence of Conversation
Analysis, Interactional
Sociolinguistics, Mediated
Discourse Analysis
Kendon, Goodwin, Norris
Analysis of Static
Texts
Influence of ‘Textual
Analysis’ (Halliday--
Systemic Functional
Grammar)
Kress and Van Leeuwen
Reading Images

4. MULTIMODAL ANALYSIS OF STATIC TEXTS
Strand I:
MULTIMODAL ANALYSIS OF STATIC TEXTS

5. Reading Images
Social Semiotics
Halliday’s three ‘metafunctions’ of language
Ideational
What’s happening? (processes and participants)
Interpersonal
Relationship between text and reader
Textual
How the text is organized

6. Typography

7. Kinds of Images Ideational Narrative Image Conceptual Image
“presenting unfolding actions and events, processes of change, transiting spatial arrangement”
(Kress & van Leeuwen 1995, p.79)
Conceptual Image
“representing participants in terms of their more generalized and more or less stable and timeless essence” (ibid.)
Analytical Process
“relate participants in terms of a part-whole structure” (p.89)
Classificational Process
“relate participants to each other in terms of a ‘kind of’ relation, a taxonomy” (p. 81)

8. Narrative Image Ideational
“what in language is realized by words of the category ‘action verbs’ is in pictures realized by elements that can be formally defined as ‘vectors’. (p. 44)

12. Ideational
Analytical Image
Parts
whole: the earth

13. Classificational Image
Ideational
Classificational Image

14. Classification + Narrative

15. Perspective/ Point of View
Ideational
Perspective/ Point of View

16. Interpersonal (Gaze)

17. Interpersonal
Demand
“the participant’s gaze (and the gesture, if present) demands something from the viewer, demands that the viewer enter into some kind of imaginary relation with him or her.” (p.122)

18. “The viewer’s role is that of an invisible onlooker”
Interpersonal
“The viewer’s role is that of an invisible onlooker”
(p.124)
“All images which do not contain human or quasi-human participants looking directly at the viewer are of this kind.” “it ‘offers’ the represented participants to the viewer as items of information, objects of contemplation, impersonally, as though they were specimens in a display case.”

19. Distance

20. Textual
Visual Space

21. Left-Right L
Given R
New

23. Top-Bottom T
Ideal B
Real

26. Intertextuality

27. Questions to Ask Rhetorical Purposes Where does the image appear?
What is the image's purpose?
Ideational Function
Does it document a  situation, event or condition?
Is it conceptual? If so, what is its point?
Who is portrayed? Describe your inferences from each feature of the person(s) - age, details of dress, gender, ethnicity, class, posture and stance, portions of the body shown, tilt of head, facial expression, gesture of hands.
What is the person looking at? Follow her gaze or eyeline. Does she look toward something else in the image? or out of the picture? What do you make of the direction of the gaze? .

28. Questions to Ask
If there are two or more people, what features suggest their relationship to each other?
If there are two or more people, does one seem dominant? How is this expressed?
If the image has a distinct background, describe it. How does it relate to the dominant focus of the images, especially people, if any?
What is the story being told in the image? Consider the people and objects in the image and their relationships to each other, the viewer, the setting and the text.
Who or what is excluded from the image? Why?
What time and place does the image suggest? What is the effect of this setting?
Is anything "out of place" in the image? What do you make of the incongruity?
Are there items or features in the image that might mean more than themselves? Consider connotations and associations of particular objects or features in the image. Relate them to the rest of the image

29. Questions to Ask Interpersonal Function
From what angle are the people shown? Do you seem to look down on them, as if they were below the viewer? Look up at them? Look right at them?
Are people shown close up or far away? What emotional effect does this have?
What do you consider to be your relationship as viewer to the person or people shown? Do you empathize with them or not? Explain why.
If there is no one represented, imagine what sort of person would be at home in this image. Explain why?

30. Questions
Is the image realistic, like a photograph or more stylized, like a cartoon or caricature?
If the image is realistic, do you detect any types of distortion? Describe any features that may be distorted.
How polished or "professional" is the image?What tone does the image project?
How seriously are we to take this image? Explain why.
Who do you think are the intended viewers of the image? What features or context suggest this audience?
Who do you think produced the image? Is the creator/photographer stated?
What would you say your relationship is to the producer or producers? Do you think they understand you, as a viewer?
What attitudes - - social, political, economic, cultural - - are suggested in this image? Who benefits from the attitudes shown? Who does not?
Who can relate to the story that the image tells? Who may not find it believable or interesting?

31. Questions to Ask Textual Function: Overall Design
What draws your eye first?
What does the dominant part of the image portray?
What is in the center of the image?
What is shown in front and larger? What is behind and smaller?
What is shown in the upper half? the lower half?
Are portions more blurred? Are there very distinct parts in sharp focus?
Is there "empty" space? What does the empty space frame?Are some areas or shapes very large? Are others very small?
Describe the major shapes and lines created. Consider what effect the shapes and lines create.
Describe the overall arrangement of parts. Are they ordered symmetrically or otherwise balanced against each other?

32. Questions to Ask
Color
Describe the colors, or absence of color, in the image.
Where is color applied?
Is the color realistic, in your view? If not, describe why you think it is not.
How does color, or its absence, make you feel about the image?
What previous associations do you have with the colors used? How do those affect your understanding of the image?
Text
If the image includes text, such as headlines, labels, captions, or paragraphs of explanation, relate the text to the image.
In what ways does the text help you make sense of the image? Does it answer questions about the image, or only raise more questions?
What is the personality and tone of the typography, or the fonts that the text uses?

33. MULTIMODAL ANALYSIS OF INTERACTIONS
Strand II:
MULTIMODAL ANALYSIS OF INTERACTIONS

34. Multimodality Proxemics and Posture Written Text Gaze Gesture Object
Handling

35. Multimodal Interaction Analysis (Norris 2004, Norris and Jones 2005)
All communicative actions are mediated
though one or more semiotic modes
Affordances
Mediational
Means
Action
Actor
Semiotic
modes
It’s important to note that semiotic modes also place affordances and constraints on the other semiotic modes that might be used with it
Reading gaze
Semiotic modes come with different sets of affordances and constraints
Constraints
Modal Density Modal Complexity

37. Modal Density: Leena Body/Head Position Spoken Language Body/Head
Gaze
Gaze
Written Text
Written Text
Object Handling

38. Modal Density: Derek Body/Head Position Body/Head Position Gaze Gaze
Spoken
Language
Body/Head
Gesture
Gesture

39. Modal Density: Leena Body/Head Position Spoken Language Gaze Body/Head
Gesture
Gaze
Object
Handling
Written
Text
Written
Text

40. Modal Complexity Gesture Media Gaze Rhythm Object and Handling Timing
The use of one semiotic mode affects all the other modes that are used along with it
Gesture
Media
Gaze
Rhythm
and
Timing
Object
Handling

41. Gaze Relationship to Speech Cultural Practices of Gaze
Kendon (1990)
Hearers gaze at speakers more than speakers gaze at hearers
Speakers typically look towards hearers at the end of phrases
Goodwin (1981)
Mutual gaze occurs more typically at turn beginnings
Cultural Practices of Gaze
Gaze and power among East Asians
Gaze and Attention/Awareness
Modal Density (Norris 2004)
Kendon 1967
Hearers gaze at speaker more than speaker gazes at hearers
Speakers typically look towards hearers at the end of phrases
Goodwin 1981
Mutual gaze at turn beginnings.

42. Relationship to Media

43. Leena and Derek

44. Mutual Gaze

45. Conversational Synchrony
Birdwhistle, 1970
Postural Echo

46. Asynchrony

47. Parallel and Sequential Synchrony
Gill, Kawamori, Katagiri, Shimojima, 2000; Gill, 2001; Gill, 2002; Gill and Borchers, 2003; Gill, 2003a; Gill, 2003b
Sequential Synchrony: action-reaction
Parallel Synchrony: parallel motion
Interaction tends to move from sequential to parallel synchrony
Sequential Synchrony
Serves to maintain the interaction
Parallel Synchrony
Serves to ‘transform’ the interaction
‘Grounding’
rhythmic synchrony (Birdwhistle, 1970)
Gill:
Sequential Body Moves have the structure of action-reaction motion, whilst Parallel Coordinated Moves have the structure of parallel motion (). They have different priorities in their functionality. Sequential moves serve to maintain the communication, whilst parallel moves serve to transform the communication. There is a pulsation in the movement from sequential to parallel action that facilitates the process of grounding. 

When a designer is making contact with the surface to act upon it, whilst the other person is doing so too, there is an attempt to engage with the body field of the other person, as found in the case of landscape architects drawing up conceptual designs together. It also happens in the example below (Figure 1) of design activity using a whiteboard. The designer on the right side, closest to us (E), enters the body field of the other one (F) who is currently drawing (action), and uses his index finger to trace out a shape to indicate a bed. He is proposing this idea to (F) who is drawing, to get his opinion (negotiation). Both action and negotiation are operating at the surface. 
The body field of the person drawing (F) is not disturbed, and as we know from the discussion of the engagement space, and overlapping space, this indicates a high degree of contact and is identifiable as a Parallel Coordinated Move (PCM).
Space is considered as the space between bodies, and it is a resonating space. When parallel coordinated action occurs, the resonance is at a peak and this expands the space, giving it salience over time. In sequential Body Moves, resonance between bodies is variable depending on the degree of contact and commitment of the persons to each other, and time is more salient than space.
Hence, imagine that I am drawing something with you on a shared sheet of paper, the sequential Body Moves that we perform of action-reaction (to show we acknowledge, agree, disagree, accept, etc), are the dynamics within the overlap of our physical communicative space, but do not overlap at a metacommunicative level. The parallel Body Moves we perform of, say, drawing different parts of the design at the same time on this sheet of paper, do not overlap in physical communicative space, but completely overlap in metacommunicative space

48. Leena

50. so...ummm

51. is there

52. anything parTICular

53. YOU think

54. you want some more

55. help with

56. mmmmm

58. I'd like to uh... 1 2

59. do you know the meaning

60. of this paragraph

61. yeah I understand

62. Leena
‘Can you feel it?’

63. and they can do anything

64. that they LIKE

65. and

66. that's a stop

68. can you feel it

70. that's a stop

71. and then you start a NEW sentence

72. Conversational Synchrony
Derek
(0.5 sec per frame)

73. is this is the introduc

74. tion

75. like the

76. yes introduction

77. the intro

78. duction

79. ehh

80. this to

81. this
this

82. yes
chapter

85. Gesture Associated with clarity and immediacy
Different kinds of gestures
Iconic/Metaphorical
Deictic (pointing)
Holding
Facial Touching
Body Adjusting
Iconic gestures
Pictoral conent
Metaphorical gestures
Abstract ideas
Deictic gestures
Beat gestures

86. ‘Work Space’ Gestures define the ‘work space’
Relationship to higher order and lower order concerns

87. Leena and Derek

88. Type of Gesture

89. Leena’s Pointing

90. Leena’s Pointing

91. you're making a comparison
Derek
you're making a comparison

92. Derek’s Hand Hold

93. Workspace
Derek

94. in this paragraph

95. I

96. want to

98. critique

99. the

100. incorrect

101. impression

102. by

103. this group of

104. people

105. Object Handling Pens Texts Books (Dictionaries, etc.) Mouse/Keyboard
‘Keep the pen or keyboard and mouse in the writer’s hands -- not in yours.”

106. Text alignment

107. 1 min.
ADD TIMESTAMPS

108. 3 min

109. 5 min

110. 7 min

111. 9 min

112. 11 min

113. 14:30

115. Leena and Derek

116. Interactional space = Text
Leena
Interactional space = Text

123. Leena and her Pen
Pointing
Writing
Holding
Gesturing
Conducting

124. ‘Do you have anything else…’ 1 sec
Leena
‘Do you have anything else…’
1 sec

126. is there

127. anything else

141. okay

142. yeah

145. Derek offers the pen

148. they don't know

149. then maybe

150. you can put that in

161. Tutor Feedback Sessions
Leena
‘I can’t believe it. I thought I was really being interactive…I tried to look at him more…but it seems not…scary!’
Derek
‘I really just wanted the client to explain more…um…his idea…it got easier after I gave him the pen.’

162. Leena 2

163. Conclusions Relationship among modes Effect of media Gaze
Conversational Synchrony
Gestures
Object Handling