Presentation is loading. Please wait.

Presentation is loading. Please wait.

MULTIMODAL LINGUISTICS: Directions of research Andrej A. Kibrik (Institute of Linguistics, RAN) CML-2008 Montenegro, September 2008.

Similar presentations


Presentation on theme: "MULTIMODAL LINGUISTICS: Directions of research Andrej A. Kibrik (Institute of Linguistics, RAN) CML-2008 Montenegro, September 2008."— Presentation transcript:

1 MULTIMODAL LINGUISTICS: Directions of research Andrej A. Kibrik (Institute of Linguistics, RAN) CML-2008 Montenegro, September 2008

2 2 The mainstream linguistic approach  Language consists of hierarchically organized segmental units, such as phonemes, morphemes, words, phrases, and sentences  Linguistic form is thus equated with verbal form  Search for “linguistic form” in Google:  The first result is:  “A meaningful unit of language, such as an affix, a word, a phrase, or a sentence.” (TheFreeDictionary.com)  «В своей совокупности языковые знаки образуют особого рода знаковую систему – язык. Наиболее типичным языковым знаком является слово Форма выражения любого словесного знака состоит из фонем» (Лингвистический энциклопедический словарь, с. 167)

3 3 In related disciplines  Assumption typically held by other cognitive scientists, for example psychologists: language consists of words, sentences, and other verbal units  “With no more than 50 to 100 K words humans can create and understand an infinite number of sentences” (Bernstein et al. 1994: )  When psychologists and neuroscientists work with “language”, they almost invariably think that language is a set of individual words or, at most, sentences

4 4 However  There are prosodic, that is non-verbal aspects to sound  Imagine prosody-free talk  or, vice versa, talk behind a wall  Apart from sound, there are other channels of communication, in the first place through vision (body language, gesture, gaze, etc.)

5 5 Multimodality  In order to understand language and communication, all aspects of linguistic form shold be taken into account  This is what is sometimes called the multimodal approach  Modality, or mode, refers to a distinct type of input  In particular, modality is a kind of stimulus associated with one the human senses, particularly hearing and sight  So the verbal component, prosody, and body language all count as modes or modalities  Hence the notion of multimodality

6 6 Goals of this talk  Emphasize the importance of prosody and visual aspects of communication in linguistic research  Show how prosody and visual communication interact with the verbal component, thus suggesting not only the multimodal, but also the cross-modal approach  Propose that linguistics cannot progress without seriously taking multimodality into account

7 7 Are these goals relevant and important?  After all, linguists and other scholars have already been pursuing these issues for many decades, and the respective research traditions are quite rich  But:  First, prosody and visual communication are marginalized in linguistics, they are located in certain “pockets” of the overall linguistic panorama and are tolerated by the mainstream as “paralinguistics”  Those focusing on these information channels often treat them as a “thing in itself”, without integration with the verbal component

8 8 Plan of talk  I. Prosody  Sentence  II. Gestures  Reference  III. Relative contribution of three information channels  IV. Signed languages  Reference  V. Wider context

9 9 I. PROSODY  Prosodic components  pausing  accents  pitch  tempo (of various scope)  registers  degrees of reduction  glottal features  loudness   Prosody is responsible for discourse segmentation into Elementary Discourse Units (EDUs), identified on the basis of several prosodic components and strongly correlated with clauses

10 10 An example of prosodically oriented discourse transcription ....(1.5)/\Озеро...(0.5)какое-то, Lakesome ..(0.3) (Или /\речка, Eitherriver  или /\озеро, orlake  но по-моему \озеро, butI guesslake  потому что’..(0.2)как-то-оw becausesomehow...(0.6)\маленькоетакое, smallsuch  \небольшое.) minor ....(1.0)’и-иh...(0.7)через/него andacrossit..(0.3)как-то\бревнокакое-то, somehowlogsome  типа\моста. likebridge ....(1.5)/\Ozero...(0.5)kakoe-to, ..(0.3) (Ili /\rečka,  ili /\ozero,  no po-moemu \ozero,  potomu čto’..(0.2) kak-to-oW...(0.6) \malen’koe takoe,  \nebol’šoe.) ....(1.0) ’i-iH...(0.7) čerez /nego..(0.3) kak-to \brevno kakoe-to,  tipa \mosta.

11 11 Night Dream Stories  Corpus of spoken Russian stories  Speakers: children and adolescents  Subject matter: retelling of night dreams  Discourse type: monologic narrative (personal stories)  Joint study with Vera Podlesskaya and a group of our graduate students

12 12 Segmentation (lines) ....(1.5)/\Озеро...(0.5)какое-то, Lakesome ..(0.3) (Или /\речка, Eitherriver  или /\озеро, orlake  но по-моему \озеро, butI guesslake  потому что’..(0.2)как-то-оw becausesomehow...(0.6)\маленькоетакое, smallsuch  \небольшое.) minor ....(1.0)’и-иh...(0.7)через/него andacrossit..(0.3)как-то\бревнокакое-то, somehowlogsome  типа\моста. likebridge ....(1.5)/\Ozero...(0.5)kakoe-to, ..(0.3) (Ili /\rečka,  ili /\ozero,  no po-moemu \ozero,  potomu čto’..(0.2) kak-to-oW...(0.6) \malen’koe takoe,  \nebol’šoe.) ....(1.0) ’i-iH...(0.7) čerez /nego..(0.3) kak-to \brevno kakoe-to,  tipa \mosta.

13 13 Pauses ....(1.5)/\Озеро...(0.5)какое-то, Lakesome ..(0.3) (Или /\речка, Eitherriver  или /\озеро, orlake  но по-моему \озеро, butI guesslake  потому что’..(0.2)как-то-оw becausesomehow...(0.6)\маленькоетакое, smallsuch  \небольшое.) minor ....(1.0)’и-иh...(0.7)через/него andacrossit..(0.3)как-то\бревнокакое-то, somehowlogsome  типа\моста. likebridge ....(1.5)/\Ozero...(0.5)kakoe-to, ..(0.3) (Ili /\rečka,  ili /\ozero,  no po-moemu \ozero,  potomu čto’..(0.2) kak-to-oW...(0.6) \malen’koe takoe,  \nebol’šoe.) ....(1.0) ’i-iH...(0.7) čerez /nego..(0.3) kak-to \brevno kakoe-to,  tipa \mosta.

14 14 Pitch accents ....(1.5)/\Озеро...(0.5)какое-то, Lakesome ..(0.3) (Или /\речка, Eitherriver  или /\озеро, orlake  но по-моему \озеро, butI guesslake  потому что’..(0.2)как-то-оw becausesomehow...(0.6)\маленькоетакое, smallsuch  \небольшое.) minor ....(1.0)’и-иh...(0.7)через/него andacrossit..(0.3)как-то\бревнокакое-то, somehowlogsome  типа\моста. likebridge ....(1.5)/\Ozero...(0.5)kakoe-to, ..(0.3) (Ili /\rečka,  ili /\ozero,  no po-moemu \ozero,  potomu čto’..(0.2) kak-to-oW...(0.6) \malen’koe takoe,  \nebol’šoe.) ....(1.0) ’i-iH...(0.7) čerez /nego..(0.3) kak-to \brevno kakoe-to,  tipa \mosta.

15 15 Tempo: wide and narrow scope ....(1.5)/\Озеро...(0.5)какое-то, Lakesome ..(0.3) (Или /\речка, Eitherriver  или /\озеро, orlake  но по-моему \озеро, butI guesslake  потому что’..(0.2)как-то-оw becausesomehow...(0.6)\маленькоетакое, smallsuch  \небольшое.) minor ....(1.0)’и-иh...(0.7)через/него andacrossit..(0.3)как-то\бревнокакое-то, somehowlogsome  типа\моста. likebridge ....(1.5)/\Ozero...(0.5)kakoe-to, ..(0.3) (Ili /\rečka,  ili /\ozero,  no po-moemu \ozero,  potomu čto’..(0.2) kak-to-oW...(0.6) \malen’koe takoe,  \nebol’šoe.) ....(1.0) ’i-iH...(0.7) čerez /nego..(0.3) kak-to \brevno kakoe-to,  tipa \mosta.

16 16 Other prosodic phenomena ....(1.5)/\Озеро...(0.5)какое-то, Lakesome ..(0.3) (Или /\речка, Eitherriver  или /\озеро, orlake  но по-моему \озеро, butI guesslake  потому что’..(0.2)как-то-оw becausesomehow...(0.6)\маленькоетакое, smallsuch  \небольшое.) minor ....(1.0)’и-иh...(0.7)через/него andacrossit..(0.3)как-то\бревнокакое-то, somehowlogsome  типа\моста. likebridge ....(1.5)/\Ozero...(0.5)kakoe-to, ..(0.3) (Ili /\rečka,  ili /\ozero,  no po-moemu \ozero,  potomu čto’..(0.2) kak-to-oW...(0.6) \malen’koe takoe,  \nebol’šoe.) ....(1.0) ’i-iH...(0.7) čerez /nego..(0.3) kak-to \brevno kakoe-to,  tipa \mosta.

17 17 Prosody and sentence  Does spoken language consist of sentences?  Sheer facts:  Spoken language is the primary form of language  Spoken language does not contain periods, question marks and other explicit signals of sentence boundaries  Research question:  Is sentence, as a theoretical construct, as identifiable and as basic for the primary form of language as it is (or as it is thought to be) for written language?

18 18 Transitional continuity  Term by J. DuBois et al  Alternative term by Sandro V. Kodzasov: phase  Discourse semantic category: ‘end’ vs. ‘non-end’ (=expectation of a forthcoming end)  End of tentative sentence – falling tonal accent  Non-end – rising tonal accent

19 19 A canonical example of the transitional continuity distinction z57:15-16 ..(0.4) /\Мы-ы’..(0.4) \как бы за них /взя-ались,..(0.4) /\My-y’..(0.4) \kak by za nix /vzja-alis’, We sort of at them got.hold ...(0.5) и-и ввь= ||..(0.2) полетели \вве-ерх. ...(0.5) i-i vv’= ||..(0.2) poleteli \vve-erx.  and flew upward  Rising (“comma”)  Non-end  Falling (“period”)  End  If things were that easy, sentence would be uncontroversial

20 20 Uncanonical situation: Non-end with a falling tonal accent ....(1.5)/\Озеро...(0.5)какое-то, ..(0.3) (Или /\речка,  или /\озеро,  но по-моему \озеро,  потому что’..(0.2)как-то-оw...(0.6)\маленькоетакое,  \небольшое.) ....(1.0)’и-иh...(0.7)через/него..(0.3)как-то\бревнокакое-то,  типа\моста. ....(1.5)/\Ozero...(0.5)kakoe-to, Lakesome ..(0.3) (Ili /\rečka, Eitherriver  ili /\ozero, orlake  no po-moemu \ozero, butI guesslake  potomu čto’..(0.2) kak-to-oW becausesomehow...(0.6) \malen’koe takoe, smallsuch  \nebol’šoe.) minor ....(1.0) ’i-iH...(0.7) čerez /nego andacrossit..(0.3) kak-to \brevno kakoe-to, somehowlogsome  tipa \mosta. likebridge

21 21 The problem of two kinds of falling  The existence of non-final falling calls relevance of sentence into question  However, the distinction between two kinds of falling is very systematic  The two kinds of falling:  are prosodically distinct  have distinct discourse functions

22 22 Prosodic criteria of the final vs. non-final falling distinction 1.Target frequency band 2.Post-accent behavior 3.Pausing pattern 4.Reset vs. latching 5.Steepness of falling 6.Interval of falling

23 23 Target frequency band  Final falling (“period”): targets at the bottom of the speaker’s F0 range  Non-final falling (“faling comma”): targets at level several dozen Hz (several semitones) higher

24 24 F0 graph for the “lake” example \ozero, \malen’koe \nebol’ \brevno kakoe \mosta. takoe, šoe.-to,

25 25 Representation of EDU continuity types in corpus

26 26 The status of sentence  In the speech of most speakers final falling is clearly distinct from non-final patterns  Final intonation, expressly distinct from non-final intonation (both rising and falling), makes the notion of sentence valid for spoken discourse  Speakers “know” when they complete a sentence and when they do not  Apparently, spoken sentences are the prototype of written sentences  However, identification of sentences is possible only on the basis of a complex analytic procedure

27 27 Conclusions on prosody and sentence  Sentence is an intermediate hierarchical grouping between a whole discourse and an EDU (roughly, clause)  Sentence is a complex, non-elementary unit of spoken language  These conclusions, possible only due to prosodic analysis, are of prime importance for linguistic theory  The notion of sentence, so salient in theories restricted to the verbal component alone, can only be evaluated relying on prosodic evidence

28 28 II. GESTURE  In the course of linguistic communication, it is not just that the speaker speaks and the addressee listens  In addition, the speaker displays, and the addressee observes  Gesture  Gaze  Mimics  Posture  Proxemics  Cultural symbolism  (see, for example, Крейдлин 2002, Бутовская 2004)

29 29 Gestures  Gestures are kinetic behaviors of arms and other limbs, capable of conveying meaning from speaker to addressee.  Among the various types of gestures (see e.g. McNeill 1992) pointing gestures are one of the most salient types.

30 30 Pointing  Возьмите игрушки там!

31 31 Elements of a canonical pointing act

32 32 Phylogeny and ontogeny  Appear an exclusive property of humans (Tomasello et al. 2007)  Are a very ancient gesture type (Крейдлин 2007)  Appear at the end of the first year  Can participate in binary multimodal constructions “word + gesture”, such as open POINT (Butcher and Goldin-Meadow 2000)

33 33 Reference and pointing  Reference is a fundamental linguistic phenomenon, accounting for about every third word in running discourse  Studies of reference (deixis, anaphora, etc.) are among the central concerns of modern linguistics  Pointing is the developmental source of reference

34 34 Pointing, deixis, and exophora  Deixis is the most widely recognized function of pointing  However, quite frequently pointing is associated with exophora, that is mention of perceptually activated referents (O'Neill 1996, Levy 2000: 219, Nikolaeva 2003 )

35 35 Exophoric reference (from Nikolaeva 2003)  a.My s Anatoliemuže mnogolet očen’rabótaem,  e.onmnogo raz zavjázyval, ‘Anatolij and I have been working together for many years, he was winding it up (drinking) many times’

36 36 Anaphora  Anaphora (reference to items activated by prior discourse) is secondary to exophora (reference to items activated by perceptual availability)  Exophora is the ontological source of anaphora  Anaphora occasionally occurs with pointing

37 37 Pointing and prosody  Pointing and accentuation are analogous phenomena, both associated with making an item salient  Levy (2000): energy expenditure  Nikolaeva (p.c.): pointing invariably cooccurs with accent

38 38 Substitution: Referent vs. demonstratum  Reference to non-specific items: Vot počemu my i obraščàemsja poroj k psixologam. ‘This is why we address psychologists now and then’  This phenomenon is known as deferred ostension, analogic deixis, ostensive metonymy, etc.  In substitution, reference does not have to be non-specific He got a big scar here (pointing to one’s cheek) (Levelt 1989)

39 39 Virtual pointing  Pointing to imaginary targets  cf. Buehler’s Deixis am Phantasma, McNeill’s abstract pointing

40 40 Frequency in two discourse types  Nikolaeva 2003 (TV shows):  5.4 poinring gestures per 100 EDUs  2.7 are virtual pointing  Nikolaeva p.c (retelling of a film):  4.2 pointing gestures per 100 EDUs  All are virtual pointing  Pointing in exophora/anaphora is as frequent as in deixis

41 41  a.… əəKogdaon exal po= podoróge,  b.on əə mm…poravnjalsjas dévočkoj, ‘As he rode along the road, he passed a girl ’

42 42  d.onzasmotrélsja naneë, ‘he gaped at her’

43 43 Establishment of spatial relations  By illustrative gestures, as in the previous example  By verbal devices a. i naprotivmenja sideli dve devočki-mulátki, y. vot êtidvedevočkinaprotivi jà, ‘‘And across from me sat two brown-skinned girls, these two girls and I ’  There is no difference for the referential system what was used to establish spatial relations  Verbal and gestural material is jointly used to convey the inner cognitive representation from the speaker to the addressee

44 44 Conclusions on pointing and reference  The pointing gesture is the developmental source of reference  The use of pointing is intimately connected to reference  Reference, a central linguistic phenomenon, cannot be understood if we fail to take gesture into account

45 45 III. Relative contribution of three information channels Discourse Vocal channelsVisual channel Verbal channel Prosodic channel

46 46 What is the contribution of different channels?  Traditional approach of mainstream linguistics: the verbal channel is so central that prosody and the visual channel are at best downgraded as “paralinguistics”  Applied psychology  It is often stated that (figures go back to Mehrabian 1971): body language conveys 55% of information prosody conveys 38% of information the verbal component conveys 7% of information  «Words may be what men use when all else fails» (Крейдлин 2002: 6)  Who is right?

47 47 Experimental study  Isolate three information channels  Present a sample discourse in all possible variants (2 3 =8)  Present each of the eight variants to a group of subjects  Assess the degree of understanding in each case  El’bert 2007, Kibrik and El’bert 2008

48 48 Experimental material  Russian TV serial “Tajny sledstvija” – “Mysteries of the investigation”  Experimental excerpt: 3 min. 20 sec.  Preceded by a 8 minutes context (that starts from the beginning of the series)  The excerpt fully consists of a conversation, to ensure that we are testing the understanding of discourse rather than of the film in general  Two vocal channels have been separated:  verbal alone – running subtitles  prosodic alone – superimposed filter creating the “behind a wall” effect  Subjects:  99 participants, divided into 8 groups  Native speakers of Russian  Each group comprised 10 to 17 subjects

49 49 Full version

50 50 Visual + verbal channels

51 51 Visual + prosodic channels

52 52 Procedure  Every subject was instructed to watch the context and the experimental excerpt and then answer a set of questions concerned with the experimental excerpt alone  Questionnaire was constructed in accordance with the received principles of test tasks (Panchenko 2000)  23 questions in questionnaire  A subject was supposed to choose only one answer out of four listed variants  What Tamara Stepanovna offers Masha before the beginning of the conversation:  a. to take off her coat  b. to have a cup of tea  c. to have a seat  d. to have a drink  Percentage of correct answers is used as an assessment of a subject’s degree of understanding

53 53 Results Group number Experimen- tal material OriginalSoundSubtitles + video Prosody + video SubtitlesProsodyVideoNothing (context only) Information channels verbal prosodic visual verbal prosodic verbal visual prosodic visual verbalprosodicvisual[none] Number of information channels Mean % of correct answers 87,4%70,4%73,9%51,2%72,0%51,1%61,7%38,3%

54 54 Each of the three information channels, taken in isolation, is quite informative Group number Experimen- tal material OriginalSoundSubtitles + video Prosody + video SubtitlesProsodyVideoNothing (context only) Information channels verbal prosodic visual verbal prosodic verbal visual prosodic visual verbalprosodicvisual[none] Number of information channels Mean % of correct answers 87,4%70,4%73,9%51,2%72,0%51,1%61,7%38,3%

55 55 The hierarchy of informativeness: verbal > visual > prosodic Group number Experimen- tal material OriginalSoundSubtitles + video Prosody + video SubtitlesProsodyVideoNothing (context only) Information channels verbal prosodic visual verbal prosodic verbal visual prosodic visual verbalprosodicvisual[none] Number of information channels Mean % of correct answers 87,4%70,4%73,9%51,2%72,0%51,1%61,7%38,3%

56 56 Combining the verbal channel with one additional channel does not increase the percentage of correct answers Group number Experimen- tal material OriginalSoundSubtitles + video Prosody + video SubtitlesProsodyVideoNothing (context only) Information channels verbal prosodic visual verbal prosodic verbal visual prosodic visual verbalprosodicvisual[none] Number of information channels Mean % of correct answers 87,4%70,4%73,9%51,2%72,0%51,1%61,7%38,3%

57 57 The combination ‘prosodic plus visual’ (group 4) leads to significantly lower result than in other pairs of channels (groups 2 and 3). Group number Experimen- tal material OriginalSoundSubtitles + video Prosody + video SubtitlesProsodyVideoNothing (context only) Information channels verbal prosodic visual verbal prosodic verbal visual prosodic visual verbalprosodicvisual[none] Number of information channels Mean % of correct answers 87,4%70,4%73,9%51,2%72,0%51,1%61,7%38,3%

58 58 Relative contribution of the three channels  For the sake of simplicity, assume that all three channels are independent  ( =185)/100  Results:  Verbal channel 39% (72:1.85≈39),  Prosodic channel 28% (51,1:1.85≈28),  Visual channel 33% (61,7:1.85≈33),

59 59 Conclusions about the relative weight of three information channels  All information channels are highly significant  the traditional linguistic viewpoint is erroneous  The verbal channel is the leading one  the viewpoint popular in applied psychology is erroneous  Information from the prosodic and the visual channels is primarily used through integration with the verbal channel, at least for this discourse type

60 60 IV. Signed languages NATURAL LANGUAGES SPOKEN SIGNED DEAF SIGN LANGUAGES  natural, full-fledged human languages  visual-spatial languages  use hands and arms, facial expressions, eye gaze, head and body posture to encode linguistic information  manual signs are produced in a three-dimensional space immediately in front of the signer – the signing arena  121 sign languages (http//:www.ethnologue.com) American Sign Language, Russian Sign Language …

61 61 Reference in RSL  Prozorova 2006, Kibrik and Prozorova 2007  Goal: to characterize referential choice of a deaf sign language as contrasted to that of spoken languages

62 62 RSL data collection  ‘The Pear Stories’ Film (Chafe 1980)  Corpus of 10 video-recorded RSL narratives based on the retellings of the Pear Film  Speakers:  6 men and 4 women  age  all based in Moscow  7 animate referents in the Pear Film  657 clauses  542 referential expressions (animate)

63 63 Deictic demonstrative reference in RSL  operates in the perceived space P  deictic expressions: pointing signs  pointing with an index finger towards the intended referent (2) DEM cat ILL‘He is ill’

64 64 Major anaphoric options in RSL  Full NPs (114)  Zero expressions (401)  Demonstratives (27)

65 65 Full NP BOY YOUNG AGE CYCLE‘A young boy is riding a bicycle’

66 66 Full NP BOY YOUNG AGE CYCLE‘A young boy is riding a bicycle’

67 67 Zero expressions 1. BOY YOUNG AGE CYCLE 2. Ø boy STOP 3. Ø boy HUMAN-STAND rightdown 4. Ø boy LOOK rightdown P-E-A-R 1. A young boy is riding a bicycle. 2. He stops. 3. He stands upright. 4. He sees the pears.

68 68 Anaphoric zero reference  Interlocutors’ shared cognitive representation contains not only perceived referents, but also referents conceived of (remembered or imagined)  We call this representation the conceived space C  Mentioning referents that are present, or activated, in the conceived space is what is known as anaphora  Anaphoric referential choice depends on a referent’s activation in the conceived space:  High  zero  Low  full NP

69 69 Demonstrative 1. Ø boy CYCLE 2. Ø boy GO signer  forward AWAY signer  forward 3. DEM man right SEE NEG 4. Ø man PICK-ROUND 1. He cycles. 2. He goes away. 3. That one doesn’t see. 4. He picks pears

70 70 Anaphoric demonstrative reference  In signed discourse the signer maps referents from the inner conceived space C onto the external signing arena  Mapping includes various parameters of referents:  locations  orientations  physical interactions  even abstract relations between them  Thus a constructed space C’ is created, inhabited by referents conceived of

71 71 How are locations of referents established in the constructed space?  Signed discourse takes place in the three- dimensional signing arena  The topology of the signing arena isomorphically represents the topology of the scenes, remembered by signers from the film  The signer establishes the locations of referents in his signing arena  These locations are isomorphic to the locations of the referents in the film, as remembered by the signer

72 72 An episode from the Pear Film

73 73 A retelling 1. ONE-MOVE front  signer MAN i 2. ONE-MOVE front  signer SHE-GOAT 3. BOY GIRL UNCLEAR 4. SHE-GOAT 5. Ø goat TWO-HORN HAVE.NEG 6. DEM i front PULL 1. A man is coming, 2. with a she-goat. 3. Male, female – it is unclear. 4. It’s a she-goat: 5. It has no horns. 6. This one is pulling it.

74 74 Anaphoric demonstratives  Once the signer has explicitly indicated the location/path of a referent, nominal demonstratives may be used for further mentions of this referent  Thus demonstratives are the basic device used for repeated mention of referents in the constructed space  Formally they are the same as deictic demonstratives  Often called ‘personal pronouns’ in the literature  Demonstratives are based on the mechanism of virtual pointing, but it is conventionalized in RSL  What is a kind of an ad hoc, fluid device in spoken languages, is an established, nearly lexical device in RSL

75 75 Two discourse factors and anaphoric referential devices factor 1:RD=1 RD=2RD=3+ TOTAL factor 2: Ant=SAnt=O full NP<1 %33 %14 %57 %59 zero NP99 %42 %67 %27 %401 nominal DEM <1 %25 %19 %16 %27 TOTAL 346 (100%) 24 (100%) 43 (100%) 74 (100%) 487

76 76 Use of zero expressions under RD > 1  49 usages (12% of all zeroes)  Pragmatic and semantic clues that help to identify the referent of a zero expression:  certain predicates associated with a particular referent (RIDE-BICYCLE; HOLD-BICYCLE)  The process of role-shifting (Padden 1986):  by shifting (rotating) the body and changing his/her facial expression the signer shows that s/he is currently “acting” for one of the referents

77 77 Role-shifting 1. Ø boy LOOK down 2. Ø boy BE-ABOUT ONE PEAR ONE TAKE-ROUND 3. Ø boy LOOK up role-shifting 4. DEM up man PICK-ROUND role-shifting 5. Ø boy LOOK down 6. Ø boy TAKE-ROUND 1. He [the boy] looks down. 2. He is about to take one pear. 3. He looks up. role-shifting 4. That one (the man) is picking pears. role-shifting 5. He (the boy) looks down. 6. He takes one.

78 78 Referential function of nominal demonstratives  Nominal demonstratives are not particularly sensitive to discourse factors: factor 1:RD=1 RD=2RD=3+ TOTAL factor 2: Ant=SAnt=O nominal DEM <1 %25 %19 %16 %27

79 79  In case of intermediate referent activation, full NPs and demonstratives compete  In case of low activation (RD=3+) full NPs strongly prevail (57%)  Apparently, information on the location of a referent in the constructed space can be assumed available to the addressee only for a limited time Full NPs vs nominal demonstratives

80 80 Full NPs vs demonstratives 1. Ø boy CYCLE 2. Ø boy OBJECT-MOVE signer  forward 3. Ø boy GO-AWAY signer  left-forward 4. DEM up MAN STILL PICK-PEAR 5. CYCLE DEM boy front 6. Ø boy OBJECT-MOVE signer  forward 1. He (the boy) is cycling. 2. He is riding forward. 3. He goes away. 4. That man is still picking pears. 5. This one is cycling. 6. He is riding forward. 11 2

81 81 Conclusions on reference in RSL  Types of referential devices and factors of reference are analogous to those of spoken languages  Some devices, only embryonically present in spoken languages, are strongly employed in RSL:  virtual pointing  role-shifting  This is apparently due to the fundamentally spatio-visual character of RSL  Studying signed languages gives us a new perspective on spoken languages  Recognition of two fundamental types of languages, spoken and signed, appears indispensable for a general theory of language

82 82 V. A wider picture  The world surrounding us is multimodal  We are multimodal organisms  Obviously language and communication are mutimodal  As it often happens, those specializing in applied fields have understood the importance of multimodality before pure scholars and theorists

83 83 Multimodality in technology  TV is superior to radio  Multimodal communication devices  Internet, especially Web 2.0, is all multimodal  Multimodal GPS

84 84  The multimodal flight finder enables rapid task completion by enabling the user to interact via a multiplicity of user interaction modalities

85 85 Stages of multimodal integration, from Cohen and Oviatt 2006

86 86 Multimodality in biological sciences  “Within biology, experimental psychology, and cognitive neuroscience, a separate rapidly growing literature has clarified that multisensory perception and integration cannot be predicted by studying the senses in isolation.” (Cohen and Oviatt 2006)

87 87 Multimodality in communication studies and semiotics  Kress G & van Leeuwen T (2001). Multimodal discourse: the modes and media of contemporary communication. London: Arnold.  ‘‘A multimodal approach assumes that the message is ‘spread across’ all the modes of communication. If this is so, then each mode is a partial bearer of the overall meaning of the message. All modes, speech and writing included, are then seen as always partial bearers of meaning only. This is a fundamental challenge to hitherto current notions of ‘language’ as a full means of making meaning’’ (Kress, 2002: 6).

88 88 Multimodal Analysis Lab (Singapore): collaboration of social scientists and computer scientists

89 89 Multimodality in computational linguistics  Gibbon D, Mertins I & Moore R (eds.) Handbook of multimodal and spoken dialogue systems: resources, terminology and product evaluation. Dordrecht: Kluwer. 2000

90 90 Multimodal corpora  LREC-2008 (Language Resources and Evaluation Conference)  Blache P., Bertrand R., Ferré G Creating and exploiting multimodal annotated corpora.  Gallo C.G., Jaeger T.F., Allen J., Swift M Production in a multimodal corpus: How speakers communicate complex actions  Kitazawa Sh., Kiriyama Sh., Kasami T., Ishikawa Sh., Otani N., Horiuchi H., Takebayashii Y A Multimodal infant behavior annotation for developmental analysis of demonstrative expressions

91 91 Synthesis  LeVine P & Scollon R (eds.) Discourse and technology: multimodal discourse analysis. Washington, DC: Georgetown University Press. 2004

92 92 Conclusions  “Normal” linguists, researching conventional verbal material, need to understand that further progress in linguistics is impossible if one ignores the multimodality of language  Language in the understanding of the 20 th century mainstream linguistics is an abstraction, very remote from reality. We live in the multimodal world, this is where language evolved and where it functions, and this is what we need to realize if we want to understand it  Taking the multimodal perspective into account can help to adequately approach classical questions of narrow linguistics  Choice between continuing the habits of mainstream linguistics and swtiching to multimodality amounts to the choice between  Impoverishment vs. richness  Stagnation vs. innovation  Isolation vs. interdisciplinarity

93 93 Acknowledgements  Julia Nikolaeva  Vera Podlesskaya  Evgenia Prozorova  Ekaterina El’bert


Download ppt "MULTIMODAL LINGUISTICS: Directions of research Andrej A. Kibrik (Institute of Linguistics, RAN) CML-2008 Montenegro, September 2008."

Similar presentations


Ads by Google