Presentation is loading. Please wait.

Presentation is loading. Please wait.

Yet More Symbol Systems; Relevant User Issues Seminar on Speech and Language Processing for Augmentative and Alternative Communication Spring 2010 Mahsa.

Similar presentations

Presentation on theme: "Yet More Symbol Systems; Relevant User Issues Seminar on Speech and Language Processing for Augmentative and Alternative Communication Spring 2010 Mahsa."— Presentation transcript:

1 Yet More Symbol Systems; Relevant User Issues Seminar on Speech and Language Processing for Augmentative and Alternative Communication Spring 2010 Mahsa A.Yarmohammadi

2 Papers for Today Patel, R., Schooley, K. & Radhakrishnan, R. (2006). Comparison of semantic versus syntactic message formulation: A pilot study. Assistive Technology Outcomes and Benefits, Volume 3, Ottem, E., Use of pictographic-articulatory symbols to promote alphabetic reading in a language-impaired boy: Case study. Augmentative and Alternative Communication 17 (March), Venkatagiri, H. S., Clinical implications of an augmentative and alternative communication taxonomy. Augmentative and Alternative Communication 18 (March), 45-57

3 Main topics of papers Patel et al. (2006) ▫ Compare message formulation using a semantic frame-based approach versus the conventional syntactic approach Ottem (2001) ▫ Use Blissymbols and a new type of pictographic- articulatory symbol to teach reading to a 9-year- old boy with specific language impairment Venkatagiri (2002) ▫ Offer a taxonomy of certain AAC system components (signs) as a framework for clinical decisions related to AAC

4 Patel et al. (2006) The use of picture symbols for communication has gained increasing appeal Picture symbols paired with voice output serve as a powerful means for expressing one’s ideas Most commercially available VOCAs use a linear(syntactic) approach to message construction ▫ “I” + “want” + “soda” ▫ At least basic knowledge of syntax is required ▫ No research support the notion that POS ordering of vocabulary icons assists in language learning

5 Approach The use of semantic schema -associated with the verb- may facilitate message construction by ▫ making the rules of grammar more transparent ▫ probably reducing keystrokes ▫ easing the cognitive burden ▫ improving access to vocabulary Compare message formulation using a semantic frame-based approach (iconCHAT) versus the conventional syntactic approach (Default) in terms of: ▫ accuracy, speed, complexity, user preference

6 Method Eight developing children between 7 and 10 without speech and language impairments The child was shown 18 pictures and 4 three- step sequences The child used the iconCHAT and Default prototypes to describe what was happening in the scene he/she was shown Both prototypes had the same vocabulary size, overall display size, icon size, graphic symbol set, and the same type of synthetic voice output A survey was done to assess ease of use and the satisfaction with both prototypes

7 iconCHAT

8 Default

9 Stimuli

10 Dependent measures Construction time per linguistic unit Time per button click Button clicks per linguistic unit Percentage accuracy ▫ semantic accurateness of the subject, verb, object Complexity ▫ the use of modifiers, one point for each modifier Use of quick reference panel ▫ one point for using the panel to select an agent or pronoun, for iconCHAT only Reuse of prior message constructions ▫ one point for each reuse, for iconCHAT only

11 Results Some participants were faster and required fewer keystrokes using iconCHAT, others performed better using Default

12 Results All participants were 100% accurate in terms of grammatical completeness using both prototypes

13 Results Sentences structured in both prototypes were either in the SVO, SVMO, or SVMMO forms

14 Results The heterogeneity of participant strategies for message formulation is evident in the results

15 Results No statistically significant differences between the prototypes in terms of time per linguistic unit

16 Results Although there was a statistically significant difference in time per button click between the two prototypes, the difference was rather small

17 Results 5 sec/click in iconCHAT, 4 sec/click in Default

18 Results A statistically significant difference in the number of button clicks per linguistic unit, iconCHAT requires fewer clicks

19 Results Participants varied in the complexity of utterances they generated

20 Results All participants used the quick reference panel to a large extent

21 Results Only one child reused a prior message construction

22 Survey results Various results across participants 2 participants thought that iconCHAT was easier to use, 4 thought that Default was easier to use, 2 rated both equally 5 participants said that they used the semantic schema as opposed to the serially construction 6 participants mentioned relying on the semantic schema rather than the serial ordering 4 participants mentioned that they liked how iconCHAT “showed what needed to be filled in”

23 Survey results Some positive feedbacks regarding Default: ▫ faster to learn ▫ liked the details (modifiers) along the bottom ▫ liked having all the categories on one page Very little negative feedback for either prototype ▫ 2 comments on the need for more vocabulary ▫ One participant noted that having to select the verb first was odd in the beginning because it was “not what I’m used to” but that “it was cool though, I guess” because it “helped me.”

24 Conclusion Semantic-based approaches to message formulation are at least as effective in facilitating accurate and complete utterances for typically developing children. The 2-dimensional semantic schema and the serial ordering of icons may be beneficial to children who use AAC given that their constructions is limited to single word and has atypical syntax illustrates the relationships between icons provides feedback regarding the output of the speech recognizer

25 Conclusion Semantic-based formulation may be especially useful for those children who struggle with grasping grammatical constructs ▫ semantic frame facilitate complete sentence production through visual cues

26 Future work Future studies need to include children who use AAC The stimuli should be expanded to include a broader range of sentence complexity To assess the usefulness of the reuse feature, multiple stimuli with the same predicate are necessary Include a modifier panel along the bottom of the iconCHAT interface

27 Ottem (2001) More than 80% of children with SLI experience problems with reading (Tallal et al.,1996) Reading problems of children with SLI are probably due to a deficit in phonological processing (Stanovich, 1998;Vellutino, 1979) To read using a grapheme-to-phoneme conversion (alphabetic) system, phoneme awareness is required Some studies support the view that articulatory awareness plays an important role in learning to read (Heilman et al.,1996;Montgomery,1981;Alexander et al.,1991) ▫ awareness of the movement of the position of one’s own articulators is important in both phonological awareness and the ability to grapheme-to-phoneme conversion ▫ The treatment of dyslexia should include speech therapy to teach individuals to be aware of their articulatory gestures

28 Ottem (2001) Graphic Representational Symbols (GRSs) may facilitate specific components of word awareness, but their overall impact on beginning reading may be minimal ▫ They don’t contain information about how the basic units of the orthographic system are pronounced by the articulators Can a GRS that contains visual information about articularoty gestures facilitate orthographic reading in a child with SLI?

29 Pictographic-articulatory system PAS was designed to portray the salient articulatory features of letter sounds PAS was used to teach letter sounds for alphabetic reading to a child with SLI Four phases of reading (Ehri, 1991): ▫ logographic phase: letters serve as visual cues ▫ transitional phase: letters provide alphabetic cues to the sounds of spoken words ▫ alphabetic phase: recode spellings phonologically into pronunciations ▫ orthographic phase: ability to read words without phonologically recoding constituent letters

30 Pictographic-articulatory system


32 Case study Erik is a child with SLI ▫ problems with the structural aspect of language ▫ very short and ungrammatical utterances at the age of 8 years ▫ poor word knowledge, unable to read any sight words ▫ limited knowledge of the letters of alphabet, unable to analyze or synthesize spoken or written words Use of Blissymbols and the PAS with Erik over a 2-year period is studied

33 Assessments Erik was assessed using some standard tests ▫ Wechsler Intelligence Scale for Children-Revised ▫ Wechsler Preschool and Primary Scale of Intelligence ▫ Reynell Developmental Language Scales ▫ Illinois Test of Psycholinguistic Abilities  Verbal Expression  Manual Expression  Sound Blending Erik was tested three times ▫ prior to the initiation of Blissymbols (8 years,2 months) ▫ prior to the introduction of the PAS (9 years,6 months) ▫ after termination of the use of the PAS (10 years, 3 months)

34 First assessment Initial scores on all of the verbal tests were quite low

35 Communication and Language Intervention: Blissymbols Erik was taught Blissymbols in order to ▫ improve his knowledge of the meaning of the words ▫ increase his vocabulary ▫ improve his verbal communication skills Teaching one 30-minutes session per day, for about 7 months He acquired more than 200 Blissymbols (150 nouns, 50 verbs, pronouns, adjectives, prepositions) He could read sentences composed of up to 7 Blissymbols

36 Second assessment Both his verbal knowledge and his phonological awareness were facilitated

37 Second assessment Erik’s ability to communicate was improved, he spoke more often, spoke longer sentences and more variable words

38 Reading Intervention: the PAS PAS was developed to proceed from Blissymbols to orthographic reading ▫ Blissymbols in combination with letters didn’t facilitate Erik’s ability to acquire beginning reading skills Teaching the PAS three hours per day in order to achieve four primary goals.

39 Reading Intervention: the PAS Promoting knowledge about the articulators ▫ teaching the articulators and their functions using models, pictures, mirror work, and play ▫ What happens in your mouth when you make the sound …? Promoting knowledge of the PAS symbols and their relationship to orthographic letters ▫ link two symbols to form a word using the pictogram cards ▫ read the words using PAS symbols Linking sounds and PAS symbols to words ▫ read two- and three-symbol words, then rearrange the cards into new words Matching PAS symbols to letters

40 The letters of the alphabet were paired with the PAS cards previously introduced A computer program for symbol-sound matching was used to practice the link between the sounds and the letters practice 30 min per day for 8 months

41 Matching PAS symbols to letters Early in the process ▫ Erik understood the meanings of the constituent parts of the PAS symbols ▫ He understood that each pictogram corresponded to a letter sound After 7 months ▫ Erik knew 17 PAS symbols and their letters ▫ He was able to read simple stories comprised of either PAS symbols or letters ▫ He reached the alphabetic phase of reading

42 Matching PAS symbols to letters After 8 months ▫ Erik refused to use the PAS any longer ▫ He could read without the aid of the symbols ▫ All of the remaining letters of the alphabet were taught orthographically only Over the next year ▫ Erik gradually reached the orthographic phase of reading ▫ He could fluently read quite complex instructions ▫ He still had difficulty interpreting the meaning of abstract words, because of his delayed conceptual development

43 Third assessment WISC-R and ITPA verbal expression results are the same as the second assessment (or slightly declined in scaled scores)

44 Third assessment The PAS didn’t appear to contribute to Erik’s conceptual or general cognitive development

45 Third assessment His comprehension of spoken language and phonological awareness had improved

46 Discussion Use of Blissymbols for 7 months didn’t facilitate Erik’s acquisition of reading, however it ▫ increased his interest in communication and improved his word knowledge ▫ promoted his understanding that words are basic units of language, so increasing his print awareness ▫ prepared him for the logographic phase of reading Use of the PAS facilitated Erik’s reading acquisition by directing his attention to salient parts of the sound system

47 Discussion Erik’s experience was sequential and hence additive, so a different outcome is possible if PAS use had proceed that of Blyssimbols Additional AAC research that examines the impact of different types of GRSs on reading performance is encouraged

48 Venkatagiri (2002) There are a huge number of AAC symbol/icon sets and symbol/icon retrieval techniques This paper offers a taxonomy of AAC system components Builds on and extends previous contributions to AAC taxonomy by Lloyd (1985), Lloyd and Fuller (1986), Lloyd and Blischak (1992), and Fuller, Lloyd, and Schlosser (1992)

49 Current AAC taxonomies A widely used taxonomy: “aided” / ”unaided” ▫ useful for distinguishing between the physical characteristics of AAC communication ▫ fails for differences in the complexity and scope A secondary-level distinction is “static” / “dynamic” ▫ confusion with respect to the use of these terms  more movement and change for dynamic communication, or  durability of message elements in time the term “symbol” includes all types of message elements involved in AAC communication, but ▫ fails to accommodate emotional states, comfort states, physical & psychological needs, cognitive states AAC traditionally has a speech-centered definition, however, it includes expressive communication in all of its forms

50 AAC taxonomy Reasons for classifying AAC system components ▫ formulate a comprehensive and coherent theory of AAC ▫ provide guidelines for clinical decision making  useful insight to the steps necessary to develop appropriate AAC systems At minimum, an AAC system consists of a set of signs and a retrieval method

51 Classification of AAC signs

52 Signs Communication takes place through signs Sign refers to all actions, events, and objects that have a communicative value ▫ objects, tokens, marks, images, movements, gestures, sound patterns, graphic patterns There are three main types of signs: ▫ indexes ▫ icons ▫ symbols

53 Indexes When the relationship between a sign and what it represents is one of cause and effect and/or physical proximity, then the sign is an index ▫ reflexive vocalizations  cry, moan, grunt, sigh ▫ facial expressions  happiness, sadness, anger, surprise ▫ body postures and movements Communication through indexical signs is unintentional Communication partner is responsible for recognizing the communicative value of indexes

54 Indexes It is useful to prepare a detailed list of indexical signs displayed by an AAC user

55 Indexes Visceral indexes ▫ Emotional states  fear, anger, happiness ▫ Comfort states  fatigue, pain, feeling cold Visceral indexes are not amenable to shaping through learning Skeletal indexes ▫ Physical and Psychological needs  hunger, companionship, toileting ▫ Cognitive states  alertness, excitement, anticipation Skeletal indexes may be modified through behavior modification and social learning procedures

56 Icons When the sign bears a physical resemblance to what it signifies, it is called an icon ▫ color, shape, size, texture, smell, sound, … Examples: some road signs (a drawing of a falling rock), pictographic writing used by many African tribes, pantomime, imitations of sounds of animals The association of icons with their referents requires some learning

57 Icons Transparent icons ▫ when a likeness of an object is used to represent that object  a picture of fire to represent fire Translucent icons ▫ icons that represent a property of the referent  a picture of fire to signify hot rather than fire itself Translucent icons involve greater learning demands than transparent ones Translucent icons must present salient physical or functional features of the referent, otherwise they are regarded as symbols Icons are well suited to represent everyday objects and events, not abstract concepts (courage), invisible objects (air), imaginary entities and events (miracle)

58 Icons Transparent or translucent icons are either ▫ Gestural  point to the nose to represent nose (transparent)  a hand placed under the nose to signify smell (translucent) ▫ Auditory  meow for the sound made by cats (transparent)  meow for cat, noise of snoring for sleep (translucent) ▫ Material  Pictographs  a painting, photograph, etc, of object or event(transparent) or for representing a salient physical property of object (translucent)  Miniatures  a toy bed to represent bed (transparent)  a small pillow to represent bed (translucent)

59 Symbols When the relationship between a sign and what it represents is arbitrary and conventional, it is called a symbol The relationship between a symbol and what it signifies is entirely learned Examples: traffic signs such as two-way traffic, religious symbols, political symbols (flags) Symbols are divided into verbal and nonverbal categories ▫ verbal: symbols used within the framework of natural language ▫ nonverbal: arbitrary and conventional signs that are not part of a natural language

60 Primary symbols Primary verbal symbols: ▫ speech (vocal) ▫ gestural language: sign languages (e.g., American Sign Language and signed English) Seamless and symbiotic relationship to cognition Cognition allows for producing meaningful spoken and gestural communication, at the same time, speech and gestural languages is used to organize one’s cognitive domain Primary nonverbal symbols: ▫ speech (vocal): grunts, cries, sighs, and other vocal sounds when used volitionally to represent arbitrary meanings ▫ gestural language: idiosyncratic and commonly understood gestures

61 Secondary symbols Secondary verbal symbols: symbols used for writing ▫ Graphic: traditional alphabetic or other type of graphic writing system ▫ Embedded-graphic: orthographic symbols in another modality  Auditory  Morse code  Tactile-visual: three-dimensional cutouts of letters  Visual-gestural  finger spelling Secondary nonverbal symbols: artificial symbol sets ▫ Tactile: abstract 3-dimensional graphic symbols recognized through tactile channels ▫ Auditory: arbitrary tones and noises used for communication, have not been used in AAC so far ▫ Visual: abstract graphic symbols with assigned meaning  many Blissymbols, color-coded geometric shapes, Lexigrams

62 Flowchart of decision making The flowchart can be used to select indexes, icons, and symbols for an AAC system AAC evaluation and system development should occur simultaneously rather than consecutively It shows the order of integrating the different types of signs into AAC system ▫ the rate of integration depends on the user’s capabilities, motivations, and communication needs

63 Indexes 1 2 Primary nonverbal symbols (speech) 3 Primary nonverbal symbols (gestural) 4 Primary nonverbal symbols (speech) 5 Gestural symbols 6 Secondary verbal symbols 7 Material icons

64 Flowchart of decision making Indexes, volitional vocalizations, speech, gestures and gestural languages are preferred to material icons communication However there is a lack of research on indexical communication ▫ it deals with vital aspects of well-being ▫ care givers are primarily responsible for indexical communication ▫ it is highly productive Disadvantages of material icons ▫ problems of portability ▫ availability ▫ adaptability ▫ scope

65 Conclusion Variables when selecting symbol/icon sets ▫ scope: what meaning may be expressed ▫ clarity: the degree of precision of meanings ▫ rate: the amount of information transferred/time ▫ learnability: the amount of time and effort to learn the skills ▫ cognitive and physical ease of message preparation ▫ availability: when communication can take place ▫ ease of comprehension for the partner Selection is consistent with the physical and cognitive capabilities of the AAC user Importance of these factors varies depending the needs and abilities of users and contexts

66 Conclusion Indexes (e.g., yawning and squirming) and primary nonverbal symbols (e.g., grunts, idiosyncratic gestures) should be fully exploited in an AAC system ▫ high rate, high availability, cognitive simplicity, ease of physical effort Gestural and auditory icons (e.g., commonly understood gestures) should be important components of an AAC system ▫ highly learnable, early acquisition by developing children Primary verbal symbols (i.e., speech and gestural languages) can be helpful in AAC systems ▫ high rate on scope, clarity, speed, comprehensibility, availability, however ▫ they are more cognitively complex and require fine motor coordination Material icons (e.g., line drawings and miniatures) should be integrated if the above signs fail to completely meet the client’s needs Secondary nonverbal symbols are problematic at best in AAC systems ▫ poor in scope, rate, clarity, comprehensibility, availability, learnability

Download ppt "Yet More Symbol Systems; Relevant User Issues Seminar on Speech and Language Processing for Augmentative and Alternative Communication Spring 2010 Mahsa."

Similar presentations

Ads by Google