The Edmonton Narrative Norms Instrument (ENNI) -- Update ASHA presentation, New Orleans, November 2001 The Edmonton Narrative Norms Instrument (ENNI) -- Update Phyllis Schneider Rita Dubé Denyse Hayward The ENNI is available on-line at: http://www.rehabmed.ualberta.ca/spa/enni

ASHA presentation, New Orleans, November 2001 Overview of the ENNI The ENNI was designed as a measure of storytelling abilities with ‘local’ norms Story stimuli: pictures drawn to story scripts by a professional cartoonist Normative sample: 377 children aged 4-9 50 children with typical development per age group 10-17 children with specific language impairments per age group

ASHA presentation, New Orleans, November 2001 Choice of task In previous research, we found that children have more difficulty telling a good story from pictures when they did not hear the story first Telling stories from pictures seems to tap the child’s storytelling abilities Retelling stories seems to tap auditory memory more than storytelling ability

ASHA presentation, New Orleans, November 2001 Example: Simple story SETTING

ASHA presentation, New Orleans, November 2001 INITIATING EVENT

ASHA presentation, New Orleans, November 2001 ATTEMPT ASHA presentation, New Orleans, November 2001

ASHA presentation, New Orleans, November 2001 OUTCOME ASHA presentation, New Orleans, November 2001

ASHA presentation, New Orleans, November 2001 REACTIONS ASHA presentation, New Orleans, November 2001

The ENNI can be used to assess… ASHA presentation, New Orleans, November 2001 The ENNI can be used to assess… Edmonton children suspected of having language impairments without cognitive delay between the ages of 4-9 They are represented in the normative sample Other children suspected of having language impairment But with caution; they are not represented in the norms

The ENNI website http://www.rehabmed.ualberta.ca/spa/enni ASHA presentation, New Orleans, November 2001 The ENNI website http://www.rehabmed.ualberta.ca/spa/enni

ASHA presentation, New Orleans, November 2001 Psychometric update

Interscorer Reliability ASHA presentation, New Orleans, November 2001 Interscorer Reliability Naomi Beswick: study of interscorer reliability for the story grammar measure Asked 4 community SLPs to score SG Calculated intraclass correlations Results: A1: 92% (range: 83%-97%) A3: 96% (92%-98%) Thus: reliability is excellent

Discrimination between groups ASHA presentation, New Orleans, November 2001 Discrimination between groups Tests should discriminate between children with and without language impairment Discriminant analysis can reveal whether a measure discriminates between groups i.e., whether the groups are identified correctly Individual ENNI scores: moderate discrimination SG: 70-84% accuracy First Mention: 73-82% accuracy

Discrimination with all variables ASHA presentation, New Orleans, November 2001 Discrimination with all variables We calculated a discriminant analysis with all variables SG simple and complex FM MLCU Complexity Index TNW and NDW Word and Utterance errors

Results using all variables ASHA presentation, New Orleans, November 2001 Results using all variables Age Group Specificity (TD as TD) Sensitivity (SLI as SLI) Overall classification 4 96.0% 83.3% 93.5% 5 94.0% 92.9% 93.8% 6 98.0% 81.8% 95.1% 7 84.6% 92.1% 8 100% 94.1% 98.5% 9 80.0% 95.0%

ASHA presentation, New Orleans, November 2001 Results continued Two variables contributed less than the others: Word and Utterance errors Without these variables, discrimination was slightly lower, but still good Recommendation: Use several ENNI measures rather than just one or two

ASHA presentation, New Orleans, November 2001 ENNI questions

Questions Graesser (1985), Goldman (1985), Trabasso et al. (1988) ASHA presentation, New Orleans, November 2001 Production and questioning tasks place different demands on child Questioning tasks reduce processing demands child doesn’t have to generate own strategy reduces information needed to be held in working memory Questions are especially likely to help when child has not made inferences spontaneously Wanted to first give you a little background on the development of the questioning tasks We included a questioning task for the following reasons (read slide) We then needed to determine theoretical models from which to develop that questions We incorporated aspects of two models of story comprehension. Story Grammar and Causal Network models

ASHA presentation, New Orleans, November 2001 Theoretical models We used the Story Grammar model to create a set of questions The questions focused on SG units We also used Causal Network model Focus on causal relations that link SG categories to produce a causal network representation Read through slide

previous research on SG questions with children ASHA presentation, New Orleans, November 2001 previous research on SG questions with children Goldman (1985,1986) - Children 5 & 10 Children understood more (i.e., answered more questions correctly) than indicated in recall task 10 year olds included more ‘understood’ information in retell than 5 yr olds Questioning tasks showed that children understood more about the about stories they had retold However, this difference decreased with age, That is older children tended to include information they had understood in their story retells but younger children’s story retells did not include much of the information they demonstrated knowledge of when questioned about the story.

Types of questions used in previous studies ASHA presentation, New Orleans, November 2001 Types of questions used in previous studies Factual/literal questions (Merritt & Liles, 1987) Inferential Questions (Harris Wright & Newhoff, 2001; Merritt & Liles, 1987) Integrative inferences (Ellis Weismer, 1985; Crais & Chapman, 1987) Research where children with LI have been questioned about stories have utilized literal and inferential questions. Generally literal questions are questions where the answer is available in the story text or pictures whereas inferential questions require the child to make an inference because the information is not supplied directly in the text or pictures. Children with LI answer literal questions more successfully than inferential questions. Inferential questions that require children to integrate the entire story or large segments of the story are more difficult for children to answer ( e.g., what was the problem in that story) than inferential questions that relate to one particular event (e.g., why did story character feel happy). There have been no studies that asked children with LI importance judgment questions

Another type of question: Importance Judgements ASHA presentation, New Orleans, November 2001 Another type of question: Importance Judgements Stein & Glenn (1979) - children aged 6 & 10 Correlation between information considered important and information included in recall task van den Broek (1989) - adults; Bourg et al. (1997) - adults Causal Network model predicts that outcomes or resolutions will be considered most important van den Broek (1989 ) children aged 5 & 10 “importance” judgments reveal ability to distinguish central ideas in story Some researchers also investigated information children considered important in stories was included when retelling stories. And this in fact what Stein and Glenn did find. Importance questions have been used as a means of validating the causal network model. Since the story problem and the outcome or resolution have the most causal connections within stories these theorists predict that not only would children judge the problem and outcome as most important but would include in their stories. Similar to Stein and Glenn CNM theorists have fund this to be true for adults and children (7 yrs and older) CNM theorists also suggest that these type of questions also show a child’s ability to distinguish central ideas of the story.

ASHA presentation, New Orleans, November 2001 ENNI Questioning Task Children in the ENNI sample also participated in the questioning task Children seen 1-2 weeks after completion of the retell part of the ENNI Order of stories – randomized (chosen by the child) Child previewed all the story pictures Returned to beginning of story and asked questions related to each story picture We completed the questioning tasks after the completion of the story productions (PH - I don’t know if you want to highlight why we did that or not) The children selected the story book they would answer questions about randomly - so some children answered questions about the complex story followed by the simple story and visa versa. The children looked thought the story book to re-familiarize themselves with the story and then we returned to the beginning of the book and asked the first set of questions. These were a series of literal and inferential questions that asked about all the story elements from beginning to end of the story. Once we had completed that we asked the second question set which was the importance judgment question, In the next few slides we’ll show you the questions for story 1A so you can see how this was accomplished.

ASHA presentation, New Orleans, November 2001 So first we give a general instruction (read top section). On these slides we have shown the story picture, the specific questions, the story grammar category each question related to and the question designation literal or inferential. So we start at the beginning of the story and while the child has the picture to look at as the questions associated with that picture and then proceed to the next picture until we reach the end of the story book Read questions/designates etc..

ASHA presentation, New Orleans, November 2001 We then proceeded to ask the Importance Judgment question. Most important thing that happened in the story? We developed several prompts for this question as the youngest children and many of the children with LI had difficulty on this question.

ASHA presentation, New Orleans, November 2001 Continue to read questions

Results Simple and Complex Story ASHA presentation, New Orleans, November 2001 Results Simple and Complex Story Literal / Inferential Questions Importance Questions Story Questions / Story Production Data are presented for 4-8 year olds So we’ll show you some of our results. First we’ll take a look at how children answered the first set of questions (literal and Inferential), the second question set - the importance judgments and finally compare how well children did across questioning and production tasks.

ASHA presentation, New Orleans, November 2001 Simple Story (ENNI A1) Literal Questions Age Effects: [4 < 5 - 8]; [5 < 6 - 8]; [6=7=8] Language Status Effects: TD > LI This graph shows the percentage of literal questions TD children (light blue) and children with LI (dark blue) answered correctly in the simple (1 episode story) Can see we have age effects where the 4 year olds …. And Language status effects where TD children answered more questions correctly than children with LI.

ASHA presentation, New Orleans, November 2001 Simple Story (ENNI 1A) Inferential Questions Age Effects: [4 < 5 - 8]; [5 < 6 - 8]; [6=7=8] Language Status Effects: TD > LI Ser the same pattern when the children answered inferential questions about the simple story. TD children (light blue) and children with LI (dark blue) For all of the analyses we found the same age and language status effects where 4 year olds …. And where TD children answered more questions correctly than children with LI. For the remainder of the slides we’ll just show 4, 5 separately and combined the results for 6, 7, and 8 year olds.

Complex Story (ENNI 3A) Literal & Inferential Questions ASHA presentation, New Orleans, November 2001 Complex Story (ENNI 3A) Literal & Inferential Questions Here are the results for the complex story (3 episodes). Again you can see the improvement with age from 4-6 for all children and that TD children answer more questions correctly than children with LI. Can also see that answering more Literal questions are answered correctly than inferential questions

ASHA presentation, New Orleans, November 2001 Importance Questions Responses assigned a Story Grammar category Examples from complex story (ENNI 3A) When she threw it in there because if she didn't there wouldn't be no story [Consequence - Episode 1] When the very ending came when all of the people were happy [Reaction Episode 3] That the Dad tried to reach it [Attempt Episode 2] That super girl came [Initiating Event – Episode 3] To score the Importance Judgment questions we first categorized the children’s answers according to which SG element in the simple or complex story they related. Here are some example responses from children when asked what was the most important thing that happened in the story. And the SG element that their responses corresponded.

ASHA presentation, New Orleans, November 2001 Importance Questions Additional codes Moral Never to run at the pool or never to play with toys at the pool No Story Grammar (NSG) Just nice story They can't colour on the deck Don’t Know (D/K) It's a hard part {child prompted to look at pictures} I think my head got lost in thought. Now we also needed to develop some additional categories because not all children gave answers that fit neatly into the SG categories. These are the categories that basically encompassed the range of responses we obtained from the entire sample of children. So let’s take a look at how children did answering these questions for the simple and complex stories.

Importance questions: best answer? ASHA presentation, New Orleans, November 2001 Importance questions: best answer? What would be considered a good answer according to the Story Grammar Model? A response that corresponds to one of the core SG units Initiating Event Attempt Consequence Causal Network: Consequence -- has most causal connections to rest of story We didn’t score as right or wrong – looked for predominant responses at different ages

Simple Story—Most common responses to Importance question ASHA presentation, New Orleans, November 2001 Simple Story—Most common responses to Importance question Age Group Typical Development Language Impairment 4 year olds IE (24%) Don’t Know (40%) 5 year olds CON (50%) Don’t Know (28%) 6 year olds CON (49%) IE/Att/Con/NSG (18%) 7 year olds CON (39%) Att/Con/React (23%) 8 year olds CON (51%) Don’t Know (26%) This shows the most frequently occurring response for age and language status Recall the CNM theorists predicted that children would consider the problem and resolution and possibly story characters reactions to the outcome as the most important and as you can see the TD children did this (IE = problem) and (CON = resolution) But a different pattern emerged fro the children with LI. You can see that for three of the age groups the most frequently occurring response was Don’t know. Some of the 5 and 6 yr olds included the problem (IE) or resolution (CON) and story characters reaction to the outcome (REACT-G) but these responses were no more frequent that some other categories (I.e attempts and no story grammar responses)

Complex Story Story—Most common responses to Importance question ASHA presentation, New Orleans, November 2001 Complex Story Story—Most common responses to Importance question TD LI 4 year olds CON 1 (30%) Giraffe/NSG/DK (20%) 5 year olds ATT 3 (28%) CON 1 (21%) 6 year olds ATT 3 (42%) NSG (18%) 7 year olds CON 3 (28%) ATT 3 (23%) 8 year olds CON 3 (36%) Moral (29%) For the complex story we again see that the most frequently occurring responses of TD children across all ages consider the problem (CON 1) and solution (ATT 3 /CON 3) the most important. [PH - you’ll need to address our ongoing dilemma with ATT3 and Con 3] This was only the case for 5 and 7 yr old children with LI.

Simple Story (ENNI 1A) Questions and Production ASHA presentation, New Orleans, November 2001 Simple Story (ENNI 1A) Questions and Production Here you can see at all ages the TD children demonstrate greater knowledge of SG elements when answering questions than what they demonstrated in their story productions but this discrepancy decreases with age. This is a similar finding to other researchers who have looked at questioning and production tasks. For the children with LI the pattern is similar but across both questioning and production tasks they do not perform as well as TD age peers

Complex Story (ENNI 3A) Questions and Production ASHA presentation, New Orleans, November 2001 Complex Story (ENNI 3A) Questions and Production For the complex story the pattern is similar

What is being assessed with the questions? ASHA presentation, New Orleans, November 2001 What is being assessed with the questions? Story comprehension? Basic questions not a pure measure of comprehension Scaffolding is a factor Children may not have provided information in production because: They understood it but didn’t include it It never occurred to them until they were asked What is really assessed: children’s ability to provide information with support

What is assessed with questions? ASHA presentation, New Orleans, November 2001 What is assessed with questions? The problem, solution, and importance questions may be better estimates of overall story knowledge They require children to reflect on the story and evaluate it Importance questions appear to differentiate children with and without language impairments

What do you gain from using the questions? ASHA presentation, New Orleans, November 2001 What do you gain from using the questions? Allows another perspective for assessing narrative abilities Questions can be applied to other stories Allow comparison of supported versus spontaneous narrative abilities Appropriate for use in dynamic assessment Appropriate for use in intervention Can see that you gain different perspectives of children’s abilities in a questioning task than in the production task. Thus the research suggests selecting multiple measures to assess narrative abilities particularly with young children (4-6) Importance judgment questions revealed very different results for TD and LI children As mentioned earlier CNM theorists also suggest that these type of questions also show a child’s ability to distinguish central ideas of the story. So it would be important to make sure you feel children with LI do understand the central ideas of stories. We hope you can see that the way we designed the question sets allows for you to adapt them to particular stories you may prefer using in clinical practice as pre-post probes or as teaching tools. This type of questioning task could be viewed as supported narrative retell from which to compare spontaneous story production abilities. The questions would be very applicable and useful for clinicians looking to develop DA protocols for narrative skills and would support intervention planning.

ASHA presentation, New Orleans, November 2001 Animated version of A1

Effects of story stimuli ASHA presentation, New Orleans, November 2001 Effects of story stimuli Previous work: the way stories are presented to children affects the way they (re)tell the story E.g.: Telling the story for retell versus asking child to tell the story from pictures We wondered: does the static nature of pictures affect younger children’s storytelling in the ENNI?

ASHA presentation, New Orleans, November 2001 Possible effects Typically developing 4-year-old children’s stories were sometimes very different from older children’s They do not always seem to ‘get’ some aspects of the story For example, in the simple story: Ball going in the water Goal-related reason for Giraffe swimming Who gave ball to whom

ASHA presentation, New Orleans, November 2001 Example 1: ENNI A1 a elephant and a cow. (that) that guy fell in the mud. and this girl picked him up # elephant. and he was very mad at her. C did not mention the ball at all Did not understand emotions Got points only for characters

ASHA presentation, New Orleans, November 2001 Example 2: ENNI A1 the elephant and the giraffe looked at a sssstone. but (they) they thought it was a balloon. (Then they were) (and) (and) and they were swimming for it. (And) (and then he got) and then the elephant got it. and they're playing soccer with it. The end. Ball (stone) is in the water – does not go in during the story

ASHA presentation, New Orleans, November 2001 Example 3: ENNI A1 one's doing a ball bounce. one threw the ball in the sand. it got stuck. one's digging in # to get the ball. (one) the elephant's going to help the giraffe. whee one's crying. This child gets the story but changes pool to sand (still got the points)

ASHA presentation, New Orleans, November 2001 Example 4: ENNI A1 once upon a time a giraffe and a elephant met. they saw # a balloon stuck (in) in their water (p) pool. (they the the) the zebra tried to get it. but he (cou) just couldn't get it. then # <the> [/] the elephant got it. C thinks that elephant gave ball to giraffe

ASHA presentation, New Orleans, November 2001 Example 5: ENNI A1(?) ASHA presentation, New Orleans, November 2001 now he's going to say aah no no no no no no no no. I'm not going take those balls # home # to his [/regi/]. (EXA: what? C: [/regi/].) that day that balls got stuck. and he said oh oh. and the lady called away and said ah I'm going to get that ball. and they said uhoh the choochoo train's coming. that way he swimmed to the ball. and now (he said) # oh he cried and cried and cried. um now he helped him to get that ball. and then now he helped to get him up. and he couldn't get up # now. No obvious reason for this one being unusual!

ASHA presentation, New Orleans, November 2001 Animated version of A1 Students from New Media program at the University of Alberta needed a project They animated the simple story from Set A of the ENNI They animated those units that are often misunderstood and/or omitted by young children

ASHA presentation, New Orleans, November 2001 What was highlighted Ball going into water Internal response (facial expression) of elephant Giraffe swimming towards ball Ball going from giraffe to elephant Let’s view the animation

Trying out the animated version ASHA presentation, New Orleans, November 2001 Trying out the animated version Nikki Dooley, Kara Kvile, Kelly Millar and Carla Monteleone conducted a study We gave the animation to 25 typically developing 4-year-old children We compared their stories to 25 4 year olds from the ENNI normative sample Results: children provided slightly but significantly more information with the animated version

Story Grammar scores – static versus animated versions ASHA presentation, New Orleans, November 2001 Story Grammar scores – static versus animated versions

Story Grammar scores – static versus animated versions ASHA presentation, New Orleans, November 2001 Story Grammar scores – static versus animated versions

ASHA presentation, New Orleans, November 2001 Example 1: animated A1 An elephant. And the ball went in the water. An elephant and water. An animal’s swimming {1 sec pause} to get the ball. (He he) the animal gave the elephant the ball. And the end.

ASHA presentation, New Orleans, November 2001 Example 2: animated A1 An elephant. A giraffe. The ball fell into the water. (um) (The elephant gets) The elephant got splashed. The giraffe is swimming. The giraffe passed the ball to the elephant. The giraffe's wet. C talks about close-up of elephant, but not with internal response

ASHA presentation, New Orleans, November 2001 Example 3: animated A1 It fell down the bridge. Splash! They swam like this. And then he got it. And then he gived it to her. And then he was wet.

ASHA presentation, New Orleans, November 2001 Example 4: animated A1 First they are bouncing the ball. And then it flies into the pool. That) The longest one goes to get it. And then what? And then he gives it to (th) the girl. Then she likes that.

Example 4: animated A1 (some are still strange!) ASHA presentation, New Orleans, November 2001 Example 4: animated A1 (some are still strange!) what is the ball doing? (E: can you tell us what’s happening?) it’s drowning # down {laughs}. water splashing up in her mouth and face. oh bubbles bubbles bubbles # bubbles bubbles. bounce. the elephant’s can squirt water. he’s drowning # the giraffe. giraffe giraffe giraffe. he giraffed. he giraffes.

Animated versus static: what we learned ASHA presentation, New Orleans, November 2001 Animated versus static: what we learned Most 4-year-olds will get some SG units in their stories told from static pictures Children who view animated pictures tend to get 1-2 more SG units Animation may make the story easier to understand However, other factors are also involved in 4-year-olds’ lower story performance

The ENNI in other languages ASHA presentation, New Orleans, November 2001 The ENNI in other languages

ENNI studies in other languages ASHA presentation, New Orleans, November 2001 ENNI studies in other languages The ENNI was designed to be useful for collecting narrative data in any language Previous research: with a given stimuli, people from other cultures and languages will tell similar stories The ENNI has been used in several other languages for various purposes

French studies using the ENNI ASHA presentation, New Orleans, November 2001 French studies using the ENNI Pilot study (A. Chambers and J. Mallette) Small sample of children aged 5 and 9 Pictures were altered to change English words to French SG and First Mention scoring were adapted Elin Thordardottir’s studies Currently using ENNI in prevalence study in Quebec with preschool aged children

French studies continued ASHA presentation, New Orleans, November 2001 French studies continued Andréanne Gagné & Martha Crago, U de Montréal Used the ENNI to compare Francophone children’s storytelling to adults versus infants Found typically developing children and children with SLI both adjust their MLT-units for infants SLI children use more SG units with infants than with adults; TD children used same Seemed to be a trade-off between content and syntax

French studies continued ASHA presentation, New Orleans, November 2001 French studies continued French as Second Language learners (L. Baker, C. Traverse, K. Pollock, P. Schneider) 10 children aged 6-8 enrolled in French immersion programs were tested in English and French PPVT and EVIP (French version of PPVT) ENNI in French and English Scores were lower on EVIP than PPVT Scores not different for A1, lower in French for A3

ASHA presentation, New Orleans, November 2001 Other languages Icelandic (Elin Thordardottir) – as a second language compared to native language (including Lithuanian, Russian, Polish, Tagalog) Cantonese (M. Y. Wong) – typically developing children and children with language impairment

ASHA presentation, New Orleans, November 2001 Finnish– one study with children with autism (P. Pelttari), one with typically developing children (H. M. Vakkuri) Kannada speakers, Mysore, India (Padakannaya, Rao, Hayward, Schneider)– typical readers and children with reading disabilities

ASHA presentation, New Orleans, November 2001 Future research Story Grammar scoring and norms for story B3 Norms for sets A and B separately First Mentions Syntax measures Word counts Narrative intervention study: will the ENNI reveal changes due to intervention?