1. Mapping Workshop on Universal Reporting Parameters for the Speech of Individuals with Cleft Palate Washington DC, April/May 2004 Tara Whitehill, Hong Kong

2. Proposed Three-Stage Plan Evaluating: The usual speech evaluation by the local speech specialist in cleft palate. Mapping: Conversion from the local speech evaluation to the universal reporting measures. Reporting: Universal parameters that describe the cardinal features of speech in individuals with cleft lip/palate and related craniofacial anomalies regardless of the individual’s primary language.

3. Already covered Evaluation (range of protocols and procedures; key issues) Parameters (proposal; key issues) Focus: mapping

4. Fanny LAW Sze Kuen International Parameters for Cleft Palate Speech: A Pilot Study Honours dissertation Bachelor of Science (Speech & Hearing Science) University of Hong Kong May 2004 Supervisor: Dr. T. L. Whitehill DRAFT - in progress

5. 6 sets of client forms from 5 centres 5 clients each set = 30 client forms Original forms only (no recordings, no additional data such as transcriptions, unless provided on original form) Two groups of “mappers” –5 “experts” - members of WG who supplied forms –5 students - little/no previous experience cleft

6. Used “current” version of parameters and guidelines Students given session to familiarize/ clarify terminology Mapped from client forms onto parameters form Identifying information masked (client, examiner, centre) Provided feedback on ease/difficulty of mapping each parameter –Either for client form or by set (centre) Weighted scores - not included in this study

7. “Outcome measures” Agreement –Within expert group –Within student group –Across all mappers Feedback (ease/difficulty)

8. Why use students? In future, unlikely mappers will be naïve However, “expert” group was restricted to individuals who had developed and debated the parameters

9. Agreement A big problem was “missing data” OK if client form stated MD (or WNL or NAD), but what if didn’t? - assume OK? tick MD? Calculated (a) absolute agreement (Condition 1) (b) disregarding “missing data” (Condition 2)

10. Table 1. Examples to illustrate how agreement was defined

11. Primary Parameters Hypernasality Agreement Experts Condition 1 - 60% Condition 2 - 63% Students Condition 1 - 73% Condition 2 - 73% All Condition 1 - 57% Condition 2 - 60% Not much missing data

12. Primary Parameters Hypernasality Feedback/Issues to consider Some difficulties when no. scale points differed (e.g., client form - 7 pt. scale; parameters form - 4 pt. scale) In such cases, mappers reported difficulty, and agreement decreased Implication: explicit instructions? Another difficulty: “mild to moderate” or “moderate to severe” (students selected less severe and experts more severe!) “Set B” - definition of “mild” = “evident but acceptable”. One expert rater mapped onto WNL.

13. Primary Parameters Hyponasality Agreement Experts Condition 1 - 53% Condition 2 - 96% Students Condition 1 - 87% Condition 2 - 100% All Condition 1 - 50% Condition 2 - 97% Excellent, when missing data is accounted for

14. Primary Parameters Hyponasality Feedback/Issues to consider Missing data Otherwise, excellent (binary scale)

15. Primary Parameters Audible Nasal Emission with/without Nasal Turbulence Level One (absent, present, MD) Experts Condition 1 - 63% Condition 2 - 87% Students Condition 1 - 77% Condition 2 - 80% All Condition 1 - 57% Condition 2 - 70% Level Two (subcategories) Experts Condition 1 - 23% Condition 2 - 60% Students Condition 1 - 30% Condition 2 - 38% All Condition 1 - 17% Condition 2 - 27%

16. Primary Parameters Audible Nasal Emission with/without Nasal Turbulence Feedback/Issues to consider Low agreement for subcategories “phoneme specific” - e.g. “audible for plosive and /s/” Confusion about pervasiveness e.g. “mild but consistent”, “phoneme specific and frequent” Results of mirror test (if not audible in speech) (NE w/wout nasal turbulence - not raised here)

17. Primary Parameters Weak Oral Pressures Agreement Experts Condition 1 - 17% Condition 2 - 75% Students Condition 1 - 13% Condition 2 - 62% AllCondition 1 - 0% Condition 2 - 46%

18. Primary Parameters Weak Oral Pressures Feedback/Issues to consider All 9 mappers found this parameter difficult to map Primarily because of lack of data on client forms - no form had specific category for this Mappers needed to derive the information from other information; particularly difficult for students Also, students unclear about definition

19. Primary Parameters Substitution Errors Level One (absent, present, MD) Experts Condition 1 - 57% Condition 2 - 73% Students Condition 1 - 73% Condition 2 - 85% All Condition 1 - 57% Condition 2 - 70% Level Two (subcategories) Experts Condition 1 - 10% Condition 2 - 30% Students Condition 1 - 10% Condition 2 - 15% All Condition 1 - 10% Condition 2 - 20% Agreement not bad for level one, but very poor for subcategories; missing data was not the main problem

20. Primary Parameters Substitution Errors Feedback/Issues to consider All forms included level one (present/absent/MD) Subcategories: most forms had patterns, but few included frequency information Some patterns which expert listeners considered ‘related to cleft palate’ not included here (e.g. nasal substitution/realization’, ‘double articulation’) Confusion between this parameter and ‘Developmental Delay or Other Articulation/Phonological Errors’ (even for experts) e.g. omission, backing /t/ -> [k].

21. Secondary Parameters Errors related to dentition, occlusion, palatal vault configuration Agreement Experts Condition 1 - 13% Condition 2 - 69% Students Condition 1 - 30% Condition 2 - 59% AllCondition 1 - 7% Condition 2 - 53% Better when missing data taken into consideration, but still pretty poor

22. Secondary Parameters Errors related to dentition, occlusion, palatal vault configuration Feedback/Issues to consider No specific category for this on client forms; information had to be extracted from other information Some forms (Sets B, E, F) included categories such as ‘lateralization’, ‘palatalization’, ‘dentalization’ - but still not clear whether these due to organic factors or “Other Articulation/Phonological Errors” - even for experts Errors related to hearing problem? fistula?

23. Secondary Parameters Developmental Delay or Other Articulation/Phonological Errors Agreement Experts Condition 1 - 16% Condition 2 - 72% Students Condition 1 - 30% Condition 2 - 50% AllCondition 1 - 10% Condition 2 - 43% Again, better when missing data taken into account, but still pretty poor

24. Secondary Parameters Developmental Delay or Other Articulation/Phonological Errors Feedback/Issues to consider Mappers needed to make judgement regarding whether errors were related to cleft or not Confusion with both “Substitution Errors” and “Errors related to dentition..” Client forms rarely specified cause or explanation of error patterns - mappers needed to judge Special cases e.g. “no oral consonants”

25. Secondary Parameters Voice/Laryngeal Disorder Agreement Experts Condition 1 - 56% Condition 2 - 90% Students Condition 1 - 63% Condition 2 - 97% AllCondition 1 - 60% Condition 2 - 96% Agreement was good, especially when missing data was taken into account

26. Secondary Parameters Voice/Laryngeal Disorder Feedback/Issues to consider Similar to hyponasality - mappers were familiar with term When data was provided on client form, it was clear Binary choice; no subcategories

27. Global Rating Severity/Intelligibility Not decided at time of this study Only 2 of the 6 sets included such data

28. Conclusions Reliability lower among students than experts Experts were those who developed form Who will be mapping in the future?