Reliability of Spanish articulation and resonance ratings collected on digitally recorded samples

M. Cleves^b, A. Lizarraga^a, L. Bermudez^a, M. Tavera^c, M. Echevery^c, B. Dueñas, M. Hanayama^d, C. Arboleda^b

^aOperation Smile, Inc., Norfolk, Virginia, USA: ^bFundación Operación Sonrisa Colombia, Santa Fe de Bogotá, Colombia; ^cFisulab, Santa Fe de Bogotá, Colombia; ^dOperação Sorriso do Brasil, Sao Paulo, Brazil; 
 

SUMMARY

This study attempts to help in the development of a perceptual assessment tool for use in the clinical practice in Spanish speaking populations with cleft palate, following the ‘universal parameters’ criteria described by the Speech Parameters Group (Henningsson et al., 2008). A set of words and sentences were created as media for evaluation, along with spontaneous speaking samples. Speech pathologists from different centers were involved in the evaluation and analysis of their results was undertaken to assess inter-rater reliability. Information obtained highlighted successes and challenges of the created tool and overall difficulties in the development of an acceptable, reliable, and easy to use cleft speech evaluation tool.

INTRODUCTION

A goal in the treatment of patients with cleft lip and palate is to provide rehabilitation which promotes active participation and integration into society. In order to accomplish this, the care must not only be individualized, but should have mechanisms to ensure that applied techniques are truly effective.

The quest for ideal care has created a need to measure, report, and compare treatment outcomes of different centers and professionals. This requires the use of established parameters for the effective measurement of determined variables and standardization for multi-center use. This need in relation to speech outcomes has been identified in the past (Morris, 1973 & Dalston et al., 1988). Attempts to develop an effective method based on evaluations via instrumental techniques, including aerodynamics (e.g., Warren & Dubois, 1964), accelerometry (e.g.,Horii, 1980), and acoustics/nasometry (Fletcher et al., 1989; Kent et al., 1989), have failed to provide a consistent correlation between instrumental and perceptual evaluation. Problems have been cited with management and calibration of equipment, experience and number of evaluators, and consistency of patient population, among others. Interestingly, few have focused on methods which are cost-effective and easy to apply, which is important for documentation and sharing of results across sites. Moreover, the use instrumental devices may not be appropriate for infants and young children.

Consequently, perceptual evaluation has become the most widely used method for the evaluation of speech in individuals with cleft palate (Peterson-Falzone et al., 2000), with the final decision on the outcome of treatment being based on the listener’s subjective measures (Moll, 1964). For this reason it is necessary to strive for a more standardized protocol to describe speech problems (Kuehn & Moller, 2000). This type of protocol should include stimuli, application criteria, evaluation parameters and scales, and documentation guidelines, in order to most accurately compare data. Comparison between studies has proven difficult given the differences in the variables stated above (Grunwell & Sell, 2001).

This study attempts to create a useful, accessible, and easy to apply assessment tool, in order to allow for multi-center comparison of surgical and therapeutic treatment of Spanish-speaking patients with cleft palate. To support development of a valid tool, recommendations of Peterson-Falzone and colleagues (2001) on using variety of sampling procedures as well as conversational speech to gather information about the consistency of the patient’s errors, in addition to the observations of Fletcher et al. (1989) and Watterson et al. (1996) on the perceptual concept of hypernasality and phonetic context of speech, were taken into consideration. Stimuli in Spanish were created to utilize in addition to spontaneous speech samples. The stimuli were developed using the universal parameters criteria established by Henningsson and colleagues (2008) as a guideline to the fullest extent possible. This was completed as a first step towards development of a standardized tool, which could be utilized by the community of professionals treating Spanish speaking patients with cleft palate. This is necessary as perceptual evaluation can determine if the patient must undergo further anatomic or functional diagnostic procedures (Marsh, 2004).

An important challenge has been the lack of reliability in methods presented in previous publications (Dalston et al., 1998). Generally, differences in evaluation criteria between raters over time have become greater due to variations in training and use of center-specific protocols. We attempted to unify the criteria of the evaluators in an effort to increase reliability in the creation of a proposed perceptual evaluation system.

METHODS

Thirty patients from three specialized cleft centers in Colombia were included in this study. Of these 17 were males, 13 females, with an age range of 4 to 31 years and an average age of 9 years.

Each patient was asked to read a set of 30 words and 33 phrases previously chosen as stimuli for analysis by our team. A spontaneous speaking sample of one minute was also requested from the patients. Based on Moller and Starr’s findings (1984) related to the use of audio in analysis of speech, all samples were recorded for multi rater analysis utilizing a digital recorder (SONY Hi-MD Recorder MZ-200) and then transferred to MP3 format to facilitate their exchange. Three linguists analyzed and selected the 30 samples with best acoustic characteristics.

The selected samples were evaluated individually by six speech pathologists (five from the centers where the samples were collected and one from a Brazilian center). The MP3 recordings were reproduced and audio headsets were utilized (Shure SE110) to eliminate background noise. Ratings using a combination of binary (presence/absence) and four point equal appearing interval scales (0=WNL, 1=Mild, 2=Moderate, 3=Severe) were made based on Henningsson and colleague’s universal parameters (2008) and the results were recorded for subsequent analysis. Inter-rater reliability was assessed with Kappa tests, using the SPSS statistical analysis software.

RESULTS

Inter-examiner reliability was rated as good/very good (Kappa 0.61 to 1.00) for hypo nasality (.8), voice disorder (.73), abnormal backing of oral target to post uvular place (.8 and .8), abnormal backing of oral targets but place remains oral (.78-.6), nasal fricative (.7 and .7), and nasal consonant for oral pressure consonant (.7); moderate (Kappa 0.41 to 0.60) for intelligibility (.41-.53), acceptability (.50), and hypernasality in words (.53); and poor (Kappa 0.39 or less) for hypernasality sentences (0.38) and audible nasal emission and/or turbulence (.32-.28). See Figure 1.

DISCUSSION

The findings of this study demonstrate greater reliability for those binary variables described as present or absent in the evaluation, than those with a severity scale. Good reliability was found for presence or absence hyponasality, voice disorders, nasal fricatives, accurate articulation placement in regards to ‘backing’, and articulation manner in regards to nasal versus pressure consonants. This could also have been expected given the ease in which these parameters can be analyzed auditorily. Moderate reliability was obtained in hypernasality, acceptability and intelligibility, probably due to increased complexity of the severity scale. Agreement between raters diminished, likely due to the additional possible options and the influence that differences in experience and rater criteria may play. The lowest reliability was found in nasal emissions and nasal fricatives. The perception of these parameters can be easily altered by noises surrounding the rater, position of the microphone and additional noises recorded during sample collection.

This study provided the opportunity for the creation, use, and review of assessment stimuli according to Henningsson and colleagues’ (2008) universal parameters. It was completed as the first step towards the use of a standardized protocol for Spanish-speakers, to assist in the exchange of data, comparative analysis, research and proper reporting of results. It highlights the importance of unifying evaluation criteria as well as documentation and evaluation processes. It also supports the need to establish universal parameters, with uniform terminology and management by speech pathologists.

Although good and moderate reliability were observed in some parameters, some still exhibit low inter-rater reliability, and their evaluation scale, criteria and limitation related to the rater’s experience still warrant further consideration. Additional studies focused on the topic, keeping considerations outlined and challenges faced by our group in mind, could advance diagnostic processes and improve the care of patients with cleft palate, easing their efforts in overcoming the psychosocial barrier imposed by their inability to communicate properly.

ACKNOWLEDGMENTS

We acknowledge Kelly Nett Cordero of Gillette Children’s Specialty Healthcare, Saint Paul, Minnesota, U.S.A., for her support with stimuli development and article editing.

REFERENCES

Dalston et al., 1988

Fletcher et al., 1989

Grunwell & Sell, 2001

HENNINGSSON G., KUEHN D., SELL D., SWEENEY T., TROST-CARDAMONE J., WHITEHILL T., Universal Parameters for Reporting Speech Outcomes in Individuals with Cleft Palate, Cleft Palate-Craniofac J, 45(1), 1-17, 2008.

Horii, 1980

HUTTERS, B. et al. Speech Outcome Following Treatment in Cross-Linguistic Cleft Palate Studies: Methodological Implications. Cleft Palate-Craniofacial Journal, 5: 544-549, 2004.

JOHNS DF, ROHRICH RJ, AWADA M,Velopharyngeal incompetence: a guide for clinical evaluation. Plast Reconstr Surg 7:1890-1897, 2003.

Kent et al., 1989

KUEHN, D.P & MOLLER, K.T., Speech and language issues in the cleft palate population: The state of the art, Cleft Palate and Craniofacial Journal, 37, 1-35, 2000.

LACZI E et al. Perceptual Evaluation of Hypernasality Compared to HONC Measures: The Role of Experience. Cleft Palate-Craniofac J., 2: 202-211,2005.

LOHMANDER, A et al.. Methodology for Perceptual Assessment of Speech in Patients with Cleft Palate: A Critical Review of the Literature. Cleft Palate-Craniofac J., 1: 64-70,2004.

MARSH JL, The evaluation and management of velopharyngeal dysfunction. Clin Plast Surg 2:261-269, 2004.

MAYO, R. Et al. Perceptual Assessment of Resonante Distortion in Unoperated Clefts of the Secondary Palate. Cleft Palate-Craniofac J., 4: 397-400,1993.

Moll, 1964

Morris, 1973

PETERSON-FALZONE, S., HARDIN, JONES, M., KARNELL, M., Cleft Palate Speech, Philadelphia, Mosby, 2000.

SMITH B, GUYETTE TW, Evaluation of cleft palate speech, Clin Plast Surg, 2: 251-260, 2004.

Warren & Dubois, 1964

Watterson et al. 1996