6/7/12

Rating Scales to Measure Change


I just read a speech/language therapy report that contained daily fluency ratings for an entire school year.  Two graphs were attached to the report, one illustrated the teacher’s ratings and one illustrated parent ratings. Quite impressive!
I used to ask parents to jot down a few notes about their children’s fluency at home. I requested they make brief videos and/or audio recordings. Very few parents did so and some were so apologetic about noncompliance that I feared I’d made matters worse. Ideally, speech therapy is supposed to keep track of speech fluency in a variety of settings; but, just how to accomplish this is a challenge.
Rating scales offer a simple way to do this. A rating scale translates a subjective judgment into a number on a continuum, say 1 – 10 for example. Some rating scales have carefully defined the numbers. I adapted this kind of rating scale to supplement my notes for almost every speech therapy session I have with every student. It is a 9-point scale originally developed to measure social interaction. (1)  Points 1-2 have this definition: “Quality of the behavior is poor; there is no evidence of a habit beginning. The quantity and duration of the examples are negligible or inappropriate.” There are five such explanations on this scale. Numbers 8-9 are defined as “Quality of the behavior is outstanding, and the behavior is used creatively in the interaction. Quantity and duration of the examples are appropriate.” I added a blank spaces to write in the behaviors being measured at each session.
There are plenty of relevant behaviors and characteristics worth measuring in the field of fluency. With respect to speech, let’s consider the child who stuttered on 20 percent of syllables in September and still stuttered on 20 percent of syllables in June. One might glance at this data and assume the child had not made any progress.  A decrease in oral-motor tension over this period of time would have been excellent progress, although invisible in a measure of percent syllables stuttered. This hypothetical child could have been experiencing tense blocks and prolongations in September but predominantly relaxed part-word repetitions in June. Let’s take a look at a few rating scales. What do they measure and how?
A rating scale may define only a few numbers along a continuum. The Test of Childhood Stuttering Observational Rating Scales measures Speech Fluency and Disfluency-Related Consequences. (2)  It has only 4 points: 0 = Never, 1 = Rarely, 2 = Sometimes, 3 = Often. There are nine items in the Speech Fluency section including descriptions of blocks, prolongations and repetitions. There are also nine items in the Disfluency Related Consequences section including secondary behaviors, avoidance, and emotional reactions. I use this scale for initial screenings and for monitoring progress. I’m not completely satisfied with the sensitivity of a 4-point scale to reflect the subtle improvements that are characteristic of fluency therapy. Note: this scale is completed by adult listeners.
I used the Assessment of the Child’s Experience of Stuttering (ACES) when it was available in a free draft version (2006). This was a lengthy, 100-item  scale divided into four sections - General Information, Your Reactions to Stuttering, Communication in Daily Situations, and Quality of Life. Each item was rated on a 5-point scale with three definitions: 1 = Always, 3 = Sometimes, 5 = Never. A student’s responses were added to determine Impact Scores that corresponded to Impact Rating definitions: mild, mild-to-moderate, moderate, moderate-to-severe, and severe. I found the ACES was sensitive to therapeutic progress for older teens. It seemed too overwhelming for my elementary school age clients. When the final revision was published commercially (3), I stopped using the draft form. Note: this scale is completed by the speaker.
Some scales define only the endpoints. The Adolescent Communication Questionnaire (12)  is a list of 39 speaking situations rated on a 5-point scale. The directions state: “How much confidence do you have about doing each of the behaviors listed below? Circle the number that best represents your confidence.” 1 = No way. I would be too uptight to speak and  5 = No problem. I would be very confident speaking. (4) Another scale that measures communication attitudes uses seven data points to measure comfort level, confidence, and feelings of success. (5) The word “extremely” appears at the extreme endpoints in one of them: 1 = Extremely Successful and 7 = Extremely Unsuccessful. Note: these scales are completed by the speaker.
There are a few qualitative differences between rating scales worth mentioning. First, for some scales (1, 4), larger values represent greater success. On others (2, 3, 5), larger numbers represent greater severity of a problem.  The Predictive Cluttering Inventory (6) uses a scale with the largest number placed on the left side of the continuum unlike all the other scales I’ve described. Second, it is very important to document who, what, when, and where every time a rating is chosen. Fluency is notorious for its variability. I think it is safe to say that no one stutters all the time in the same way. And third, without training for agreement, everyone creates their own definitions for points along the scale. (7) Very anxious parents or highly sensitive speakers may rate dysfluencies much more severely than an SLP or teacher. How about the pediatricians who counsel parents to wait and see if their toddlers will outgrow stuttering? I imagine they rate dysfluencies quite mildly.
Why use a rating scale? Besides the demand for data in our present educational system, numbers can be easier to compare over time. In the report I mentioned earlier, teacher ratings on a 10-point scale indicated stuttering decreased quite dramatically from the beginning to the end of the school year. I expect this data represents careful collaboration between the teacher and the school-based SLP. My congratulations to everyone involved in this case. It’s always a delight to discover  school district willing to help children who stutter. How can I and others adapt this kind of collaboration to meet the unique needs of other students?
One possibility is to define points along the 1-10 scale. Combined with training for listener agreement, carefully worded definitions may help us compare across listeners with some accuracy. Then we might feel more confident that a teacher rating of 7 could be the same as a parent’s rating of 7. If training is not possible, it is still helpful to pursue the question of why fluency ratings change for one listener over time. This could reveal new and/or confirm known strategies that were especially effective. This is not carefully controlled research, but I would argue it is still beneficial.
A little aside: when data is collected as part of a research study, a statistical analysis determines whether or not change is significant and not due to chance. Well-designed research has many controls in place so that experimental results confirm or deny a specific hypothesis.  For example, well-designed studies have a control group of people who receive no treatment.  If the control group does not change during the course of the experiment yet the treatment group does change after receiving treatment, one might argue that treatment was responsible for the change. Such research is difficult to do with children who stutter because it means withholding treatment for those children in a control group.
Ratings focus our attention on individual issues that affect fluency and communication.  This focus helps us to understand and control for those factors to promote greater fluency, healthy attitudes (8), and more effective communication. Let’s return to our example of measuring of stuttering frequency. Sometimes children may stutter less because they successfully generalize fluency enhancing skills and educate the listeners in their environment.  Or, it might be that greater fluency is the result of a parent/caregiver-implemented, home-based therapy plan. (9)  In these examples, everyone can feel empowered and optimistic about prognosis for improvement.
Alternately, it may be that children stutter less because they are talking less or avoiding difficult words. These are maladaptive speech/language/communication behaviors. Many experts in the field propose that it is healthier for a child to express herself freely while stuttering than to reduce verbal output for the sake of fluency.
Sometimes fluency fluctuates in conjunction with life events. Illness, fatigue, moving to a new home, family changes, academic demands, annual celebrations, and growth spurts in language development can all affect fluency. In these cases, it would be cruel to hold a child accountable for increases in stuttering. Stuttering management is a multifaceted endeavor that requires team work between parents, teachers, caregivers, SLPs and others. It is a puzzling disorder that demands patience, understanding, and kindness from everyone involved.
Here’s what I mean, an infant sibling may keep all the other family members awake at night and impatient during the day. We don’t expect a young child to automatically know how to cope in this situation. A young child is not likely to say, “Hey! Mom and dad! I need help falling back to sleep when that charming new little kiddo wakes me up with his crying in the middle of the night! And, by the way, could we schedule some more time for me!”  However, this child might have more angry outbursts, be less cooperative, and demonstrate more dysfluency instead, IMHO. Does this child need environmental management or direct fluency therapy?  Rating scales may help determine the effectiveness of the chosen treatment.
Perhaps  the older child has discovered his stuttering is affecting his grades.  He may be frustrated by failed attempts to change his speech on his own and has stopped raising his hand in class.  He might be noticing an inability to keep up with the rapid, fluent speech of his peers. This child may need a treatment plan that includes learning more about speech physiology and instructs him in ways to advocate for himself. Rating scales related to quality of life issues, self-confidence and communication attitudes could be crucial to a total treatment program. (10)
Rating scales can be a valuable asset in fluency therapy. (11) We need to consider several choices of what to measure, how to define points along the continuum, and how we expect the results to affect future treatment. I’m looking forward to using rating scales more often and more creatively in my own practice.
On a scale of 1-5 (where 1= most unsatisfactory and 5 = most satisfactory) , I hope you have a 10 kind of day!

(1)  MacDonald, J.D. & Carroll, J.Y. (1992) A Social Partnership Model for Assessing Early Communication Development: An Intervention Model for Preconversational Children. Language, Speech, Hearing Services in Schools, 23, p. 115.
(2)  Gillam, R. B., Logan, K.J., Pearson, N.A. (2009) Austin TX: PRO-ED
(3)  Yaruss, J.S., Quesall, R., Coleman, C. (2010) Overall Assessment of the Speaker’s Experience of Stuttering (OASES), Pearson Education, Inc.,  http://psychcorp.pearsonassessments.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=PAaOASES&Mode=summary
(4)  Bray, M.A., Kehle, T. J., Lawless, K.A., Theodore, L.A., (2003) The Relationship Between Self-Efficacy to Depression, American Journal of Speech Language Pathology, 12, p. 431.
(5)  Gottwald, S.R. (2011) “Rating Scales as a Clinical Tool” http://www.mnsu.edu/comdis/isad15/papers/therapy15/gottwald15.html
(6)   Daley, D. (2007) “Cluttering: Characteristics Identified as Diagnostically Significant by 60 Fluency Experts” http://www.mnsu.edu/comdis/isad10/papers/daly10/daly10.html
(7)  One teen directed me to his favorite online comic strip, xkcd, one that poked fun at use of a 1-10 rating scale of pain. http://xkcd.com/883/  
(8)  Yaruss, J. S. & Coleman, C., Stuttering Center of Western Pennsylvania, “Helping Children Who Stutter Develop Healthy Communication Attitudes,”  http://www.stutteringcenter.org/Forms/2009-stuttering-center-parent-attitudes%20handout.pdf
(9)  Lidcombe Program, Australian Stuttering Research Center, http://sydney.edu.au/health_sciences/asrc/clinic/parents/lidcombe.shtml
(10)               Blood, G. The POWERR Game, Stuttering Foundation, http://secure.stutteringhelp.org/Merchant5/merchant.mvc?Screen=PROD&Store_Code=SFA&Product_Code=0250&Category_Code=N

(11)  O’Brian, S., Packman, A., Onslow, M. (2004) Self-Rating of Stuttering Severity as a Clinical Tool, American Journal of Speech-Language Pathology, 13, 219-226.

(12) M. A. Bray, et. al. (2003) The Relationship of Self-Efficacy and Depression to Stuttering. American Journal of Speech Language Pathology, Vol. 12, 425-431, the rating scale is on page 431.

No comments:

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.