It’s good to count stuff. We can count how frequently someone makes a grammatical error. Or how many speech sounds they find difficult. Or how many times they point to the wrong picture in a comprehension test. Once we’ve done our sums we can get out or look-up table and find out if someone is language-typical or language-impaired. We can find out if an intervention programme is working. We can demonstrate that speech and language therapy works. There’s no need for messy subjective judgments. We can let the data do the speaking. We might even run a few statistical tests which will enable us to quantify our degree of certainty. And we can even count that certainty (using p-values).
There’s no escaping that Speech and Language Therapy needs to embrace quantitative methods. Yet at the same time we risk making the wrong conclusions if we cannot think “beyond” the numbers. One particular pitfall is understanding the difference between causation and correlation. People who live in large houses are more likely to vote Conservative, but this doesn’t mean that the size of one’s house actually determines one’s voting habits. Fortunately, SLT students with their advanced statistical training are able to spot such dodgy claims a mile off. And tear them apart with savage gusto.
But let’s not be too smug. There are plenty of research areas where vigorous debates thrive on the causation versus correlation dilemma. One example is the field of Verbal Working Memory (VWM). VWM tasks involve retaining and recalling linguistic items while simultaneously doing some kind of additional processing task. A classic example is “listening span” where you have to say if sentences are true (the processing task), and then recall the final words from the last few sentences (the recall task). Because performance on VWM is strongly associated with language skill (e.g. comprehension of complex sentences), many have argued that VWM determines language skill. But as every SLT student will know, correlation does not equal causation. An alternative, and I think, far more convincing explanation, is that performance on both language and VWM tasks are influenced by the same underlying factor, which is… err… language. Viewed from this perspective, the idea that VWM determines language skill simply vanishes in a puff of smoke.
To be fair, proponents of VWM accounts have countered these criticisms with some rather sophisticated methods (e.g. structural equation models). No one has yet struck a killer blow and the debate rages on. But stepping back one or two paces, there is a larger lesson to be learned. Running stats is just the starting point. The crucial thing is how we interpret the stats. At the end of the day, though we do lots of complex stuff with numbers, it’s all about the fine art of crafting a convincing and persuasive argument. And in that sense, perhaps we have more in common with the humanities than we’d like to think…
Ever since I first qualified as a speech and language therapist we have been discussing the need to measure the outcome of our interventions. Almost twenty years ago when I was a service manager in Newcastle we knew that we would need to provide data about our outcomes to our managers and commissioners. In an attempt to get ahead of the game and be well prepared for when the request came, we invited Pam Enderby up to Newcastle to talk to us about her newly developing Therapy Outcome Measures (TOMs). Based on the principles of the newly emerging WHO ICF (since redeveloped and modified) TOMs was, as far as I know, the first outcome measurement tool that considered multiple aspects of the client and did so in a holistic way (Enderby & John, 1999). At that time we really collected data for our own edification and amusement as no one ever asked us for it.
We live in a different world now and the need to demonstrate the effectiveness of our interventions has become pressing, as the structure and drivers behind commissioning have changed. However, we have no set of outcome measures that are acceptable to all and the danger we have now is that services will develop outcome measures independently that are not comparable. Commissioners will not be able to accurately or appropriately compare services with each other or evaluate their outcomes and the consequence will have a negative impact on the quality of client care. As a profession we need to be able to rise above our fierce independence as autonomous practitioners and devise a system for measuring outcomes that enables us to compare like with like.
The Royal College of Speech and Language Therapists (RCSLT) has picked up the baton and is leading an initiative to develop national outcome measures by consensus through engaging as many SLTs in England as possible. On October 10th 2013 RCSLT held a Summit meeting in Birmingham to start the process. Although I am no longer working at the service delivery sharp end, I have been involved in the outcome measure work with RCSLT because it links into a research project that I am involved with in collaboration with RCSLT and Newcastle University Culture Lab (http://www.ncl.ac.uk/culturelab/). Consequently, at the Summit, I sat with the dysfluency group although I have no expertise and limited knowledge in that area. A Theory of Change (http://www.theoryofchange.org/what-is-theory-of-change/#6) process was used to develop some preliminary ideas at this meeting which we took forward with the North East Regional Dysfluency Clinical Excellence Network (NERDCEN) a fortnight later.
Theory of Change provided a framework for us to consider the outcomes of intervention for children, young people and adults with dysfluent speech. Theory of Change takes us from the ultimate outcomes of our intervention to consider the intermediate outcomes that contribute to them and then to the activities that we need to undertake in order to achieve the goals. The combined outstanding brain power in the room took remarkably little time to come to a consensus in each of these areas. I think this was partly down to the fact that they had been talking about outcome measures for several years and had a fairly good idea where they were heading and partly down to the structure provided by Theory of Change. Another contributing and helpful factor on the day was the presence of Dominique Lowenthal, Professional Development Services Manager at RCSLT, who has a lot of experience using Theory of Change in different RCSLT projects. Dominique guided us through each stage so that the relevant points were considered and everyone’s views were taken into account. The result is not yet an outcome measure (or two) but a path on the way and a consensus of the experts in this region. This work will continue with input from NERDCEN and RCSLT over the next few months. The aim is to have a set of outcome measures that are agreed across all SLTs that can be used in the RCSLT/Culture Lab/Speech & Language Sciences project. We are on our way to being able to compare apples with apples (even if some are red and some are green) rather than continuing to have apples in one hand and broccoli in the other while we vainly search for similarities.
Thanks to: @domlowenthal and all the NERDCEN SLTs.
Enderby, Pam ; John, Alex; (1999). Therapy outcome measures in speech and language therapy: comparing performance between different providers. International Journal of Language & Communication Disorders, 34(4), 417-429. doi: doi:10.1080/136828299247360