Are randomised control trials into learning the answer?

11 September 2015

I’ve long been a fan of Tim Harford (from Radio 4s More or Less and author of The Undercover Economist) and was therefore excited to hear that he was to speak at this week’s European conference on Realising Impact: Making a difference through adult learning. While admitting upfront that he was no expert in adult learning, his mother’s experience of teaching adult literacy, no doubt informed his understanding of the power of teaching adults to read. 

Drawing on the history of randomised control trials (RCTs) and their widespread effective use in medicine, Tim challenged us to put aside our often held God-complex – of being convinced that we already have the answer to complex situations – but to test and gather robust evidence on what works and why. He argued that even where there is universal acceptance that something is a good idea – such as teaching people to read – trials can show us how best to do this, and where to focus limited resources.

Randomised control trials are well used in the health sector, with doctors conducting an average of 75 trials per day. In contrast, a review of the evidence around adult literacy identified just 9 RCTs conducted between 1980-2002. And what have we learned? Well, that being taught has more impact than not being taught; that using computers might help; and that we need more and better designed trials if we are going to establish clear and compelling evidence of what works.

While it is true that we have been busy gathering a tremendous amount of data for decades, there is a risk that we have been wasting much of our time on studies that are not sufficiently rigorous, and that are often more descriptive than analytical. Using a range of fascinating examples, the conference heard how RCTs have been used to show some unexpected and surprising things – that the once accepted treatment for brain injuries actually caused more people to die, that the much-promoted norm in the 70s of putting newborns to sleep on their stomachs resulted in the death of around 50-60,000 babies, and that paying people to come to literacy classes is a bad idea.

Of course, establishing control trials within education is not without significant challenge. Careful consideration needs to be given to issues of ethics, consent and communication, the separation of direct and indirect outcomes, and the importance of identifying long term as well as short term impacts. Differences in the profile, networking and ongoing CPD of the health and adult education workforces make it harder for the findings of educational research to be translated into widespread practice. Securing funding for trials is likely to be much more difficult than for health research: rich and powerful people get ill, but rarely have to learn to read as an adult, and there are no equivalents to the large pharmaceuticals in the adult education world. 

However, these issues are not insurmountable. NIACE is currently involved in two RCTs to test the effectiveness of delivering English and maths through blended or face-to-face learning, and to evaluate a Citizens’ Curriculum approach to supporting ESA claimants into work. We should only give up on the idea of control trials in adult education when we are absolutely satisfied that we already have compelling evidence to demonstrate that we know exactly what works.