Sunday, 18 December 2016

Evidence on Evidence Based Policy

You were no doubt as surprised as I was when the Blair government announced it was henceforth doing evidence based policy. It was just like when the medical profession said it was going to do evidence based medicine. You mean—they weren’t already? Still, even though the promised reform doesn’t really sweeten the bitter truth, it is a move in the right direction. Or at least, it a promise to take a move in the right direction. But was the move taken?  When it comes to the, perhaps minor, example of speed cameras, we have clear evidence whether the policy was evidence based. And the evidence says, No!

According to official figures today ‘Speed cameras have failed to cut accidents on many roads and have actually led to a rise in casualties on some routes’

In fact, this shouldn’t be a surprise at all. This evidence has been around for a long time. All the common errors and biases in thinking have been put forward to justify speed cameras, from confirmation bias to ignoring base rates. Statisticians of many stripes pointed out on many occasions the fallacious statistical thinking being used to justify their introduction. For example, the claim that speed cameras cut accidents was based on noting that there were fewer accidents at a camera site in the year after it was introduced than in the year before. Sound convincing? It shouldn’t do. If you examine nearly all the placed that had accidents last year, they had fewer this year. This is called regression to the mean. The very incidents that are used to justify introducing a camera are highly likely to be a random coincidence. Taking this to be evidence is like a lousy archer shooting arrows at a barn and then painting his target round the area of a lucky cluster.

A further symptom of evidence based policy would be making possible the systematic examination of whether a policy is succeeding by publishing all the relevant data. In this case, it would not be hard to do, but the fact is that councils are not publishing the data and will not give you the data if you ask—just as bad as the climate scientists who refused to share their data with critics because those critics were trying to prove them wrong! Apparently, the Department of Transport will ‘now conduct a detailed statistical analysis of the data to assess the effectiveness of speed cameras in improving road safety’. So 19 years after they were introduced only 17 years after independent statisticians proved the inadequacy of the claims being made that speed cameras improve safety, the DOT gets round to checking if it was a good idea.

Perhaps the most worrying fact is that it is probably na├»ve to expect politicians to institute real evidence based policy and most especially to expect them to institute evidence based assessment of the success of their policies. Most of the benefit to a politician of a policy is in the announcement and the nice title for the law—if it wasn’t so obviously bogus I’m sure we would have the ‘making everything good’ law.  Provided the policy sounds like it fits their prejudices his electorate will re-elect him, so there is little additional benefit to showing it worked but a big cost if the evidence shows it didn’t. Or perhaps you thought politicians were motivated by more noble sentiments than the rest of us?

Originally at

No comments:

Post a Comment