I work with loads of very clever, highly paid, successful people who really don’t have the first clue how to interpret research. No, really, you may even have heard of quite a few of them.
I’m not talking about agency creative people here. Some of the people I’m thinking of work in marketing. Big grown up business-type jobs, not flaky ‘colouring-in’ type jobs. Quite a few of the people I’m thinking of have the letter "C" in their job title.
And don’t get me started on the journalists.
Maybe we could use the headline "Really clever, advanced thinking about data" so those who think they know loads about research will read it anyway, and you can print it out and read it on the tube without feeling embarrassed, like those grown-up Harry Potter covers.
This is an attempt to rectify that situation, a little. Maybe we could use the headline "Really clever, advanced thinking about data" so those who think they know loads about research will read it anyway, and you can print it out and read it on the tube without feeling embarrassed, like those grown-up Harry Potter covers.
When we talk about data, the first thing that tends to get mixed up is studies about what happened in the real world and surveys about what people tell you they might do, or think.
That first category is what people are sometimes talking about when they say "big data". It’s the kind of data that tells you that your customers abandon their basket if they have to click three times before completing a transaction on your website. Or that your Facebook advertising has a better click through rate than your Google adwords. It’s what actually happened.
The second category is the kind of data that told you that 55% of people think they will vote to remain in the EU. It is equally true – at the time of asking, those 55% of people probably were feeling that way. What it can never tell you is whether or not that feeling will result in action.
When we talk about data, the first thing that tends to get mixed up is studies about what happened in the real world and surveys about what people tell you they might do, or think
It’s not worthless, or garbage, as some pundits like to say nowadays, it just needs to be treated differently. Because we like simple stories, and dramatic headlines, what tends to happen is that this aggregation of people’s own predictions of their own behaviour is treated as a prediction of the future by the research company (companies who often, bizarrely, frame it as such, in their attempt to get coverage).
One is about behaviour, the other opinion. Behavioural data is good because it is true, but bad because it can’t tell you what will happen in the future. Opinion data is good because it can tell you what might happen in the future, but bad because this sentence has the words "can" and "might" in it. You need to know which kind you’re looking at before you can use it.
The next mix-up is data based on samples, and data from the total universe. If you can ask everyone, then there is no need for clever maths. But usually you can’t, so you use a sample.
Sampled data is where otherwise very clever people start to become completely unstuck.
Research is based on samples because asking everyone is too expensive. But more importantly (and not widely understood), because asking everyone is completely unnecessary. Quite simple statistical analysis will show you that if you have a decent sized random sample, then the chance of error is very low, and also completely predictable. You can keep asking more and more people, but it won’t make your data more accurate.
A little more fuzzy
Some people do know a little bit about this. They’re the ones who ask you how big your sample was. They think that big samples produce more accurate data. They’re wrong. Or rather, they’re thinking about it the wrong way. The only difference between big and small samples is the margin of error. People don’t really like margin of error, because it makes stories a little more fuzzy.
The challenging thing for journalists is that the way sampling works, margin of error is greatest when opinions are evenly split. So when races are really tight 50/50 splits, it’s very exciting, and everyone wants to pick a winner (leave/remain, Trump/Clinton), but margins of error are higher. Not because polling companies are incompetent. Not because the media is biased, just because that’s how maths works. When it’s a 70/30 race (Macron/LePen), margins of error are narrower, but no one cares if the polls were ten percentage points out, because it doesn’t affect the overall result.
Check the asterisk
And this matters when you’re looking at your brand data, too. That little asterisk the research company puts next to a figure? It usually means that it indicates a significant difference. Not colloquially significant, but statistically significant. It means that there is a high chance (usually a 95% chance), that these two numbers are actually different from one another, rather than within a margin of error. It’s the only way that two numbers can be considered different if they’re based on sampled data.
If the person presenting the data to you uses the word "directional" (to describe differences between numbers that are not statistically significant) then they either don’t know what they’re talking about, or they are way too eager to be liked. And no, it doesn’t matter if all the numbers you want to go up have gone up, but not to the point that any of the differences are statistically significant.
Understanding research well gives you an advantage over your colleagues and your competitors because most of them just don’t bother to understand this stuff. Perhaps they’ve been seduced by that view that research doesn’t work nowadays, and anything can happen. It would be a smart strategy for you to encourage them to think that way.They’re going to make some really dumb decisions.
Craig Mawdsley is the joint chief strategy officer of Abbott Mead Vickers BBDO