The problem of more data was investigated by Paul Slovic, Professor of Psychology at the University of Oregon. He ran an experiment with professional horseracing handicap setters in which they were given a list of 88 variables that were useful in predicting a horse’s performance. The participants then had to predict the outcome of the race and their confidence in their prediction. They repeated these tasks with access to different levels of data: either 5, 10, 20, 30 or 40 of the variables.
The results were illuminating. Accuracy was the same regardless of the number of variables used. However, overconfidence grew as more data was harnessed. Experts overestimated the importance of factors that had a limited value. It was only when five data points were used that accuracy and confidence were well calibrated.
Marketers face a similar set of problems. They have access to more data than ever before and many believe that because the information exists they should use it. The Slovic experiment suggests otherwise. We shouldn’t harness data just because we can. Instead, as much time should be spent choosing which data sets to ignore as which to use.
The study of marketing is so young that we would be arrogant to believe that we know it all, or even that we have got the basics right yet. We can draw an analogy with medical practice. For centuries this noble profession has attracted some of the best and brightest people in society, who were typically far better educated than other professionals. Yet for 2,500 years these experts enthusiastically and universally taught and practised bloodletting (a generally useless and often fatal ‘cure’). Only vet recently, about 80 years ago, medical professional started doing the very opposite, and today blood transfusions save numerous lives every day. Marketing manager operate a bit like medieval doctors — working on anecdotal experience, impressions and myth-based explanations.
Consider Nespresso. They sell in distinctive pods, which provide the right amount of coffee for a cup. Because they’re sold in that unit we compare their price to other places selling by the cup, such as Costa or Caffe Nero. When compared to the £2.50 Costa charge, Nespresso pods, costing 30p-37p, feel like a bargain.
But stop for a second and remember back to when they launched. If Nespresso had sold their coffee in standard packaging the natural comparison set would have other brands of roast and ground coffee, like Taylor’s or illy. Their price would have been judged against the norm for other coffees — roughly £4.00 for 227g. Even with tens of millions of pounds of advertising they could never have persuaded consumers to pay £34 for a 454g bag. But that £34 figure equates to 7p per gram, exactly what they’re charging now.
But when Meghan Busse, Duncan Simester and Florian Zettelmayer, academics from MIT and the Kellogg School of Management, investigated they discovered a curious anomaly. In the previous weeks the car companies had been cutting prices so much that the employee discount was generally no better and occasionally more expensive, than existing deals.
The academics hypothesised that it was the price cue, not the price, which mattered. Consumers reacted to the plausibility of the deal rather than the actual discount. When consumers don’t trust brands they treat deals sceptically, but when they’re accompanied by a back story they have more heft.
When you are contemplating promotions don’t rely on an eye-watering discount. Numbers leave customers cold. We’re not natural statisticians – stories move us to action far better.
The experiments prove that it’s hard to overturn negative opinions. Rejecters of your brand are difficult to convince because they interpret your message through a lens of negativity.
As the legendary stock market investor, Charlie Munger, said:
“The human mind is a lot like the human egg, in that the human egg has a shut-off device. One sperm gets in, and it shuts down so that the nnext one can’t get in. The human mind has a big tendency of the same sort.”
An example from Seth Stephens-Davidowitz illustates the problem. He looked at the gender of Katy Perry Facebook fans and found that they were overwhelmingly female. However, Spotify listening data revealed the gender split was much more balanced: Perry was in the top ten artists for both genders. If the music label used the Facebook data to target their advertising they’d be way out.
Does that mean the new data streams are junk and best ignored?
Not at all. Observed data is an improvement on claimed data, but it’s still flawed. To understand customers we need a balanced approach, using multiple techniques. If each technique tells us the same story then we can give it greater credence. If they jar then we need to generate a hypothesis to explain the contradiction.
Let’s go back to the Katy Perry example. A simple explanation would be that while both genders enjoy listening to her, far more women are comfortable expressing that publicly. If a record label wants to sell Katy Perry songs or encourage streaming, then Spotify data would be ideal. However, if they want to promote her concerts, it would be better to use the Facebook numbers. Neither data set is right in any absolutist sense – they are right in certain circumstances.
The obsession with easily quantified date crowds out the need for discretion and judgement.
Two examples illustrate the resulting issues. First is the experience of Terry Leahy who, when he was head of marketing at Tesco, analysed the performance of their gluten-free products. The sales data hinted it was an under-performing section – those that bought gluten-free goods only spent a few pounds on these items each shopping trip. A naive interpretation suggested de-listing them to free up valuable shelf space.
However, sceptical of the number, Leahy interviewed gluten-free shoppers and discovered that their choice of supermarket was determined by the availability of those products. They didn’t want to make multiple shopping trips, so the visited whoever had the specialist goods. After all, every shop had milk and eggs but only sone stocked gluten-free goods. Leahy used this insight to launch Tesco’s hugely successful “Free From” range long before the competition.
One successful example was Sainsbury’s in 2004 who realised much supermarket shopping was done in a daze. “Sleep shopping” as they termed it. Shoppers were buying the same items week in, week out — restricting themselves to the same 150 items despite there being 30,000 on offer.
AMV BBDO, Sainsbury’s creative agency, went to great lengths to dramatise the extent of sleep shopping. They hired a man dressed in a gorilla suit and sent him to a Sainsbury’s to do his week’s shopping. They questioned shoppes as they were leaving the store and a surprisingly low percentage had noticed him. When shoppers are on autopilot it’s hard to grab their attention.
Excerpt from: The Choice Factory by Richard Shotton
Another example, this time involving Manchester United manager, Sir Alex Ferguson, didn’t have such a happy ending. Opta data showed that his star defender, Jaap Stam, was making fewer tackles each season. Ferguson promptly offloaded him in August 2001 to Lazio — keen to earn a high transfer fee before the decline became apparent to rival clubs.
However, Stam’s career blossomed in Italy and Ferguson realised his error — the lower number of tackles was a sign of Stam’s improvement, not decline. He was losing the ball less and intercepting more passes that he needed to make fewer tackles. Ferguson says selling Stam was the biggest mistake of his managerial career. From then on he refused to be seduced by simplistic data.
These criticisms don’t mean you should disregard tracking data. Expecting any methodology to be perfect is to burden it with unreasonable expectations. Instead, you need to be aware that it merely provides evidence to which you need to apply your discretion and judgement.
If Rudder’s study hunted at lying, the National Survey of Sexual Attitudes and Lifestyle (NATSAL) categorically confirms it. The survey, conducted among 15,000 respondents by UCL and the London School of Hygiene and Tropical Medicine, is the gold standard of research. In 2010 it found that British heterosexual women admit to a mean of eight sexual partners, compared to twelve for men. The difference is logically impossible. If everyone is telling the truth the mean for each gender must be the same.
All of this foes to show that advertisers trying to understand their customers have a problem: if they listen uncritically to consumers, they’ll be misled.