On the danger of priming in surveys

The responses to questions can also be influenced by what has been asked beforehand, a process known as priming. Official surveys of wellbeing estimate that around 10% of young people in the UK consider themselves lonely, but an online questionnaire by the BBC found the far higher proportion of 42% among those choosing to answer. This figure may have been inflated by two factors: the self-reported nature of the voluntary ‘survey’, and the fact that the question about loneliness had been preceded by a long series of enquires as to whether the respondent in general felt a lack of companionship, isolated, left out, and so on, all of which might have primed them to give a positive response to the crucial question of feeling lonely.

Excerpt from: The Art of Statistics: Learning from Data by David Spiegelhalter

Before believing a study take a look at its methodology

For example, in 2017 budget airline Ryanair announced that 92% of their passengers were satisfied with their flight experience. It turned out that their satisfaction survey only permitted the answers, ‘Excellent, very good, good, fair, OK’.

Excerpt from: The Art of Statistics: Learning from Data by David Spiegelhalter

In our keenness to attribute success or failure to an intervention, we often forget that the change may well be a case of reversion to the mean

It is not only sports teams that are ranked in league tables. Take the example of the PISA Global Education Tables, which compare different countries’ school systems in mathematics. A change in league table position between 2003 and 2013 was strongly negatively correlated with initial position, meaning that countries at the top tended to go down, and those at the bottom tended to go up. The correlation was -0.60, and some theory shows that if the rankings were complete chance and all that was operating were regression-to-the-mean, the correlation would be expected to be -0.71, not very different from what was observed. This suggest the differences between countries were far less than claimed, and that change in league position had little do with changes in teaching philosophy.

Excerpt from: The Art of Statistics: Learning from Data by David Spiegelhalter