Just how wrong is wrong?

Declaration of interest: In this post I talk about research done by polling company Ipsos MORI, which builds upon work done previously with the Royal Statistical Society on public understanding of statistics on key social issues. I am a member of the Royal Statistical Society and from 2005 till July 2014 was a member of its staff.

UPDATE 2 November 2014: This post has been updated to reflect information in Ipsos MORI’s technical note on the calculation of the index of ignorance.

Question: which country’s people are most ignorant of key numbers about their society?

Polling company, Ipsos MORI, has an answer. It’s Italy.

They have calculated an “Index of Ignorance” for fourteen (14) countries based on the accuracy of responses to nine questions about society.

First, let me be clear – this is interesting research and useful to some extent in the debates around a range of public policy issues. For those of us in the UK going to the polls on 7 May 2015 – the UK General Election – it is particularly important.

But we should be cautious.

Here’s Ipsos MORI’s full ranking – their ‘least accurate’ first.

Position Country
1st Italy
2nd USA
3rd South Korea
4th Poland
5th Hungary
6th France
7th Canada
8th Belgium
9th Australia
10th Great Britain
11th Spain
12th Japan
13th Germany
14th Sweden

How sure of this can we be?

The answer is… who knows?

The problem with any ranking – as sports fans will know – is that it doesn’t matter how big the difference between figures.

So, it could be that Italy is a lot more ‘ignorant’ than the USA, but Sweden just pips Germany to most knowledgeable.

More uncertainty comes from how the polling was done – who was asked and what the sample size was.

Here’s another table…

Country Who How many
Italy 16-64 1,000
USA 18-64 1,000
South Korea 16-64 500
Poland 16-64 500
Hungary 16-64 500
France 16-64 1,000
Canada 18-64 1,000
Belgium 16-64 500
Australia 16-64 1,000
Great Britain 16-64 1,000
Spain 16-64 1,000
Japan 16-64 1,000
Germany 16-64 1,000
Sweden 16-64 500

NB: the figures for ‘how many’ are approximate. Ipsos MORI give either 500+ or 1,000+.

Now, any pollster will caution that the smaller the sample size the greater the uncertainty aka margin of error. For a ‘perfect’ sample survey, this figure is a little under 4.5 percentage points either way for a sample of 500; a tad over 3 percentage points either way for a sample of 1,000. This is for the usual 95 per cent confidence level.

For underlying reasons about survey statistics, the uncertainty figures won’t be quite 3 or 4.5 percentage points. But they are a good guide.

This means the ‘true’ figure for any particular country on any particular question is most likely to be in a range of 6 percentage points or 9 percentage points, depending on sample size.

So the ‘index of ignorance’ has been calculated from numbers that could readily be 3 or 4.5 percentage points out for each of eight questions.

In addition, it has been done by taking the mean average of the individual estimates. The problem with the mean is that any response that is particularly wrong can have, in a sense, a disproportionate effect. One way to think of this is to imagine you and a group of friends at a meeting. The mean income of the group will be massively increased if Bill Gates joins the group unless Bill is a friend already and was already in the room!).

Bear in mind also for two countries (USA, Canada) those polled were aged 18 or above, the rest were 16 plus. And that it’s not clear which average has been used; and what the uncertainty in the ‘correct’ figure for each country is.

This suggests that the ranking can at best be treated as ‘just a bit of fun’.

At worst, though, public perception may be that it is ‘official’ that Italians are more ignorant than those of thirteen other countries.

Apparently, if the English Premier League – a ranking system with some certainty – can’t be settled on points then it goes to goal difference, and then goals scored.If there is still no winner then there’s a play-off.

Thankfully, Great Britain (10th) and Germany (13th) are ranked three places apart. Otherwise, we’d likely lose on penalties after extra time in the World Ignorance Cup!

NB how ignorant are you? Take the Ipsos MORI quiz for yourself.


Who ate all the Smarties?

I’ve used Smarties many times in statistics workshops to introduce ideas of sampling and distributions.*

In a loose sense, each box is a sample from the “Great Smarties Universe”. Individual boxes don’t all contain exactly the same number of sweets. Nor do they have the same distribution of colours.

So I thought I’d use Smarties in my presentation on communicating statistics at an event organised by the Chartered Institute of Public Relations’ Greater London Group.

I put out lots of boxes ahead of the talk so that attendees could count the total number of sweets in a box. We could then see how much variation in numbers there was. We could even get counts of the numbers for different colours.

But #fail.

As my co-presenter, Katharine Peacock of ComRes, was giving excellent advice on commissioning surveys, I became very aware of much munching in the audience.

Out of 35 boxes I got counts of total numbers from just seven: 30, 30, 30, 31, 32, 32 and 32. How can we interpret these numbers? From lowest to highest that’s just two sweets – so not much variation. Is that as much variation as ever occurs? Are there boxes with 29 or fewer Smarties, or 33 and more?

It would be great to have got more data. Unfortunately it was eaten.

*Other colourful sweets are available and could be used, of course!