Statistics pitfall #2: Aggregated data

There are plenty of arguments for letting data guide your decisions. "Numbers don't lie" is a common saying, and at first glance that is entirely true. Data is an impartial reflection of reality and a good test for assumptions and gut feelings. Unfortunately, interpreting data correctly is not always easy. It is no coincidence that universities in the Netherlands almost without exception fill part of their curriculum with statistics classes. Anyone who works with data risks stepping into one of the many pitfalls. In this blog series we explain a number of common pitfalls and give you concrete tips to avoid them.

The danger of aggregated data:

On 4 August 2017, the film "An Inconvenient Sequel: Truth to Power" was released, a sequel to Al Gore's hit documentary. About 25 days later, the film had a score of 5.4 on the well-known Internet Movie Database (IMDB), based on a total of 2,645 reviews. A mediocre score that won't necessarily put the film high on many movie lovers' watchlists. What is interesting, however, is that the average score does not seem to tell the whole story. If we look at the distribution of the scores given in a chart, we see the following:

As you can see, opinions about the film are sharply divided, and the scores 1 and 10 in particular were given very often. If you use the average score to decide whether the film is worth watching, you will probably get it wrong, because the chance that you will love it or hate it seems to be almost fifty-fifty. Researchers at ABC news found at the time that there were large differences in opinion between critics and the public, men and women, and even between people who had seen the film and people who had not.

The example of Al Gore's film shows the danger of using aggregated metrics. The advantage of such metrics is that it does not matter how large your dataset is: you can always summarize it using statistical formulas. But as with any summary, you also lose all context. In 1973, the mathematician Anscombe showed that you can construct datasets that appear to be exactly the same dataset when summarized with aggregated metrics. He created 4 datasets that all had the same mean, but also the same variance and correlation. If you visualize them in a chart, however, the datasets all look very different:

Are aggregations "bad" then? No, not necessarily. Like any other summary, they help you quickly get a general idea and can trigger you to investigate further. In fact, they are sometimes even necessary to form the complete picture. A good example is the University of Berkeley case. In the 1970s, Berkeley was accused of discriminating on gender: women were structurally admitted less often than men. Researchers looked at the data of the university's 6 largest departments and saw the following:

In the table, the figures in the column where they are highest for that category are underlined per department. If you look at these 6 departments, it actually appears to be the reverse. In 4 of the 6 departments, a higher percentage of women than men were admitted. Looking at the total, however, we see confirmation of the claim that men are admitted more often. This effect is also known as Simpson's Paradox. What happens here is that women more often applied to programs where many people are rejected anyway.

As has often proven to be the case, the way you look at your data is crucial to forming the right insight. The tips below can help you avoid the pitfalls of aggregated data:

#1 Summary

Treat an aggregated metric as what it is: a summary. Give the user of your dashboard context by visualizing the data.

#2 Totals

Adding a total row to a table makes it easier to recognize an effect like Simpson's paradox.

#3 Check size

Check your data for sample size. Simpson's paradox occurs when you have many more observations for one dimension value than for another dimension value.

More statistics pitfalls?

Has this blog made you curious about the other blogs in this series? Read them all at your leisure via the buttons below.

Pitfall 1Pitfall 3 Pitfall 4