Statistics Pitfall #1: Cherry Picking

There are plenty of arguments for letting data guide your decisions. “Numbers don't lie” is a phrase you hear often. At first glance, that's entirely true. Data is an unbiased reflection of reality and a good check on assumptions and gut feelings. Unfortunately, it's not always easy to interpret data correctly. It's no coincidence that universities in the Netherlands, almost without exception, fill part of their curriculum with statistics courses. Anyone who works with data runs the risk of falling into one of many pitfalls. In this series of blogs, we'll therefore explain a number of common pitfalls and give you concrete tips to avoid them. In this first blog, we discuss ‘Cherry Picking’.

Cherry Picking:

Perhaps the best-known pitfall, since it's also often deliberately abused. Cherry picking means pulling exactly the data points from the set that best support your story and leaving out the rest when presenting it to others. A good example is this clip from Zondag met Lubach from October 2016.

The Dutch news ran a bold headline: “Police checks on people with a migration background unjustified in 40% of cases.” That sounds pretty serious, but if you looked at the full study, you'd see it only concerned 12 out of 29 cases reviewed, in a set of 272 stops. The researchers themselves concluded in the report: “the quantitative analysis shows there is no significant relationship between citizens' appearance (…) and the decision to intervene.”

The tricky thing about cherry picking is that it isn't always deliberate. For example, you might come across a KPI object on a dashboard like this one:

Last month conversion was 40% and this month it's 60%. A huge improvement, things are going well! But what if the chart over the past few months looks like this?

That 40% last month was a massive dip, this month we're at 60%, but we're still nowhere near the 97% we hit in the first quarter. So things aren't going well at all! We need to get to work as soon as possible to bring the ratio back up to its old level. Or do we? Because what if we also put last year's figures alongside it?

We achieved a higher conversion rate almost every month compared to last year, and the figures show a similar pattern. So nothing to worry about, we're doing slightly better than last year, but the differences are small.

So you can create a visualization to support pretty much any conclusion with the same dataset. Pretty dangerous, but what can you do to arm yourself against cherry picking? Here are 3 tips:

#1 Ask questions

When someone presents you with data, ask yourself: “What am I not being told? Do I have enough context?” If you have access to the dataset yourself, it's often useful to explore the data on your own. A dashboard can help you with this.

#2 Transparent filters

Make clear which filtering you've applied when presenting data yourself. In Qlik Sense, for example, the selections are always shown at the top of your screen, but in Power BI this isn't always the case. If you're creating your own export or report, it's good practice to note which selections you used.

#3 Definitions

Make sure you have solid definitions for KPIs. If you've clearly agreed in advance how performance indicators should be measured, the room for (deliberate or unintentional) cherry picking becomes significantly smaller.

Written by Lennaert van den Brink
Senior Consultant