Statistics pitfall #4: Unbiased Sample

There are plenty of reasons to let data guide your decisions. "Numbers don't lie" is a phrase you hear often, and at first glance it is entirely true. Data is an impartial reflection of reality and a good test for assumptions and gut feelings. Unfortunately, interpreting data correctly is not always easy. It is no coincidence that universities in the Netherlands almost without exception fill part of their curriculum with statistics classes. Anyone who works with data risks falling into one of the many pitfalls. In this blog series we explain a number of common pitfalls and give you concrete tips to avoid them. In this fourth and final blog of the series, we discuss the 'Unbiased Sample'.

The search for an unbiased sample:

These days we all collect as much data as possible. After all, data is the "new gold" and should help us make better decisions. In the previous blogs we showed that you need to be careful how you analyze data to avoid drawing the wrong conclusion. Today we explain that the way you collect the data can also have a major influence on getting the right results.

In 1948, the American presidential election was between Truman and Dewey. It was a tense race, but the newspaper The Chicago Daily Tribune was certain. They had conducted a poll, and the Republican candidate Dewey was going to win. They were so sure of themselves that they had already printed the June 6 edition with the headline "Dewey defeats Truman" before the official result was known. The result, however, was an overwhelming win for Truman, who triumphantly posed for the photographers with the newspaper in his hand.

What went wrong?

The newspaper had conducted its polls by randomly picking telephone numbers and calling them, asking whom the person who answered the phone would vote for. In 1948, however, a home telephone was a luxury item, so it was mainly the wealthier middle-class Americans who had a telephone number. As a result, the newspaper's sample over-represented Republicans.

Another pitfall in collecting data is the so-called "survivorship bias". During the Second World War, the American military studied its combat aircraft. They wanted to reinforce the aircraft with extra armor. To determine where to put the extra armor, they looked at the bullet holes in aircraft that had returned after being shot at.

Initially they decided to put armor on the fuselage and the wing tips. After all, that is where the aircraft were hit most often. The statistician Abraham Wald, however, advised reinforcing the cockpit and the engines instead. Why? Because all the aircraft that were analyzed had flown back. The aircraft whose cockpit or engines were hit had all gone down, and so were not part of the set of aircraft in the analysis.

So it is important to think carefully about who or what you measure when collecting data, but how you carry out the measurement can also have unintended effects. Suppose, for example, that you ask a group of people in a survey which color they like best: red, green or yellow? The results are:

You cannot then claim that their favorite color is green, only that it is preferred over red and yellow. After all, you gave them a choice between those three options, but colors such as blue or orange could not be given as an answer. Factors such as the wording of the question and even the order in which you ask them can also influence the answers.

In fact, the very fact that you are trying to measure an effect can influence the values you measure. That sounds paradoxical, but it is a well-known phenomenon, also known as the "Hawthorne Effect". Hawthorne Works was a factory in Illinois in the 1920s where people tried to increase efficiency by making small adjustments and then studying the effect on productivity. Most adjustments seemed to have a positive effect during the study itself, but these effects disappeared almost immediately once the study stopped. The analysis showed that the employees worked harder as soon as the researchers were "watching over their shoulders". We also often see a similar effect in election exit polls, where voters tell the researcher they voted for the "socially desirable" party, while in reality they voted for a completely different one.

4 tips for good samples

A good analysis therefore starts with collecting good data. When working with data you will often hear the mantra "garbage in, garbage out". Here are a few tips to improve the quality of your sample:

#1 Random samples

If you cannot measure all transactions, try to work with random samples as much as possible. The more random, the better, so also vary the moments at which you measure, for example.

#2 Quotas

If you know in advance which factors have a (large) influence on your process, you can apply quotas to your sample. For example, if gender is important and you know the male-female split in your target group is 50%-50%, you survey 10 men and 10 women (this is also known as "Stratified Sampling").

#3 Stratified Sampling

If you have collected enough data, you can also apply Stratified Sampling after the fact by randomly drawing data points from your full dataset that meet the required categories and leaving the rest of the data out of your study.

#4 A good sample

The Hawthorne effect mainly occurs when the people being observed feel threatened. For a good sample, it is in your own interest to ensure a socially safe and open atmosphere. Sometimes it is possible to measure without the observed person being aware of it, but watch out for ethical and moral objections.

More statistics pitfalls?

Has this blog made you curious about the other blogs in this series? Read them all at your leisure via the buttons below.

Pitfall 1 Pitfall 2 Pitfall 3