4.9/5 - (34 votes)

A stranger points at a chalk target on a barn door, a bullet hole dead in its center, and swears he hit it from a hundred yards away. He is telling the truth. He is also cheating, and the trick he used still fools scientists, doctors and pretty much everyone else today.

A bullet hole, a piece of chalk and a true story

The scene comes from the mathematician John Venn, who described it in 1866. The man really had fired the shot from an old fowling-piece at a hundred yards. What he left out was the order of events.

He had aimed vaguely at the barn door, hit it somewhere, and only then chalked the target around the hole. Venn added a warning that still holds up: “A deception analogous to this is, I think, often practised unconsciously in other matters.”

The story is even older than Venn. A version with arrows is attributed to the Dubno Maggid, who died in 1804. The “Texas” part only shows up in the statistical literature in 1977, perhaps a nod to the Texan reputation for tall tales, and Atul Gawande popularized it in 1999.

Why the trick works so well on smart people

Statisticians today call it the Texas sharpshooter fallacy: finding meaning in data that is essentially random. The key mistake is simple. The hypothesis is written after the data has been seen, not before.

It feeds on a quirk of the human brain called the clustering illusion. We expect randomness to look evenly spread out, so when real random data produces clumps, as it always does, we assume something must be causing them. Daniel Kahneman and Amos Tversky traced this misjudgment to what they called the representativeness heuristic. It is a close cousin of confirmation bias, and it also sits behind the debate over hot streaks in basketball.

See also  Four datasets share the same averages and correlation, yet look nothing alike

The Swedish study that checked 800 illnesses

In 1993, Swedish researchers set out to see whether living near high-voltage power lines harmed health. They surveyed people living within 300 metres of the lines over a 25-year period, then compared them with people living further away on more than 800 medical conditions.

One result jumped out: childhood leukemia was four times more common near the lines. It prompted calls for action by the Swedish government.

The catch is the number 800. Test enough conditions and one of them will look alarming by pure chance. At the usual 5 percent significance threshold, 800 independent tests on something with zero real effect would produce about 40 false alarms on average, and the odds of getting at least one are practically 100 percent. Later studies failed to find any link between power lines and childhood leukemia.

Miracle drugs, twins and Nostradamus

The same target painting shows up in drug research. A drug with no effect at all can look impressive in one small subgroup if researchers run enough analyses. A useless pill might seem highly effective in twins, for instance, even though nobody planned to study twins. Run the test again and the effect vanishes.

Prophecy works the same way. Modern readers of Nostradamus take his quatrains, translate them loosely from Middle French, strip away their historical context, and match them to events only after those events have happened. The target always gets drawn once the bullet has landed.

The bombs that fell on London

During the Second World War, Londoners stared at maps of where German flying bombs had struck and built precise theories about the pattern.

See also  The Sports Illustrated cover jinx isn't magic, it's just math

In an analysis published in 1946, R. D. Clarke showed the impacts were a close fit to a purely random distribution. The clusters people feared were exactly what chance produces.

That does not mean every cluster is an illusion. Real ones exist, from scrotal cancer among chimney sweeps in 18th-century London to bone cancer in women who painted watch dials. In the US, health departments receive more than 1,000 reports of suspected cancer clusters a year, yet only 5 to 15 percent of suspected clusters turn out to be statistically significant. A cluster deserves more attention when it involves a single type of cancer, a rare one, or one unusual for the age group.

Draw the target first

The cure Venn hinted at is almost childishly simple. Decide what you are looking for before you look. If the data suggests a new idea, test it on fresh data, never on the same numbers that gave you the idea.

It sounds obvious, yet a lot of headlines about miracle foods, cursed neighborhoods and uncanny predictions are still just bullet holes with a neat circle of chalk around them.