Misleading statistics
Presenting numbers that are accurate in themselves but chosen, scaled, or displayed so that the audience draws a conclusion the full data would not support.
Numbers carry authority, and most people cannot audit them. The device exploits the gap with a small set of moves that Darrell Huff cataloged in 1954 and that have not changed: the relative increase without the absolute base ("risk doubles," from one in a million to two); the truncated axis that turns a ripple into a cliff; the average that hides the distribution; the sample too small or too selected to mean anything; the correlation presented as cause.
Each figure survives fact-checking. The deception is in what the audience is led to conclude from it, and the device counts on the reader's respect for a number being greater than their ability to interrogate it. Exploits Anchoring and Denominator Neglect.
Historical examples
Huff's How to Lie with Statistics collected the moves in one short book: "the well-chosen average," "the gee-whiz graph," "the semi-attached figure," "post hoc rides again." Its examples came from advertising and journalism of the time, and it has sold more copies than any other statistics book because the moves it describes are still in daily use.
Huff, Darrell. How to Lie with Statistics. New York: W. W. Norton, 1954.
Gigerenzer and colleagues documented how medical risks are reported in relative terms, "cuts the risk by 50%," when the absolute change was from two cases in a thousand to one, and showed that patients, journalists, and doctors alike misjudged the benefit when given relative figures. Their proposed remedy, natural frequencies, is the counter-move in statistical form.
Gigerenzer, Gerd, et al. "Helping Doctors and Patients Make Sense of Health Statistics." Psychological Science in the Public Interest 8, no. 2 (2007): 53–96. Source
In advertising and commerce
"Four out of five dentists surveyed recommend sugarless gum for their patients who chew gum," the Trident claim of the 1960s, is a model of the semi-attached figure: the survey concerned sugarless gum in general, not the brand; the population was dentists who were asked about patients who already chewed; and the fifth dentist presumably recommended not chewing. The number was real and the conclusion drawn from it was the advertiser's.
Best, Joel. More Damned Lies and Statistics: How Numbers Confuse Public Issues. Berkeley: University of California Press, 2004; Huff, Darrell. How to Lie with Statistics. New York: W. W. Norton, 1954, on "the semi-attached figure."
Find a statistic in the news that says something rose or fell by a percentage. Find the actual numbers before and after. Does the percentage give the same impression?
In the wild
Short examples of the technique at work, each with what to notice. The same examples are questions in the quiz, where the answer is not given away.
"Cases of the illness have doubled in the region this year." The count went from three to six.
What to notice. A true relative increase with no base rate; the absolute numbers would not have alarmed anyone.
A chart shows the company's share price "collapsing"; the axis runs from $48 to $50.
What to notice. The truncated axis turns a four-percent dip into a cliff. Chart distortion is the visual form of the same device.
"The average bonus at the firm was $80,000." One partner received $2 million; most staff received $2,000.
What to notice. The well-chosen average hides the distribution; the median would tell the truth.
Sources
- Huff, Darrell. How to Lie with Statistics. New York: W. W. Norton, 1954. ISBN 978-0393310726
- Best, Joel. Damned Lies and Statistics. Berkeley: University of California Press, 2001. ISBN 978-0520219786
- Gigerenzer, Gerd, Wolfgang Gaissmaier, Elke Kurz-Milcke, Lisa M. Schwartz, and Steven Woloshin. "Helping Doctors and Patients Make Sense of Health Statistics." Psychological Science in the Public Interest 8, no. 2 (2007): 53–96. doi:10.1111/j.1539-6053.2008.00033.x
- Tufte, Edward R. The Visual Display of Quantitative Information. 2nd ed. Cheshire: Graphics Press, 2001. ISBN 978-1930824133