Cherry-picking
Presenting only the data points, studies, or cases that support a conclusion while ignoring the larger body of evidence that would undermine it.
Any large body of evidence contains outliers, and a determined selector can build a case from them alone. The device works because each cherry is real: the study exists, the year was cold, the patient recovered. The audience checks the cherries, finds them sound, and does not see the orchard. Card stacking is the general habit of one-sided presentation; cherry-picking is its application to evidence, and it is the workhorse of manufactured doubt.
The tell is the start point, the sample, and the silence. A trend that begins at a convenient peak, a survey that names one study among fifty, a testimonial that omits the rest of the trial: each is true and each misleads by what surrounds it. Exploits Confirmation Bias and Availability Heuristic.
Historical examples
For fifteen years, claims that global warming had "stopped" or "paused" were built by starting the trend line at 1998, an exceptionally warm year driven by a strong El Niño. Any series starting a year earlier or later showed the underlying warming. The choice of start point was the whole argument, and it was repeated across opinion pages and hearings.
Oreskes, Naomi, and Erik M. Conway. Merchants of Doubt. New York: Bloomsbury, 2010; Lewandowsky, Stephan, James S. Risbey, and Naomi Oreskes. "The 'Pause' in Global Warming: Turning a Routine Fluctuation into a Problem for Science." Bulletin of the American Meteorological Society 97, no. 5 (2016): 723–733.
The industry's public case rested on the studies it chose to cite: individual papers with null results, animal studies that failed to produce tumours, statisticians who questioned one method. The dozens of studies establishing the link were not mentioned. Internal documents show the selection was managed by counsel.
Proctor, Robert N. Golden Holocaust. Berkeley: University of California Press, 2011; Brandt, Allan M. The Cigarette Century. New York: Basic Books, 2007.
In advertising and commerce
Product pages that display selected reviews, film advertisements that quote the one critic who liked the picture, and "before and after" photographs of the trial's best responder are all cherry-picking applied to sales. Regulators in several countries now require that testimonials reflect typical results, a rule that exists because they didn't.
Federal Trade Commission. Guides Concerning the Use of Endorsements and Testimonials in Advertising. 16 CFR Part 255, revised 2009.
Find a claim backed by "a study." Look for the other studies on the same question. How many are there, and which way do most of them point?
In the wild
Short examples of the technique at work, each with what to notice. The same examples are questions in the quiz, where the answer is not given away.
"Unemployment has risen 30% since the mayor took office." The series starts at the lowest month in twenty years.
What to notice. The start point is chosen for the trend it produces; the fuller series tells a different story.
"A 2011 study found no link between the additive and illness." Forty other studies found one.
What to notice. One real study is offered against the body of evidence, which goes unmentioned.
"Three of our clients doubled their income in a year." The program had four hundred clients.
What to notice. Cases without a denominator; the best three stand for the four hundred.
Sources
- Oreskes, Naomi, and Erik M. Conway. Merchants of Doubt. New York: Bloomsbury, 2010. ISBN 978-1596916104
- Lewandowsky, Stephan, James S. Risbey, and Naomi Oreskes. "The 'Pause' in Global Warming." Bulletin of the American Meteorological Society 97, no. 5 (2016): 723–733. doi:10.1175/BAMS-D-14-00106.1
- Huff, Darrell. How to Lie with Statistics. New York: W. W. Norton, 1954. ISBN 978-0393310726
- Proctor, Robert N. Golden Holocaust. Berkeley: University of California Press, 2011. ISBN 978-0520270169