In the summer of last year ProPublica published a major investigation into air pollution in Florida, and its connection to the sugar industry. The story itself, Black Snow, is an inspiring example of scrollytelling — but equally instructive is the methodology article which accompanies it, responding to criticisms from the sugar industry.
Not only does it demonstrate how to respond when large organisations attack a piece of journalism — it also provides a great lesson on the tactics that are adopted by organisations when attacking data-driven stories.
In this post I want to break down the three most common attack tactics, how ProPublica deal with two of those, and how to use the same tactics during planning to ensure your project design isn’t flawed.
Tactic 1: Cherry picking
When an organisation is presented with data that they don’t like, they may point to other data that they prefer.
This is often (but not always) a form of cherry picking: instead of gathering a range of the evidence to reach a conclusion, only select evidence is presented.
Here’s the form that takes in the ProPublica story:
A U.S. Sugar spokesperson said that “data collected and authorized by local health departments and the FDEP continue to show that the air quality in the Glades is safe, healthy and consistently better than the standards set by the EPA.”
ProPublica explain in detail how that data differs from the data that they have collected using sensors, and why the particular methodology was used.
They also involve experts in the planning and reviewing of the method, too, to ensure that any potential issues are addressed at the planning stage.
Tactic 2: Ad hominem attacks
“Ad hominem” arguments attack the person, not the information. This is perhaps the most common tactic used by US Sugar. In one line, for example, they attack the “low-cost” (note: not “low quality”) sensors “operated by non-professionals”.
ProPublica’s response points out that the Environmental Protection Agency (EPA) uses the same sensors, and has already explored potential flaws in the sensors, adopting the same approach as the EPA.
The second example of this tactic comes in this line:
U.S. Sugar questioned how we found our sensor hosts, pressing us on whether they were activists with the Sierra Club.
Anticipating this is important — and ProPublica seem to have done so:
They are not. None of the residents who hosted the monitors have worked with environmental advocacy organizations in the Glades.
[…] We used a number of tools to reach hundreds of residents, including sending letters to public school teachers and custodians across the Glades; knocking on doors in neighborhoods in Belle Glade, Pahokee and South Bay; attending a virtual church service; canvassing food distribution sites; delivering flyers to businesses and nonprofits; and calling local doctors and nurses.
Tactic 3: The straw man
There is a third tactic to mention here as well, which is to suggest that an article says something that it doesn’t by responding to an exaggerated or distorted version of its basic premise.
We don’t really see this in the ProPublica example, but the companies could, for example, have denied that 100% of the pollution was created by sugar cane, or argued that its employees are “good people”, or that it “has always cooperated with environmental laws”.
A right of reply is an attempt to understand, not defend
Identifying these tactics can make it easier to deal with criticism from the subject of an investigation — but it doesn’t necessarily mean that criticism is not justified.
Offering a right of reply should be a genuine attempt to understand a story from another perspective, but it should also be a dialogue as you clarify what they mean, and what they think you mean.
Many people will not even be aware that they are using these strategies — the ad hominem, straw man and cherry picking fallacies are all cognitive biases which form part of unconscious reactions, so it may take time and prompting to help someone to clarify what exactly they are criticising.
As cognitive biases, we are vulnerable to them as well: we can ignore valid criticisms because of who they come from (ad hominem) or because we have not looked at data that contradicts what we expected (cherry picking) or because we hear an exaggerated version of what was actually said (straw man).
The three strategies are also well worn media management tactics, so it can be useful to incorporate them into early story planning, as part of a ‘devil’s advocate’ stress test of the methodology:
- Cherry picking: how do ensure we’re not cherry picking? What data might someone else cherry pick to challenge our findings? Is that data more ‘right’ than ours?
- Ad hominem: who might be attacked as an ‘activist’ or biased in some way? How do we select participants and ensure that the results are reliable?
- Straw man: what are the extreme or distorted versions of our story that might be responded to? How do we make it clear what we are not claiming in the story?
You can find posts about cognitive bias and journalism at onlinejournalismblog.com/tag/cognitive-bias
UPDATE [Feb 2023]: This research also looks at how the objects of investigations respond.
UPDATE [April 2023]: This taxonomy of contrarian claims around global warming appears to follow similar themes. “Impacts are not bad” and “Human greenhouse gases are not causing global warming” both fall into the cherry picking category, while “Movement is unreliable” seems to follow the ad hominem fallacy and arguments like “Too hard” may be a ‘straw man’.
Online Journalism Blog 