Defending an investigation — and planning one: lessons from ProPublica’s Black Snow

Sugar Companies Said Our Investigation Is Flawed and Biased. Let’s Dive Into Why That’s Not the Case.

In the summer of last year ProPublica published a major investigation into air pollution in Florida, and its connection to the sugar industry. The story itself, Black Snow, is an inspiring example of scrollytelling — but equally instructive is the methodology article which accompanies it, responding to criticisms from the sugar industry.

Not only does it demonstrate how to respond when large organisations attack a piece of journalism — it also provides a great lesson on the tactics that are adopted by organisations when attacking data-driven stories.

In this post I want to break down the three most common attack tactics, how ProPublica deal with two of those, and how to use the same tactics during planning to ensure your project design isn’t flawed.

Tactic 1: Cherry picking

When an organisation is presented with data that they don’t like, they may point to other data that they prefer.

This is often (but not always) a form of cherry picking: instead of gathering a range of the evidence to reach a conclusion, only select evidence is presented.

Here’s the form that takes in the ProPublica story:

A U.S. Sugar spokesperson said that “data collected and authorized by local health departments and the FDEP continue to show that the air quality in the Glades is safe, healthy and consistently better than the standards set by the EPA.”

ProPublica explain in detail how that data differs from the data that they have collected using sensors, and why the particular methodology was used.

They also involve experts in the planning and reviewing of the method, too, to ensure that any potential issues are addressed at the planning stage.

Tactic 2: Ad hominem attacks

“Ad hominem” arguments attack the person, not the information. This is perhaps the most common tactic used by US Sugar. In one line, for example, they attack the “low-cost” (note: not “low quality”) sensors “operated by non-professionals”.

ProPublica’s response points out that the Environmental Protection Agency (EPA) uses the same sensors, and has already explored potential flaws in the sensors, adopting the same approach as the EPA.

The second example of this tactic comes in this line:

U.S. Sugar questioned how we found our sensor hosts, pressing us on whether they were activists with the Sierra Club.

Anticipating this is important — and ProPublica seem to have done so:

They are not. None of the residents who hosted the monitors have worked with environmental advocacy organizations in the Glades.

[…] We used a number of tools to reach hundreds of residents, including sending letters to public school teachers and custodians across the Glades; knocking on doors in neighborhoods in Belle Glade, Pahokee and South Bay; attending a virtual church service; canvassing food distribution sites; delivering flyers to businesses and nonprofits; and calling local doctors and nurses.

Tactic 3: The straw man

There is a third tactic to mention here as well, which is to suggest that an article says something that it doesn’t by responding to an exaggerated or distorted version of its basic premise.

We don’t really see this in the ProPublica example, but the companies could, for example, have denied that 100% of the pollution was created by sugar cane, or argued that its employees are “good people”, or that it “has always cooperated with environmental laws”.

A right of reply is an attempt to understand, not defend

Identifying these tactics can make it easier to deal with criticism from the subject of an investigation — but it doesn’t necessarily mean that criticism is not justified.

Offering a right of reply should be a genuine attempt to understand a story from another perspective, but it should also be a dialogue as you clarify what they mean, and what they think you mean.

Many people will not even be aware that they are using these strategies — the ad hominem, straw man and cherry picking fallacies are all cognitive biases which form part of unconscious reactions, so it may take time and prompting to help someone to clarify what exactly they are criticising.

As cognitive biases, we are vulnerable to them as well: we can ignore valid criticisms because of who they come from (ad hominem) or because we have not looked at data that contradicts what we expected (cherry picking) or because we hear an exaggerated version of what was actually said (straw man).

The three strategies are also well worn media management tactics, so it can be useful to incorporate them into early story planning, as part of a ‘devil’s advocate’ stress test of the methodology:

  • Cherry picking: how do ensure we’re not cherry picking? What data might someone else cherry pick to challenge our findings? Is that data more ‘right’ than ours?
  • Ad hominem: who might be attacked as an ‘activist’ or biased in some way? How do we select participants and ensure that the results are reliable?
  • Straw man: what are the extreme or distorted versions of our story that might be responded to? How do we make it clear what we are not claiming in the story?

You can find posts about cognitive bias and journalism at onlinejournalismblog.com/tag/cognitive-bias

Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.