Bad data PR: how the NSPCC sunk to a new low in data churnalism

One of the oldest forms of data churnalism is the dodgy poll. Typically used by holiday firms to invent the saddest day of the year, or by property websites to find the happiest places to live you can sometimes excuse journalists for playing along. It’s only a bit of fun, right?

But when the dodgy poll is done with children and relates to porn and sexually explicit videos, you’d expect journalists to exercise a little scepticism.

Unfortunately, when the NSPCC sent out a press release saying that one in ten 12-13 year olds are worried that they are addicted to porn and 12% have participated in sexually explicit videos, dozens of journalists appear to have simply played along – despite there being no report and little explanation of where the figures came from.

Articles on the NSPCC dodgy poll

Dozens of news websites repeated the NSPCC’s claims about porn addiction in children

Only Vice magazine decided to ask questions of the stats. And this is what they found:

“It turns out the study was conducted by a “creative market research” group calledOnePoll. “Generate content and news angles with a OnePoll PR survey, and secure exposure for your brand,” reads the company’s blurb. “Our PR survey team can help draft questions, find news angles, design infographics, write and distribute your story.

“… The OnePoll survey included just 11 multiple-choice questions, which could be filled in online. Children were recruited via their parents, who were already signed up to OnePoll.”

There are so many methodological issues here I can’t list them all, but let’s try. Firstly, there’s the issue of how representative OnePoll users are as a whole and how accurately they complete the survey (the site pays 20p per survey completed, and you have to reach £40 before you can withdraw). There’s the issue of self-selection (PDF) and of whether children are in an environment to give honest answers. And there’s the issue of leading questions, an issue which OnePoll has been criticised for in the past.

As Vice’s article points out, research into this area is normally carried out very carefully to avoid these problems:

“When the London School of Economics carried out research into children’s internet usage a long list of safeguards were put in place, knowing that children would be asked about sensitive topics such as porn. These included pilot tests to gauge children’s state of mind, face-to-face interviews, a self-completion section for sensitive questions to avoid being heard by parents, family members or the interviewer, detailed surveys about the children themselves and measures of mediating factors such as psychological vulnerability. You can read the full 60-page report here.

“Professor Clarissa Smith is Professor of Sexual Cultures at the University of Sunderland and a veteran researcher in the field of young people and sexuality [says]:

“‘There’s absolutely no way an organisation like [OnePoll] could conduct the kind of in depth interviews you need to really engage with young people on pornography. I cannot conceive of a child answering honestly in front of a parent. The dimensions of parent-pleasing there are horrific. I wouldn’t want to sit and answer a questionnaire about porn in front of my dad.'”

There’s also the very important omission of any information on the margin of error. The margin of error (also called the confidence interval) is the range within which the survey’s figures can be reasonably confident of being accurate.

The smaller the sample size (compared to the number of 12-13 year olds in the UK as a whole), the wider the margin of error. According to this press release 700 of the respondents were aged 12-13, generalising to a wider population of 1.36 million children aged 12 or 13 (based on 2011 census figures).

It’s not as if NSPCC doesn’t have the money to conduct a proper piece of research: in its most recent accounts (PDF) it spent £25m on ‘child protection advice and awareness’

NSPCC activities costsUnfortunately by cutting corners on awareness-raising research the charity ends up doing much more costly damage to its reputation. It’s a false economy.

So next time a charity claims to have done a survey which throws up some surprising figures, ask to see the report behind it.

And if you’re a PR person with a children’s charity, make sure you can say what safeguarding your polling company has put in place.

h/t Glyn Moody

UPDATE [April 6]:

The NSPCC have issued a press release which appears to defend the survey results on the basis that they are “from real young people” and “consistent with” two pieces of research they link to.

But the research listed does not refer to porn addiction or participation in sexually explicit videos, instead dealing with the broader issue of porn consumption. One of the two pieces of research also deals with an older age group.

Ironically, one of the briefing papers at the link does state that:

“The media can contribute to public perceptions that online abuse or bullying is more widespread than is in fact the case.”

OnePoll is defended as “an established company that works with nearly every major charity in the sector” but again, concerns over methodology or appropriateness are not directly addressed other than the line “they were able to ensure that all the young people taking part had full parental consent.”

Management Today are more strident in their criticism of the company:

“What naive readers may not realise is that much of what is reported as scientific is not in fact genuine research at all, but dishonest marketing concocted by PR firms. The most prolific company in the business is OnePoll who pay their members 10p per survey with a minimum payout of £40, meaning that their members have to complete four hundred surveys before they can claim any money. It’s not hard to imagine how this could make the survey participants just a little button-happy, but that’s just the beginning of the story.”

The charity do announce that they have:

“In conjunction with the Office of the Children’s Commissioner and the British Board of Film Classification (BBFC) commissioned Middlesex University to undertake a study on the impact of pornography on young people’s behaviours and we will be sharing the results of that study in due course.”

This was announced in January. But while we wait for the Middlesex study, Conservative internet censorship policy is already being argued on the basis of the OnePoll work.

Some users of OnePoll have posted comments since this post was published, describing their own experiences of filling in surveys on children for the research firm.

11 thoughts on “Bad data PR: how the NSPCC sunk to a new low in data churnalism

  1. pfstjjp

    “But when the dodgy poll is done with children and relates to porn and sexually explicit videos, you’d expect journalists to exercise a little scepticism”. Actually, on this particular subject, scepticism is the very last thing which I would expect from British journalists.

    Reply
  2. Pingback: Bad data PR: how the NSPCC sunk to a new low in data churnalism | Liquid Newsroom

  3. Pingback: Start up: hacking nannycams, S6 SD/battery poll, Watch wait, and more | The Overspill: when there's more that I want to say

  4. Alan G

    OnePoll is notorious for this type of sensationalist polling.
    Many of these organisations pay people tiny amounts to fill in surveys.
    The particpants only incentive is to fill in as many surveys as possible as fast as possible irrespective of whether the survey is applicable to them and certainly without encouraging any thought.
    Using OnePoll data is dangerous, it risks hard won credibility, and it costs money.
    If by any chance OnePoll did not charge for the data, I would take that as a sign they are trying to claim credibility by association, “OnePoll as used by the NSPCC.”

    Reply
  5. Robert S

    “This was announced in January, which begs the question why they felt they had to commission the separate, PR-oriented survey.”

    It doesn’t beg the question as that’s a form of logical fallacy. What you should have said is “raises the question”.

    Reply
  6. Sarah

    I’ve personally answered lots of surveys on many sites like YouGov, OnePoll, ICM and they often appear in the press. OnePoll are actually one of the better payers. IPSOS pay 5p sometimes.

    Reply
  7. Google Gaspar Statements

    I am one of the OnePoll panel. ALL surveys for children are fundamentally flawed. Here’s why: You log on and are presented with a list of open surveys. There is no way of knowing which are for kids and which are not. If you click on one and it is for kids it says something like “This is for children aged X to Y, if they are available please will hand over to them.now”. The problem is there is no option to say “They aren’t here at the moment, keep the survey open”. So you either select “No” and the survey disappears from the list and cannot be selected when the child is available… or you sthink “sod it, I’m ny losing 20p so I’ll select “Yes” and try and guess what my kid might say”. I work in an office of 5 and we all do OnePoll surveys online whilst we are tied up on long phonecalls. We all fill out the children’s surveys. Three of us presumably try and second guess what our kids might say, but two of us haven’t even got kids!! I bet OnePoll never analyse when the surveys are completed… I bet most are filled out when kids would be at school!!😉

    Reply
  8. Pingback: NSPCC’s stats on child addiction to porn don’t stand up to scrutiny | Giving Evidence

  9. Pingback: A new data journalism tool – and a new way of reporting uncertainty | Online Journalism Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s