One of the oldest forms of data churnalism is the dodgy poll. Typically used by holiday firms to invent the saddest day of the year, or by property websites to find the happiest places to live you can sometimes excuse journalists for playing along. It’s only a bit of fun, right?

But when the dodgy poll is done with children and relates to porn and sexually explicit videos, you’d expect journalists to exercise a little scepticism.

Unfortunately, when the NSPCC sent out a press release saying that one in ten 12-13 year olds are worried that they are addicted to porn and 12% have participated in sexually explicit videos, dozens of journalists appear to have simply played along – despite there being no report and little explanation of where the figures came from.

Only Vice magazine decided to ask questions of the stats. And this is what they found:

“It turns out the study was conducted by a “creative market research” group calledOnePoll. “Generate content and news angles with a OnePoll PR survey, and secure exposure for your brand,” reads the company’s blurb. “Our PR survey team can help draft questions, find news angles, design infographics, write and distribute your story.

“… The OnePoll survey included just 11 multiple-choice questions, which could be filled in online. Children were recruited via their parents, who were already signed up to OnePoll.”

There are so many methodological issues here I can’t list them all, but let’s try. Firstly, there’s the issue of how representative OnePoll users are as a whole and how accurately they complete the survey (the site pays 20p per survey completed, and you have to reach £40 before you can withdraw). There’s the issue of self-selection (PDF) and of whether children are in an environment to give honest answers. And there’s the issue of leading questions, an issue which OnePoll has been criticised for in the past.

As Vice’s article points out, research into this area is normally carried out very carefully to avoid these problems:

“When the London School of Economics carried out research into children’s internet usage a long list of safeguards were put in place, knowing that children would be asked about sensitive topics such as porn. These included pilot tests to gauge children’s state of mind, face-to-face interviews, a self-completion section for sensitive questions to avoid being heard by parents, family members or the interviewer, detailed surveys about the children themselves and measures of mediating factors such as psychological vulnerability. You can read the full 60-page report here.

“Professor Clarissa Smith is Professor of Sexual Cultures at the University of Sunderland and a veteran researcher in the field of young people and sexuality [says]:

“‘There’s absolutely no way an organisation like [OnePoll] could conduct the kind of in depth interviews you need to really engage with young people on pornography. I cannot conceive of a child answering honestly in front of a parent. The dimensions of parent-pleasing there are horrific. I wouldn’t want to sit and answer a questionnaire about porn in front of my dad.'”

There’s also the very important omission of any information on the margin of error. The margin of error (also called the confidence interval) is the range within which the survey’s figures can be reasonably confident of being accurate.

The smaller the sample size (compared to the number of 12-13 year olds in the UK as a whole), the wider the margin of error. According to this press release 700 of the respondents were aged 12-13, generalising to a wider population of 1.36 million children aged 12 or 13 (based on 2011 census figures).

It’s not as if NSPCC doesn’t have the money to conduct a proper piece of research: in its most recent accounts (PDF) it spent £25m on ‘child protection advice and awareness’

Unfortunately by cutting corners on awareness-raising research the charity ends up doing much more costly damage to its reputation. It’s a false economy.

So next time a charity claims to have done a survey which throws up some surprising figures, ask to see the report behind it.

And if you’re a PR person with a children’s charity, make sure you can say what safeguarding your polling company has put in place.

h/t Glyn Moody

UPDATE [April 6]:

The NSPCC have issued a press release which appears to defend the survey results on the basis that they are “from real young people” and “consistent with” two pieces of research they link to.

But the research listed does not refer to porn addiction or participation in sexually explicit videos, instead dealing with the broader issue of porn consumption. One of the two pieces of research also deals with an older age group.

Ironically, one of the briefing papers at the link does state that:

“The media can contribute to public perceptions that online abuse or bullying is more widespread than is in fact the case.”

OnePoll is defended as “an established company that works with nearly every major charity in the sector” but again, concerns over methodology or appropriateness are not directly addressed other than the line “they were able to ensure that all the young people taking part had full parental consent.”

Management Today are more strident in their criticism of the company:

“What naive readers may not realise is that much of what is reported as scientific is not in fact genuine research at all, but dishonest marketing concocted by PR firms. The most prolific company in the business is OnePoll who pay their members 10p per survey with a minimum payout of £40, meaning that their members have to complete four hundred surveys before they can claim any money. It’s not hard to imagine how this could make the survey participants just a little button-happy, but that’s just the beginning of the story.”

The charity do announce that they have:

“In conjunction with the Office of the Children’s Commissioner and the British Board of Film Classification (BBFC) commissioned Middlesex University to undertake a study on the impact of pornography on young people’s behaviours and we will be sharing the results of that study in due course.”

This was announced in January. But while we wait for the Middlesex study, Conservative internet censorship policy is already being argued on the basis of the OnePoll work.

Some users of OnePoll have posted comments since this post was published, describing their own experiences of filling in surveys on children for the research firm.