Consider these two unrelated events:

1. A bill is proposed to record every contact (and possibly search) made by every UK citizen, to be available to law enforcement agencies and stored by communication service providers
2. An inquiry into press standards and a leaked Home Office report both uncover the ease with which private investigators can access personal records through law enforcement and other agencies

I’m worried about 1. because of 2. And tonight’s Dispatches: Watching the Detectives does a particularly good job of illustrating why. It is “the ease and extent to which the unregulated private investigation industry is willing to acquire personal data for a price” – not just from the police services, but the health services, benefits system, and other bodies, including commercial ones such as communications service providers (for an illustration of the data security of private companies, witness the Information Commissioner’s Office targeting them after a series of data protection breaches).

If you’re a journalist, student journalist or blogger with any interest in protecting your sources, you should be watching the Communications Data Bill closely and understanding how it affects your job.

In the meantime, it’s also worth developing some good habits to protect your stories and your sources against unwanted snooping. More on my Delicious bookmarks under ‘security’.

PHP Freelancer

There have been quite a few scraping-related stories that I’ve been meaning to blog about – so many I’ve decided to write a round up instead. It demonstrates just the increasing role that scraping is playing in journalism – and the possibilities for those who don’t know them:

Scraping company information

Chris Taggart explains how he built a database of corporations which will be particularly useful to journalists and anyone looking at public spending:

“Let’s have a look at one we did earlier: the Isle of Man (there’s also one for Gibraltar, Ireland, and in the US, the District of Columbia) … In the space of a couple of hours not only have we liberated the data, but both the code and the data are there for anyone else to use too, as well as being imported in OpenCorporates.”

OpenCorporates are also offering a bounty for programmers who can scrape company information from other jurisdictions.

Scraperwiki on the front page of The Guardian…

The Scraperwiki blog gives the story behind a front page investigation by James Ball on lobbyist influence in the UK Parliament:

“James Ball’s story is helped and supported by a ScraperWiki script that took data from registers across parliament that is located on different servers and aggregates them into one source table that can be viewed in a spreadsheet or document.  This is now a living source of data that can be automatically updated.  http://scraperwiki.com/scrapers/all_party_groups/

“Journalists can put down markers that run and update automatically and they can monitor the data over time with the objective of holding ‘power and money’ to account. The added value  of this technique is that in one step the data is represented in a uniform structure and linked to the source thus ensuring its provenance.  The software code that collects the data can be inspected by others in a peer review process to ensure the fidelity of the data.”

…and on Channel 4′s Dispatches

From the Open Knowledge Foundation blog (more on Scraperwiki’s blog):

“ScraperWiki worked with Channel 4 News and Dispatches to make two supporting data visualisations, to help viewers understand what assets the UK Government owns … The first is a bubble chart of what central Government owns. The PDFs were mined by hand (by Nicola) to make the visualisation, and if you drill down you will see an image of the PDF with the source of the data highlighted. That’s quite an innovation – one of the goals of the new data industry is transparency of source. Without knowing the source of data, you can’t fully understand the implications of making a decision based on it.

“The second is a map of brownfield landed owned by local councils in England … The dataset is compiled by the Homes and Communities Agency, who have a goal of improving use of brownfield land to help reduce the housing shortage. It’s quite interesting that a dataset gathered for purposes of developing housing is also useful, as an aside, for measuring what the state owns. It’s that kind of twist of use of data that really requires understanding of the source of the data.

Which chiropractors were making “bogus” claims?

This is an example from last summer. Following the Simon Singh case Simon Perry wrote a script to check which chiropractors were making the same “bogus claims” that Singh was being sued over:

“The BCA web site lists all it’s 1029 members online, including for many of them, about 400 web site URLs. I wrote a quick computer program to download the member details, record them in a database and then download the individual web sites. I then searched the data for the word “colic” and then manually checked each site to verify that the chiropractors were either claiming to treat colic, or implying that chiropractic was an efficacious treatment for it. I found 160 practices in total, with around 500 individual chiropractors.

“The final piece in the puzzle was a simple mail-merge. Not wanting to simultaneously report several quacks to the same Trading Standards office, I limited the mail-merge to one per authority and sent out 84 letters.

“On the 10th, the science blogs went wild when Le Canard Noir published a very amusing email from the McTimoney Chiropractic Association, advising their members to take down their web site. It didn’t matter, I had copies of all the web sites.”

PHP Freelancer

dispatches is a new current affairs quarterly with a companion website, Rethink-Dispatches.com featuring original content as well as extracts from the magazine.

Virtual Intern Natalie Chillington put forward a few questions to editor and art director Gary Knight about the online side to dispatches.

Why is dispatches not published online?

“Principally [the fact,] that our content is not web compatible in so far as web users don’t respond to 25,000 words and 80 images. The point of dispatches is depth. Having said that we do have a website and it will grow.”

Knight hopes that Rethink-Dispatches.com will be used in addition to rather than instead of dispatches.

“I am currently reading a 700 page book on the Korean War by David Halberstam and I would never do that on the web,” says Knight, “but next to me is my laptop and when I want more background on the players in the book I look them up on the web.”

The fact that dispatches is produced by just four people – three of whom have other jobs – also means that the team has had no choice but to prioritise. “We decided to get the book right first.”

In comparison to other sites

Perhaps as a result of having a small team, the website is rather simple and lacking in interactive features – although effective nonetheless in, as the Mission Statement says, “aiming for the widest possible appeal.”

I asked Knight how he felt the website compared to other magazine websites.

“We have made no attempt to put the whole book online. Most magazines have a staff of over a hundred and come out weekly. We are properly compared with Granta or the Paris Review, both of which have more than one member of staff! I would judge our website in a year.”

It seems Rethink-Dispatches.com is a work in progress and it will be interesting to see how the team chooses to make use of multimedia to enhance their product.

“As time goes by I think our visual presence will be much stronger than our competitors’. I think it will be one of the places to go to for photography on the web, but that takes time and we don’t have too much of that.”

The website contains a lot of requests to “Subscribe to read the rest of this…” So has providing ‘teasers’ of the magazine content online increased sales of dispatches in print? Knight thinks probably not.

“The point is that this is a small operation paying for original content and we have to monetise it somehow. Giving it away on the web would probably not be smart at this stage in our evolution, especially as it is not an appropriate platform on which to read our content.”

Knight’s view on the media landscape

I asked Knight what his perspectives were on how the media landscape is changing.

“I think the media is changing but that is not such a bad thing as long as we have high standards. Delivery methods change all the time, big deal. TV did not, as predicted, kill off the radio and I doubt the web will kill off print or TV either.

“What is important is not the delivery system, it is the content, and when old media understands how to use the web, I think the web content will improve.

“I think MySpace and YouTube are not the media, nor are bloggers. They are the 21st century equivalent of Hyde Park Corner [sic] and youth clubs. The future of news delivery on the web demands better editing and citation so people know where to get reliable well written/filmed material.

“Currently, it’s like walking into a book store the size of a football field looking for a book to read on the beach. It’s confusing.”

Knight feels that publishing dispatches only in print will be sustainable. “We need depth, independence and clarity, not sound bites.”

Perhaps as Rosenblum puts it in the editor’s note, “Subjects as important as the fate of our world deserve the permanence of print.”

PHP Freelancer