Category Archives: data journalism

How-to: Scraping ugly HTML using ‘regular expressions’ in an OutWit Hub scraper

11 Replies
Regular Expressions cartoon on xkcd

Regular Expressions cartoon from xkcd

The following is the first part of an extract from Chapter 10 of Scraping for Journalists. It introduces a particularly useful tool in scraping – regex – which is designed to look for ‘regular expressions’ such as specific words, prefixes or particular types of code. I hope you find it useful. 

This tutorial will show you how to scrape a particularly badly formatted piece of data. In this case, the UK Labour Party’s publication of meetings and dinners with donors and trade union general secretaries.

To do this, you’ll need to install the free scraping tool OutWit Hub. Regex can be used in other tools and programming as well, but this tool is a good way to learn it without knowing any other programming. Continue reading →

Data alone isn’t enough – Tim Davies on “complexity and complementarity”

Leave a reply

If people aren’t using data it isn’t just a problem for web developers – it’s a problem for journalists too. If not enough people are looking at information on crime, politics, health, education, or welfare then it makes our work harder.

On that subject, Tim Davies writes about the challenges of ‘getting data used’ and the inclination to focus on data-centric solutions. “Data quality, poor meta-data, inaccessible language, and the difficulty of finding wheat amongst the chaff of data were all diagnosed [at one hack day] as part of the problem,” he reports. “Yet these diagnosis and solutions are still based on linear thinking: when a dataset is truly accessible, then it will be used, and economic benefits will flow. Continue reading →

Data visualisation training

Leave a reply

If you’re interested in data visualisation I’m delivering a training course on November 7 with the excellent Caroline Beavon. Here’s what we’re covering:

  • Pick the right chart for your story – against a deadline
  • Mapping tricks and techniques: using Fusion Tables and other tools to map Olympic torchbearers
  • Picking the right data to visualise
  • Visualisation tips for free chart tools
  • Avoiding common visualisation mistakes
  • Create an infographic with Tableau and Illustrator
  • Making data interactive

More details here. Places can be booked here.

Data Journalism Handbook now in Russian

3 Replies

The Data Journalism Handbook, a free ebook with contributions from dozens of data journalism folk (including me), is now available in Russian.

The translation was “produced and published by the Russian International News & Information Agency (RIA Novosti) with support from the European Journalism Centre”, according to an EJC press release.

Q&A: 5 questions about the pros and cons of data journalism (Cross-post)

1 Reply

The following Q&A is cross-posted from a post on the Media And Digital Enterprise project of the School of Journalism, Media and Communication at the University of Central Lancashire.

Why do journalists need to learn data skills?

For two key reasons: firstly because information is more widely available, and data skills are one of the few remaining ways for journalists to establish their value in that environment.

And secondly, because data is becoming a very important source of both news and the business case for media organisations. Continue reading →

A reading list for studying online journalism

3 Replies

As a new semester nears, I thought I would anticipate the ‘What should I read?’ enquiries by sharing an aggregated reading list from the classes I teach at both Birmingham City University and City University London. Here are 10 key topics with varying numbers of books for each – I’d very much welcome other suggestions:

  1. Working in networks: Yochai Benkler, The Wealth of Networks; Richard Millington, The Proven Path (PDF)
  2. Content strategy: John Battelle, The Search; Bill Tancer, Click; David Kirkpatrick, The Facebook Effect
  3. Platforms: Mark Luckie: The Digital Journalist’s Handbook
  4. Live and mobile journalism: Mark Briggs, Journalism Next; Dan Gillmor, Mediactive
  5. Multimedia: Janet Kolodzy, Convergence Journalism and Practicing Convergence Journalism; Atton & Hamilton, Alternative Journalism; Wilma de Jong, Creative Documentary
  6. UGC, social media and community management: Axel Bruns, Gatewatching; Andrew Lih, Wikipedia Revolution; Jeff Jarvis, What Would Google Do?
  7. Data journalism: Bradshaw and Rohumaa, The Online Journalism Handbook; Andrew Dilnot, The Tiger That Isn’t; Darrell Huff, How to Lie With Statistics; Dona Wong, The Wall Street Guide to Information Graphics; Nathan Yau, Visualize This; Paul Bradshaw, Scraping for Journalists
  8. Law, ethics and online journalism: Friend and Singer, Online Journalism Ethics; Lawrence Lessig, Code; O’Hara and Shadbolt, Spy in the Coffee Machine; Curran, Fenton & Freedman, Misunderstanding the Internet
  9. Experimentation: Clay Shirky, Here Comes Everybody (ch10: Failure for Free); Michalko, Thinkertoys chapter 9; Ian Bogost, Newsgames; Matt Mason, The Pirate’s Dilemma (ch5: Boundaries)
  10. Enterprise: Ken Doctor, Newsonomics; Simon Waldman, Creative Disruption; David Weinberger, Everything is Miscellaneous

You might also find previous posts useful:

Data journalism training – places available

Leave a reply

If you want to learn some basic or intermediate data journalism skills I’m running two single-day courses next week, with places still available.

The first is Introduction to data journalism: taming the numbers on Tuesday September 11.

The second is Intermediate data journalism: take data to the next level on Thursday September 13.

They’re being run with Journalism.co.uk and you can book places on either course through their site. If you book on both days you save £50.

How to teach a journalist programming

8 Replies

Cross-posted from Data Driven Journalism.

Earlier this year I set out to tackle a problem that was bothering me: journalists who had started to learn programming were giving up.

They were hitting a wall. In trying to learn the more advanced programming techniques – particularly those involved in scraping – they seemed to fall into one of two camps:

  • People who learned programming, but were taking far too long to apply it, and so losing momentum – the generalists
  • People who learned how to write one scraper, but could not extend it to others, and so becoming frustrated – the specialists

Continue reading →

Has the increase in data changed your newsroom?

Leave a reply

I’m currently researching if newsrooms have been changed by the increase in availability of data – from FOI and data.gov sites to open data and APIs. Specifically I’m interesting in the watchdog role of journalism, but any other uses are relevant too.

If you work in this area I’d really appreciate it if you can complete the survey below – and share it with others you know can contribute. Here it is.