Tag Archives: scraping

I’m delivering a 3 day workshop on scraping for journalists in January

Person looking at map

From January 23-25 I’ll be delivering a 3 day workshop on scraping in London at The Centre for Investigative Journalism. You don’t need any knowledge of scraping (automatically collecting information from multiple webpages or documents) or programming to take part.

Scraping has been used to report stories ranging from hard news items like “Half of GP surgeries open for under eight hours a day” to lighter stories in arts and culture such as “Movie Trilogies Get Worse with Each Film. Book Trilogies Get Better“.

By the end of the workshop you will be able to use scraping tools (without programming) and have the basis of the skills needed to write your own, more advanced and powerful, scrapers. You will also be able to communicate with programmers on relevant projects and think about editorial ideas for using scrapers.

A full outline of the course can be found on the Centre for Investigative Journalism website, where bookings can also be made, including discounts for freelancers and students.

Advertisements

Here’s the thinking behind my new MA in Data Journalism

A few weeks ago I announced that I was launching a new MA in Data Journalism, and promised that I would write more about the thinking behind it. Here, then, are some of the key ideas underpinning the new course — from coding and storytelling to security and relationships with industry — and how they have informed its development. Continue reading

How the BBC England data unit scraped airport noise complaints

heathrow-noise-story

This news story used scraping to gather data on noise complaints

BBC England Data Unit’s Daniel Wainwright tried to explain basic web scraping at this year’s Data Journalism Conference but technical problems got in the way. This is what should have happened:

I’d wondered for a while why no-one who had talked about scraping at conferences had actually demonstrated the procedure. It seemed to me to be one of the most sought-after skills for any investigative journalist.

Then I tried to do so myself in an impromptu session at the first Data Journalism Conference in Birmingham (#DJUK16) and found out why: it’s not as easy as it’s supposed to look.

To anyone new to data journalism, a scraper is as close to magic as you get with a spreadsheet and no wand. Continue reading

ScraperWiki has rediscovered its old free scraping tool – and is now calling it QuickCode

A screenshot from before the 2013 relaunch of Scraperwiki

A screenshot from before the 2013 relaunch of Scraperwiki

7 years ago ScraperWiki launched with a plan to make scraping accessible to a wider public. It did this by creating an online space where people could easily write and run scrapers; and by making it possible to read and adapt scrapers written by other users (the ‘wiki’ part).

I loved it. The platform inspired me to learn Python, write Scraping for Journalists, and has been part of my journalism workflow since. Continue reading

How one Mexican data team uncovered the story of 4,000 missing women

4534

by Maria Crosas Batista

Mexican newspaper El Universal has put a face to the 4,534 women who have gone missing in Mexico City and the State of Mexico over the last decade: Ausencias Ignoradas (Ignored Absences) aims to put pressure on the government and eradicate this situation.

Daniela Guazo, from the data journalism team, explains how they gathered the data and presented the information not as numbers but as close people: Continue reading

Create your own Instagram/Facebook/Twitter API with Google Drive and IFTTT

Skateboarding images

My Birmingham City University colleague Nick Moreton has a neat little hack for connecting a JavaScript app to social media accounts by combining the automation tool IFTTT, and Google Drive. As he explains:

“Most of the big web apps provide their API in JSON format (Facebook, Twitter, Instagram) however, as you may know if you’ve ever tried to use these, they often require an OAuth login in order to access the API.”

Continue reading

Over 1000 journalists are now exploring scraping techniques. Incredible.

Scraping for Journalists book coverLast week the number of people who have bought my ebook Scraping for Journalists passed the 1,000 mark. That is, to me, incredible. A thousand journalists interested enough in scraping to buy a book? What happened?

When I first began writing the book I imagined there might be perhaps 100 people in the world who would be interested in buying it. It was such a niche subject I didn’t even consider pitching it to my normal publishers.

Now it’s so mainstream that the 1000th ‘book’ was actually 12: purchased by a university which wanted multiple copies for its students to borrow – one of a number of such institutions to approach me to do so.  Continue reading