This is a draft from a book chapter on data journalism (part 1 looks at finding data; part 2 at interrogating data; part 3 at visualisation, and 4 at visualisation tools). I’d really appreciate any additions or comments you can make – particularly around tips and tools.

UPDATE: It has now been published in The Online Journalism Handbook.

Mashing data

Wikipedia defines a mashup particularly succinctly, as “a web page or application that uses or combines data or functionality from two or many more external sources to create a new service.” Those sources may be online spreadsheets or tables; maps; RSS feeds (which could be anything from Twitter tweets, blog posts or news articles to images, video, audio or search results); or anything else which is structured enough to ‘match’ against another source.

This ‘match’ is typically what makes a mashup. It might be matching a city mentioned in a news article against the same city in a map; or it may be matching the name of an author with that same name in the tags of a photo; or matching the search results for ‘earthquake’ from a number of different sources. The results can be useful to you as a journalist, to the user, or both.

Why make a mashup?

Mashups can be particularly useful in providing live coverage of a particular event or ongoing issue – mashing images from a protest march, for example, against a map. Creating a mashup online is not too dissimilar from how, in broadcast journalism, you might set up cameras at key points around a physical location in anticipation of an event from which you will later ‘pull’ live feeds: in a mashup you are effectively doing exactly the same thing – only in a virtual space rather than a physical one. So, instead of setting up a feed at the corner of an important junction, you might decide to pull a feed from Flickr of any images that are tagged with the words ‘protest’ and ‘anti-fascist’.

Some web developers have built entire sites that are mashups. Twazzup (twazzup.com) for example, will show you a mix of Twitter tweets, images from Flickr, news updates and websites – all based on the search term you enter. And Friendfeed (friendfeed.com) pulls in data that you and your social circle post to a range of social networking sites, and displays them in one place.

Mashups also provide a different way for users to interact with content – either by choosing how to navigate (for instance by using a map), or by inviting them to input something (for instance, a search term, or selecting a point on a slider). The Super Tuesday YouTube/Google Maps mashup, for instance, provided an at-a-glance overview of what election-related videos were being uploaded where across the US.

Finally, mashups offer an opportunity for juxtaposing different datasets to provide fresh, sometimes ongoing, insights. The MySociety/Channel 4 project Mapumental, for example, combines house price data with travel information and data on the ‘scenicness’ of different locations to provide an interactive map of a location which the user can interrogate based on their individual preferences.

Mashup tools

Like so many aspects of online journalism, the ease with which you can create a mashup has increased significantly in recent years. An increase in the number and power of online tools, combined with the increasing ‘mashability’ of websites and data, mean that journalists can now create a basic mashup through the simple procedures of drag-and-drop or copy-and-paste.

A simple RSS mashup, which combines the feeds from a number of different sources into one, for example, can now be created using an RSS aggregator such as xFruits (xfruits.com) or Jumbra (jumbra.com).

Likewise, you can mix two maps together using the website MapTube (maptube.org) which also contains a number of maps for you to play with.

And if you want to mix two sources of data into one visualisation the site DataMasher (datamasher.org) will let you do that – although you’ll have to make do with the US data that the site provides. Google Public Data Explorer (google.com/publicdata) is a similar tool which allows you to play with global data.

But perhaps the most useful tool for news mashups is Yahoo! Pipes (pipes.yahoo.com).

Yahoo! Pipes allows you to choose a source of data – it might be an RSS feed, an online spreadsheet or something that the user will input – and do a variety of things with it. Here are just some of the basic things you might do:

• Add it to other sources
• Combine it with other sources – for instance, matching images to text
• Filter it
• Count it
• Annotate it
• Translate it
• Create a gallery from the results
• Place results on a map

You could write a whole book on how to use Yahoo! Pipes – indeed, people have – so we will not cover the practicalities of using all of those features here. There are also dozens of websites and help files devoted to the site (which you should explore). Below, however, is a short tutorial to introduce you to the website and how it works – this is a good way to understand how basic mashups work, and how easily they can be created.

Mashups and APIs

Although there are a number of easy-to-use mashup creators listed above, really impressive mashups tend to be written by people with knowledge of programming languages, and use APIs. APIs (Application Programming Interface) allow websites to interact with other websites. The launch of the Google Maps API in 2005, for example, has been described as a ‘huge tipping point’ in mashup history (Duvander, 2008) as it allowed web developers to ‘mash’ countless other sources of data with maps. Since then it has become commonplace for new websites, particularly in the social media arena, to launch their own APIs in order to allow web developers to do interesting things with their feeds and data – not just mashups, but applications and services too.

If you want to develop a particularly ambitious mashup it is likely that you will need to teach yourself some programming skills, and familiarise yourself with some APIs (the APIs of Twitter, Google Maps and Flickr are good places to start).

Box-out: Anatomy of a feed

The image below from ReadWriteWeb shows the code behind a simple Twitter update. It includes information about the author, their location, whether the update was a reply to someone else, what time and where it was created, and lots more besides. Each of these values can be used by a mashup in various ways – for example, you might match the author of this tweet with the author of a blog or image; you might match its time against other things being published at that moment; or you might use their location to plot this update on a map.

While the code can be intimidating, you do not need to understand programming in order to be able to do things with it. Of course, it will help if you do…

PHP Freelancer

From 10am UK time today I will be reading Clay Shirky’s new book Here Comes Everybody: The Power of Organizing Without Organizations (Amazon US) – and reviewing it on Twitter as I go.

And I won’t be alone. Joining me will be Antonio Gould, Dave Briggs, Jon Bounds, Paul Inman and Brendadada.

All six twitterers – plus a Tweetscan search for ‘Here Comes Everybody’ – will be aggregated at http://xfruits.com/paulbradshaw/?id=38799 so you can follow them all, or join in yourself.

PHP Freelancer

This year I started my online journalism module with three things: Twitter, Del.icio.us, and RSS readers. I asked students to:

• socially bookmark useful webpages,
• subscribe to useful feeds through their RSS reader,
• use social recommendation and tags to discover new sources
• - and to twitter the whole process.

The results? Frankly, disappointing.

If you think 19- and 20-year-olds are au fait with Twitter, think again. Only one had used it before starting the class. And even afterwards, the journalism students I was teaching hardly hit the ground running.

In fact, in the ten days since my class, around half have not twittered at all.

That’s the bad news. Here’s the good.

Those that have used the technology have demonstrated how well it can work as a distribution and publishing tool.

Step up Natalie Chillington, Stephanie Grant, Alice Fanning, Laura Blood, Tuuli Platner, and Anu Gamangari, who are all using it to post updates about their search for stories as they get to grips with reporting on an area completely new to them.

And Kasper Sorensen, who has been a little more creative in using it to post news updates and factoids.

Aggregation, mashup

But this is just the start.

I have aggregated their feeds – and those of the rest of the class – using the wonderful RSS mashup service xFruits.

The combined feed (see it here) is then fed to an email update (also through xFruits), delivered to my inbox so I get a daily update what my reporters are up to.

And of course, so could the readers. Because, even if Twitter doesn’t become a mass market consumption channel, it remains a very effective publishing and distribution one.

Interestingly, that mashed up feed has already had 27 views, despite no publicity – presumably through people browsing xFruits.

But there’s still a further cultural hurdle here. Despite orders to engage, to immerse, and to network, only one student is following a twitterer outside their class (another is following the BBC twitter feed but that hardly counts).

My feeling is that there is still a clear one-way – and gated – publishing mentality from journalism students. My challenge over the following eight lessons is to demonstrate that, online, journalism is not just writing a webpage or filming a video; it is commenting on a blog, or bookmarking a webpage. That there are no walls in cyberspace, only links; and that journalism lies in every act that you commit online. You just need to make it visible.

PHP Freelancer