This is a draft from a book chapter on data journalism (part 1 looks at finding data; part 2 at interrogating data; part 3 at visualisation, and 4 at visualisation tools). I’d really appreciate any additions or comments you can make – particularly around tips and tools.

UPDATE: It has now been published in The Online Journalism Handbook.

Mashing data

Wikipedia defines a mashup particularly succinctly, as “a web page or application that uses or combines data or functionality from two or many more external sources to create a new service.” Those sources may be online spreadsheets or tables; maps; RSS feeds (which could be anything from Twitter tweets, blog posts or news articles to images, video, audio or search results); or anything else which is structured enough to ‘match’ against another source.

This ‘match’ is typically what makes a mashup. It might be matching a city mentioned in a news article against the same city in a map; or it may be matching the name of an author with that same name in the tags of a photo; or matching the search results for ‘earthquake’ from a number of different sources. The results can be useful to you as a journalist, to the user, or both.

Why make a mashup?

Mashups can be particularly useful in providing live coverage of a particular event or ongoing issue – mashing images from a protest march, for example, against a map. Creating a mashup online is not too dissimilar from how, in broadcast journalism, you might set up cameras at key points around a physical location in anticipation of an event from which you will later ‘pull’ live feeds: in a mashup you are effectively doing exactly the same thing – only in a virtual space rather than a physical one. So, instead of setting up a feed at the corner of an important junction, you might decide to pull a feed from Flickr of any images that are tagged with the words ‘protest’ and ‘anti-fascist’.

Some web developers have built entire sites that are mashups. Twazzup (twazzup.com) for example, will show you a mix of Twitter tweets, images from Flickr, news updates and websites – all based on the search term you enter. And Friendfeed (friendfeed.com) pulls in data that you and your social circle post to a range of social networking sites, and displays them in one place.

Mashups also provide a different way for users to interact with content – either by choosing how to navigate (for instance by using a map), or by inviting them to input something (for instance, a search term, or selecting a point on a slider). The Super Tuesday YouTube/Google Maps mashup, for instance, provided an at-a-glance overview of what election-related videos were being uploaded where across the US.

Finally, mashups offer an opportunity for juxtaposing different datasets to provide fresh, sometimes ongoing, insights. The MySociety/Channel 4 project Mapumental, for example, combines house price data with travel information and data on the ‘scenicness’ of different locations to provide an interactive map of a location which the user can interrogate based on their individual preferences.

Mashup tools

Like so many aspects of online journalism, the ease with which you can create a mashup has increased significantly in recent years. An increase in the number and power of online tools, combined with the increasing ‘mashability’ of websites and data, mean that journalists can now create a basic mashup through the simple procedures of drag-and-drop or copy-and-paste.

A simple RSS mashup, which combines the feeds from a number of different sources into one, for example, can now be created using an RSS aggregator such as xFruits (xfruits.com) or Jumbra (jumbra.com).

Likewise, you can mix two maps together using the website MapTube (maptube.org) which also contains a number of maps for you to play with.

And if you want to mix two sources of data into one visualisation the site DataMasher (datamasher.org) will let you do that – although you’ll have to make do with the US data that the site provides. Google Public Data Explorer (google.com/publicdata) is a similar tool which allows you to play with global data.

But perhaps the most useful tool for news mashups is Yahoo! Pipes (pipes.yahoo.com).

Yahoo! Pipes allows you to choose a source of data – it might be an RSS feed, an online spreadsheet or something that the user will input – and do a variety of things with it. Here are just some of the basic things you might do:

• Add it to other sources
• Combine it with other sources – for instance, matching images to text
• Filter it
• Count it
• Annotate it
• Translate it
• Create a gallery from the results
• Place results on a map

You could write a whole book on how to use Yahoo! Pipes – indeed, people have – so we will not cover the practicalities of using all of those features here. There are also dozens of websites and help files devoted to the site (which you should explore). Below, however, is a short tutorial to introduce you to the website and how it works – this is a good way to understand how basic mashups work, and how easily they can be created.

Mashups and APIs

Although there are a number of easy-to-use mashup creators listed above, really impressive mashups tend to be written by people with knowledge of programming languages, and use APIs. APIs (Application Programming Interface) allow websites to interact with other websites. The launch of the Google Maps API in 2005, for example, has been described as a ‘huge tipping point’ in mashup history (Duvander, 2008) as it allowed web developers to ‘mash’ countless other sources of data with maps. Since then it has become commonplace for new websites, particularly in the social media arena, to launch their own APIs in order to allow web developers to do interesting things with their feeds and data – not just mashups, but applications and services too.

If you want to develop a particularly ambitious mashup it is likely that you will need to teach yourself some programming skills, and familiarise yourself with some APIs (the APIs of Twitter, Google Maps and Flickr are good places to start).

Box-out: Anatomy of a feed

The image below from ReadWriteWeb shows the code behind a simple Twitter update. It includes information about the author, their location, whether the update was a reply to someone else, what time and where it was created, and lots more besides. Each of these values can be used by a mashup in various ways – for example, you might match the author of this tweet with the author of a blog or image; you might match its time against other things being published at that moment; or you might use their location to plot this update on a map.

While the code can be intimidating, you do not need to understand programming in order to be able to do things with it. Of course, it will help if you do…

I’ve never been fond of the search engine on Twitter, not the one on search.twitter.com anyway. I have found the ones build on it’s API much friendlier and more intuitive, such as Twitterfall and the integration in Tweetdeck. But none of them work for finding old tweets. Google is not much help either, unless you know how to create your own search engine.

Friendfeed aggregates and stores all the activity that is fed into the system. Most FF users bring in their Twitter feed, in effect storing all their tweets. It works a little bit like Google Reader, once it’s there, it will always be there, even if the original is deleted.

The advanced search features of Friendfeed makes it a pretty good twitter search alternative. It even supports real-time, so you can make your own twitter news monitors.

Searching old tweets

Twitter only keeps tweets in it’s search database for a few weeks, after that they disappear. They’re still available on the web, just not searchable from Twitter (or any thrid party app). That’s great if you just want the real-time view, but not practical when looking for an exact tweet a few weeks old.

I needed to find this tweet from Paul Bradshaw for a presentation, but it was long gone from the internal database. I knew that Paul is using Friendfeed, not actively but he’s sharing his tweets there, so I did this search (Bingo, no. 2 from the top). Here’s the equivalent twitter search which is no help.

From any Friendfeed page, you simple select advanced search at the top, fill in the blanks and you got it. Here’s how mine was filled in.

Real-time “noise” filtering

Some hashtags can get ugly, real quick. There’s no way to filter out the high quality tweets either. People can favorite tweets, but you can’t search them, so no way to filter. When news breaks, there will be a few quality tweets in the beginning, people will retweet the most important. But people quickly starts talking about the event which brings no real value to the table, other than twitter-chatter. Eyewitness accounts and other useful information is lost in the stream because people have no way of marking important tweets for later retrieval (search).

On Friendfeed, people have the option of liking entries, and the advanced search let’s you filter items based on likes or comments. You can now rely on the FF community to mark the important stuff and cut through all (some of) the noise.

Here’s an example of a search that filters all tweets with the #iranelection hashtag, and shows only tweets that has 3 or more likes.

Other uses

There are many other ways to search Friendfeed and you can filter for all services like facebook, blogs etc. You can save searches and use them as filters. I have several live searches saved in Friendfeed. Here’s an example of a search that gives me all twitter entries from my friends with one or more comments.

Friendfeed suffers from the fact that it’s userbase is not as big as Twitter’s, but the ‘real’ real-time search more than makes up for that in my opinion. What I mean by real, is that items are published automatically from all services. If you bring in your Flickr, comments and blog activity to Friendfeed, they will publish automatically. Twitter doesn’t do that, you have to actively share the link after you have uploaded to Flickr, made a comment somewhere or updated your blog.

It’s often said that Twitter’s big advantage over Google is its ability to allow you to conduct ‘real time search’ – if an event is happening right now, you don’t search Google, you search Twitter.

But today Google has announced a series of features that, while still not offering real time search, take it just that bit closer. For me it is the most significant change to Google’s core service in years. 

Here’s the video:

This week, while talking to my students about the ability to search by date in Google, the computer assisted reporting blogger Murray Dick mentioned how unreliable the feature was, so I wouldn’t get too excited. 

What is new, however, is the ‘recent search’ facility, which brings up results from the past hour or two.

Perhaps most significantly, the intuitive nature of the controls makes it much easier to find relevant information – and has obvious uses if you are a journalist only interested in recent reports. I tried it with an event I attended today and the results were clearly better than a generic search.

The ‘Timeline’ feature is particularly useful – you can now drill down to a specific period, allowing you to do the opposite – rule out recent reports – if, for example, you’re looking for background on someone currently in the news.

Also useful is the horrendously multicoloured ‘Wonder Wheel’ which will generate related terms that you can browse through.

So, a search for ‘Paul Bradshaw’ brings up online journalism, journalism blog, Press Gazette and Birmingham City University – an obvious way to get an overview of the areas and companies a person is connected with, and drill down to information about those. 

‘More text’ will give expanded descriptions of results; you can choose to include images from the pages; and it is easier to switch to searches of video, forums and reviews. 

Also worth noting is that Google appear to now be supporting microformats: this means you can filter reviews by sentiment, and more easily disambiguate people in searches. 

It also means you should be looking at using microformats for your own content, of course… 

If you can see any other applications of Search Options, or find any features I’ve not spotted, let me know.

UPDATE: TechCrunch compare Google’s performance against Twitter Search, FriendFeed, OneRiot, Tweetmeme and Scoopler (which I love) in this real-time search-off.

UPDATE 2: ZDNet spots the addition of public comments on search results. Very interesting.

To mark 1000 posts on this blog, I thought I’d reflect on what I’ve learned since post #1.

UPDATE: Now available in German, Spanish, Hebrew, and Portuguese.

UPDATE 2: I’ll be posting further ’1000 things’ via Twitter – you can find them with this search or this RSS feed.

1. Blogging is not ‘writing a blog’. Blogging is linking and commenting. Any writing is a bonus.
2. Regular posting is important…
3. But quality posting is even more important. Spending a week or more on a single post can be one of the most important things you ever do.
4. First knowledge, then analysis, then ideas.
5. A picture is worth a thousand words. More importantly, a picture is worth a thousand words in two hundred countries. The fact that readers don’t need to speak English to understand what you’re communicating can make a word-free post – or at least one with a good image – your most successful one.
6. For similar reasons, video works. It may not be search engine-friendly, but if people can embed it the word is more likely to spread.
7. When video meets conversation, good stuff can happen.
8. Everyone looks ugly on video. Get over it.
9. Online video is not online TV
10. Podcasts work better when there’s more than one of you
11. It takes time. Sometimes years. Persistence counts.
12. Being early matters
13. A big idea travels far
14. Pingback/trackback is a wonderful thing, a form of distribution news websites are still struggling to match. What can be more interesting than someone who is interested in you?
15. Cliques and old boys’ networks exist in the blogosphere too
16. We are too fucking Anglo-American
17. Language is a massive barrier (but having multilingual friends helps – see updates at top of post)
    
18. BASIC principles matter
19. Social bookmarking makes researching a post much easier
20. The best reason to blog is not to show everyone else what you know, but to find out what everyone else knows
21. RSS is one of the most undervalued technologies in the world. Once you understand what to do with it, you can bring the world to your desktop, your mobile, and your blog, and vice versa.
22. A blog doesn’t open doors for you, it just gives you the idea to try knocking.
23. When people Google you, it saves a lot of time explaining things.
24. Blogs are just one part of a social media ecology. Half the stuff that used to go on this blog now goes on Twitter; more goes on Delicious; and some on Flickr and on Seesmic.
25. Don’t get me started on FriendFeed, Plurk, Jaiku, etc.
26. Humour is effective, but not everyone will get it
27. I seem to like linking on verbs
28. Streaming live video from your mobile is a pretty amazing thing when you think about it
29. Streaming live video from your mobile uses up your battery quickly
30. Web browsing on your mobile also uses up your battery quickly
31. If you’re moblogging an event, bring a power lead, an extension lead – and a spare phone
32. The N95 kicks iPhone‘s ass
33. (But I’m prepared to be persuaded otherwise)
34. WordPress plugins are addictive
35. Firefox extensions are addictive
36. Signing up for beta web services is addictive
37. I don’t really care about Twitterspam
38. A simple, fun idea can be around the world in minutes
39. If you want to campaign against something, you already have the technology
40. If you want a service, create it yourself
41. Google is the biggest popularity contest in the world
42. When you realise you don’t have a readership – you have a community – then you also realise you can mobilise, and get things done.
43. Technology is easy; community is hard
44. Meeting in person is important: I read blogs by people I’ve met much more often than those I haven’t
45. Geography still matters
46. Birmingham has a lot of bloggers
47. Liveblogging and Twitter-blogging are not the same thing
48. Privacy is a fluid concept: just because it’s in the public domain doesn’t mean it’s not private
49. WordPress.com is better than Blogger
50. WordPress.org is better than WordPress.com (see Thing 34)
51. Content is not king.
52. Conversation is king.
53. Conversation is the kingdom.
54. We shouldn’t try to be the media
55. If someone is sending you a press release about something, you shouldn’t blog about it
56. As a journalist, blogging is a good way to rediscover the joy of journalism
57. Blogging is also a great way to rediscover how great having a good editor can be
58. Do what you do best and link to the rest
59. Blogs aren’t worth dying for. That’s what family is for.
60. Setting yourself a maximum number of posts per day is a good idea
61. Setting yourself a set time to look at your RSS subs every day is also a good idea
62. If you rely on third party services, prepare for the rug to be pulled from under your feet
63. If you publish the comments widget high up on your blog, more people comment
64. A blog without a comments facility is broken
65. A site that has comments, but edits or buries them, is not just broken, it’s malevolent.
66. Leave posts open ended if you want people to comment
67. Leave a post at the top of your site for more than a day if you want people to comment
68. Being transparent about your sources is not only good journalism, it’s good distribution.
69. The search engine optimisation industry is the new snake oil. I can tell you all you need to know about SEO in five minutes
70. Although it might take me another five hours to answer the resulting questions
71. If you expect to make lots of money from blogging, you are either naive, stupid, or Robert Scoble.
72. If you expect to make lots of money from blogging, don’t expect to make it through advertising
73. Being read by a few, key, people can be worth more, professionally, than having lots of visitors
74. Being frequently linked to can be worth more, commercially, than having lots of visitors
75. Beware advertisers bearing text-based gifts, or generous offerings of ‘free’ articles. Understand linkspam
76. Be aware that you have an ego
77. Be aware that everyone else has an ego
78. Unconferences are great
79. There’s only so much talking you can do. Sometimes you have to do something.
80. There should be more money available to do something
81. Ideas aren’t a problem. Knowing which ones to pursue is
82. Only 10% of Americans read blogs
83. But 26% of Americans write blogs
84. How does that work?
85. Blogs are far more ethnically representative than mainstream media
86. People may not trust the print and broadcast media, but they trust online news even less
87. The 1-9-90 rule
88. Rushing off a blog entry just before bed is a bad idea
89. Rushing off a blog entry hours before your wife goes into labour is not a good idea either
90. Some news travels faster than an aftershock
91. People don’t need managers to organise them – just connections
92. When I can record a video comment straight from my mobile phone, I’ll be a happy man
93. Don’t underestimate the power of corporatisation
94. Don’t underestimate the power of big corporations
95. Don’t underestimate the power of governments
96. If, after all this, we have to go back to living in caves and eating rats, it’ll be a real shame
97. Lists have become the biggest cliche in blogging and the most shameless tactic for getting to the top of delicious/digg/reddit.
98. But people still read them.
99. Have you bookmarked this yet, by the way?

1000. I can’t count.

↑ Grab this Headline Animator

I’ve written two reviews over at JournalismEnterprise.com:

CoverItLive is “a Twitter-meets-chatroom-meets-poll that you can embed on your site.”

and the much-hyped FriendFeed is “Facebook, but uglier and more flexible.”