The Online Journalism Handbook, written with Liisa Rohumaa, has now been published. You can get it here.

I’ve been blogging throughout the process of writing the book – particularly the chapters on data journalism, blogging and UGC – and you can still find those blog posts under the tag ‘Online Journalism Book‘.

Other chapters cover interactivity, audio slideshows and podcasting, video, law, some of the history that helps in understanding online journalism, and writing for the web (including SEO and SMO).

Meanwhile, I’ve created a blog, Facebook page and Twitter account (@OJhandbook) to provide updates, corrections and additions to the book.

If you spot anything in the book that needs updating or correcting, let me know. Likewise, let me know what you think of the book and anything you’d like to see added in future.

In researching my book chapter (UPDATE: now published) I asked a group of journalists who worked with data what led them to do so. Here are their answers:

Jonathon Richards, The Times:

The flood of information online presents an amazing opportunity for journalists, but also a challenge: how on earth does one keep up with; make sense of it? You could go about it in the traditional way, fossicking in individual sites, but much of the journalistic value in this outpouring, it seems, comes in aggregation: in processing large amounts of data, distilling them, and exploring them for patterns. To do that – unless you’re superhuman, or have a small army of volunteers – you need the help of a computer.

I ‘got into’ data journalism because I find this mix exciting. It appeals to the traditional journalistic instinct, but also calls for a new skill which, once harnessed, dramatically expands the realm of ‘stories I could possibly investigate…’

Mary Hamilton, Eastern Daily Press:

I started coding out of necessity, not out of desire. In my day-to-day work for local newspapers I came across stories that couldn’t be told any other way. Excel spreadsheets full of data that I knew was relevant to readers if I could break it down or aggregate it up. Lists of locations that meant nothing on the page without a map. Timelines of events and stacks of documents. The logical response for me was to try to develop the skills to parse data to get to the stories it can tell, and to present it in interactive, interesting and – crucially – relevant ways. I see data journalism as an important skill in my storytelling toolkit – not the only option, but an increasingly important way to open up information to readers and users.

Charles Arthur, The Guardian:

When I was really young, I read a book about computers which made the point – rather effectively – that if you found yourself doing the same process again and again, you should hand it over to a computer. That became a rule for me: never do some task more than once if you can possibly get a computer to do it.

Obviously, to implement that you have to do a bit of programming. It turns out all programming languages are much the same – they vary in their grammar, but they’re all about making the computer do stuff. And it’s often the same stuff (at least in my ambit) – fetch a web page, mash up two sets of data, filter out some rubbish and find the information you want.

I got into data journalism because I also did statistics – and that taught me that people are notoriously bad at understanding data. Visualisation and simplification and exposition are key to helping people understand.

So data journalism is a compound of all those things: determination to make the computer do the slog, confidence that I can program it to, and the desire to tell the story that the data is holding and hiding.

I don’t think there was any particular point where I suddenly said “ooh, this is data journalism” – it’s more that the process of thinking “oh, big dataset, stuff it into an ad-hoc MySQL database, left join against that other database I’ve got, see what comes out” goes from being a huge experiment to your natural reaction.

It’s not just data though – I use programming to slough off the repetitive tasks of the day, such as collecting links, or resizing pictures, or getting the picture URL and photographer and licence from a Flickr page and stuffing it into a blogpost.

Data journalism is actually only half the story. The other half is that journalists should be **actively unwilling** to do repetitive tasks if it’s machine-like (say, removing line breaks from a piece of copy, or changing a link format).

Time spent doing those sorts of tasks is time lost to journalism and given up to being a machine. Let the damn machines do it. Humans have better things to do.

Stijn Debrouwere, Belgian information designer:

I used to love reading the daily newspaper, but lately I can’t seem to be bothered anymore. I’m part of that generation of people news execs fear so much: those that simply don’t care about what newspapers and news magazines have to offer. I enjoy being an information designer because it gives me a chance to help reinvent the way we engage and inform communities through news and analysis, both offline and online. Technology doesn’t solve everything, but it sure can help. My professional goal is simply this: make myself love news and newspapers again, and thereby hopefully getting others to love it too.

As part of the research into a forthcoming book on online journalism (UPDATE: now published), I interviewed Ewen Rankin of independent podcast Bagel Tech News. Here are his responses in full:

The background

My background is as a commercial photographer. I started life in graphic design and quickly moved to shooting photographs for the agency at which I worked. It was kind of a lucky transition as I wasn’t much cop as a graphic artist. I took fairly low level stuff to start with (picture business cards were all the rage in the 80s) and then moved to more commercial work shooting the advertising shots for Pretty Polly and Golden Lady tights in about 1988.

I start broadcasting in July 2008 and after two weeks Amber Macarthur made us Podcast of the Week on the Net@Night show with Leo Laporte. Listenership rose and we began to grow.

The Daily News show was published… daily until November 2008 and then I started publishing the BOG Show with Marc Silk, and was opened by Andy Ihnatko on 30th November 2008. I removed Marc from the show in Christmas 2009 and installed a ‘Skype Wall’ in January 2010 to run a more panel based show. More shows have been added in the intervening period and the network now has 7 active shows

• Bagel Tech News – 70k Dloads PCM
• Bagel Tech BIG – 3k Dloads/Week
• Bagel Profits – No Show since May due to Athos Work committments. Generally around 250-500 per episode
• Bagel Tech Foto – New podcast on Photography – 5 episodes produced 250 Dloads Per episode
• Bagel Tech Media – Formerly Sonic Beyond Podcast – 500 Dloads per episode
• Bagel Tech Rage – Formerly Tech Rage News – 250 Dloads per Episode.
• Bagel Tech Mac will begin airing in September
• Bagel Tech Law will begin in 2011

Apart from the Daily Show, all podcasts are produced weekly.

Bagel Tech Media Group will also add non tech related shows in 2011.

Preparing the show

The Daily Show is prepared each morning at 5.30am with a trawl through around 300 stories gathered using the Firefox Plugin ‘Brief’ and then saved and Synced as bookmarks using ‘X Marks’ After that the chosen stories are ordered and then the podcast is recorded. This is generally about 10 minutes of audio including fluffs and rereads and edits to between 5 and 6.5 mins.

Then the pictures are added to the M4a Version and then the website is updated.

Stories are selected based on whether I believe that the story is either something that the listenership would Want to Know, but I also include stories which I think they SHOULD know or could know. And every podcast has an ‘And Finally’ to sign off with a snigger.

The Weekly shows are more relaxed and there is minimal prep for these.

Tricks of the Trade… hmm. I guess I have just got more efficient at reviewing stories and creating the podcast and website. I have learnt more tools which can save me time and I am already set up to work from locations across the country. I am truly a mobile office and studio and it is rare for me to miss an episode of the Daily Show. The process is time consuming in prep more than delivery. Some mornings are hard to get motivated, others come easier.

Advice

Broadcast with enthusiasm and passion for the subject. Make sure that podcasting is your hobby first and try to make money second. If you show your financial hand too early then you will alienate listeners.

Concentrate on community. Let people feel part of the ‘X Show’ community rather than isolated listeners. Open a chatroom and live feed while you record for the interaction which ensures this develops.

Lastly, broadcast to more than 1000 people every day. It doesn’t matter if there arent 1000 people on the other side of the microphone… always broadcast like there are or it will show through.

Here’s another snippet from my data journalism book chapter (now published). As part of my research David McCandless, author of the very lovely book and website Information is Beautiful gave  his 5 tips for visualising data:

1. Double source data wherever possible – even the UN and WorldBank can make mistakes
2. Take information out – there’s a long tradition among statistical journalists of showing everything. All data points. The whole range. Every column and row. But stories are about clear threads with extraneous information fuzzed out. And journalism is about telling stories. You can only truly do that when you mask out the irrelevant or the minor data. The same applies to design which is about reducing something to its functional essence.
3. Avoid standard abstract units – tons of carbon, billions of dollars – these kinds of units are over-used and impossible to imagine or relate to. Try to rework or process units down to ‘everyday’ measures. Try to give meaningful context for huge figures whenever possible.
4. Self-sufficency – all graphs, charts and infographics should be self-sufficient. That is, you shouldn’t require any other information to understand them. They’re like interfaces. So each should have a clear title, legend, source, labels etc. And credit yourself. I’ve seen too many great visuals with no credit or name at the bottom.
5. Show your workings - transparency seems like a new front for journalists. Google Docs makes it incredibly easy to share your data and thought processes with readers. Who can then participate.

This is a draft from a book chapter on data journalism (part 1 looks at finding data; part 2 at interrogating data; part 3 at visualisation, and 4 at visualisation tools). I’d really appreciate any additions or comments you can make – particularly around tips and tools.

UPDATE: It has now been published in The Online Journalism Handbook.

Mashing data

Wikipedia defines a mashup particularly succinctly, as “a web page or application that uses or combines data or functionality from two or many more external sources to create a new service.” Those sources may be online spreadsheets or tables; maps; RSS feeds (which could be anything from Twitter tweets, blog posts or news articles to images, video, audio or search results); or anything else which is structured enough to ‘match’ against another source.

This ‘match’ is typically what makes a mashup. It might be matching a city mentioned in a news article against the same city in a map; or it may be matching the name of an author with that same name in the tags of a photo; or matching the search results for ‘earthquake’ from a number of different sources. The results can be useful to you as a journalist, to the user, or both.

Why make a mashup?

Mashups can be particularly useful in providing live coverage of a particular event or ongoing issue – mashing images from a protest march, for example, against a map. Creating a mashup online is not too dissimilar from how, in broadcast journalism, you might set up cameras at key points around a physical location in anticipation of an event from which you will later ‘pull’ live feeds: in a mashup you are effectively doing exactly the same thing – only in a virtual space rather than a physical one. So, instead of setting up a feed at the corner of an important junction, you might decide to pull a feed from Flickr of any images that are tagged with the words ‘protest’ and ‘anti-fascist’.

Some web developers have built entire sites that are mashups. Twazzup (twazzup.com) for example, will show you a mix of Twitter tweets, images from Flickr, news updates and websites – all based on the search term you enter. And Friendfeed (friendfeed.com) pulls in data that you and your social circle post to a range of social networking sites, and displays them in one place.

Mashups also provide a different way for users to interact with content – either by choosing how to navigate (for instance by using a map), or by inviting them to input something (for instance, a search term, or selecting a point on a slider). The Super Tuesday YouTube/Google Maps mashup, for instance, provided an at-a-glance overview of what election-related videos were being uploaded where across the US.

Finally, mashups offer an opportunity for juxtaposing different datasets to provide fresh, sometimes ongoing, insights. The MySociety/Channel 4 project Mapumental, for example, combines house price data with travel information and data on the ‘scenicness’ of different locations to provide an interactive map of a location which the user can interrogate based on their individual preferences.

Mashup tools

Like so many aspects of online journalism, the ease with which you can create a mashup has increased significantly in recent years. An increase in the number and power of online tools, combined with the increasing ‘mashability’ of websites and data, mean that journalists can now create a basic mashup through the simple procedures of drag-and-drop or copy-and-paste.

A simple RSS mashup, which combines the feeds from a number of different sources into one, for example, can now be created using an RSS aggregator such as xFruits (xfruits.com) or Jumbra (jumbra.com).

Likewise, you can mix two maps together using the website MapTube (maptube.org) which also contains a number of maps for you to play with.

And if you want to mix two sources of data into one visualisation the site DataMasher (datamasher.org) will let you do that – although you’ll have to make do with the US data that the site provides. Google Public Data Explorer (google.com/publicdata) is a similar tool which allows you to play with global data.

But perhaps the most useful tool for news mashups is Yahoo! Pipes (pipes.yahoo.com).

Yahoo! Pipes allows you to choose a source of data – it might be an RSS feed, an online spreadsheet or something that the user will input – and do a variety of things with it. Here are just some of the basic things you might do:

• Add it to other sources
• Combine it with other sources – for instance, matching images to text
• Filter it
• Count it
• Annotate it
• Translate it
• Create a gallery from the results
• Place results on a map

You could write a whole book on how to use Yahoo! Pipes – indeed, people have – so we will not cover the practicalities of using all of those features here. There are also dozens of websites and help files devoted to the site (which you should explore). Below, however, is a short tutorial to introduce you to the website and how it works – this is a good way to understand how basic mashups work, and how easily they can be created.

Mashups and APIs

Although there are a number of easy-to-use mashup creators listed above, really impressive mashups tend to be written by people with knowledge of programming languages, and use APIs. APIs (Application Programming Interface) allow websites to interact with other websites. The launch of the Google Maps API in 2005, for example, has been described as a ‘huge tipping point’ in mashup history (Duvander, 2008) as it allowed web developers to ‘mash’ countless other sources of data with maps. Since then it has become commonplace for new websites, particularly in the social media arena, to launch their own APIs in order to allow web developers to do interesting things with their feeds and data – not just mashups, but applications and services too.

If you want to develop a particularly ambitious mashup it is likely that you will need to teach yourself some programming skills, and familiarise yourself with some APIs (the APIs of Twitter, Google Maps and Flickr are good places to start).

Box-out: Anatomy of a feed

The image below from ReadWriteWeb shows the code behind a simple Twitter update. It includes information about the author, their location, whether the update was a reply to someone else, what time and where it was created, and lots more besides. Each of these values can be used by a mashup in various ways – for example, you might match the author of this tweet with the author of a blog or image; you might match its time against other things being published at that moment; or you might use their location to plot this update on a map.

While the code can be intimidating, you do not need to understand programming in order to be able to do things with it. Of course, it will help if you do…

This is a draft from a book chapter on data journalism (here are parts 1; two; and three, which looks the charts side of visualisation). I’d really appreciate any additions or comments you can make – particularly around tips and tools.

UPDATE: It has now been published in The Online Journalism Handbook.

Visualisation tools

So if you want to visualise some data or text, how do you do it? Thankfully there are now dozens of free and cheap pieces of software that you can use to quickly turn your tables into charts, graphs and clouds.

The best-known tool for creating word clouds is Wordle (wordle.net). Simply paste a block of text into the site, or the address of an RSS feed, and the site will generate a word cloud whose fonts and colours you can change to your preferences. Similar tools include Tagxedo (tagxedo.com) and Wordlings (http://wordlin.gs), both of which allow you to put your word cloud into a particular shape.

ManyEyes (manyeyes.alphaworks.ibm.com/manyeyes/) also allows you to create word clouds and tag clouds – as well as word trees and phrase nets that allow you to see common phrases. But it is perhaps most useful in allowing you to easily create scattergrams, bar charts, bubble charts and other forms. The site also contains a raft of existing data that you can play with to get a feel for the site. Similar tools that allow access to other data include Factual (factual.com), Swivel (swivel.com)[see comments], Socrata (socrata.com) and Verifiable.com (verifiable.com). And Google Fusion Tables (tables.googlelabs.com) is particularly useful if you want to collaborate on tables of data, as well as offering visualisation options.

More general visualisation tools include widgenie (widgenie.com), iCharts (icharts.net), ChartTool (onlinecharttool.com) and ChartGo (www.chartgo.com). FusionCharts is a piece of visualisation software with a Google Gadget service that publishers may find useful. You can find instructions on how to use it at www.fusioncharts.com/GG/Docs

If you want more control over your visualisation – or want it to update dynamically when the source information is updated, Google Chart Tools (code.google.com/apis/charttools) is worth exploring. This requires some technical knowledge, but there is a lot of guidance and help on the site to get you started quickly.

Tableau Public is a piece of free software you can download (tableausoftware.com/public) with some powerful visualisation options. You will also find visualisation options on spreadsheet applications such as Excel or the free Google Docs spreadsheet service. These are worth exploring as a way to quickly generate charts from your data on the fly.

Publishing your visualisation

There will come a point when you’ve visualised your data and need to publish it somehow. The simplest way to do this is to take an image (screengrab) of the chart or graph. This can be done with a web-based screencapture tool like Kwout (kwout.com), a free desktop application like Skitch (skitch.com) or Jing (jingproject.com), or by simply using the ‘Print Screen’ button on a PC keyboard (cmd+shift+3 on a Mac) and pasting the screengrab into a graphics package such as Photoshop.

The advantage of using a screengrab is that the image can be easily distributed on social networks, image sharing websites (such as Flickr), and blogs – driving traffic to the page on your site where it is explained.

If you are more technically minded, you can instead choose to embed your chart or graph. Many visualisation tools will give you a piece of code which you can copy and paste into the HTML of an article or blog post in the place you wish to display it (this will not work on most third party blog hosting services, such as WordPress.com). One particular advantage of this approach is that the visualisation can update itself if the source data is updated.

Alternatively, an understanding of Javascript can allow you to build ‘progressively enhanced’ charts which allow users to access the original data or see what happens when it is changed.

Showing your raw data

It is generally a good idea to give users access to your raw data alongside its visualisation. This not only allows them to check it against your visualisation but add insights you may not otherwise gain. It is relatively straightforward to publish a spreadsheet online using Google Docs (see the sidebar on publishing a spreadsheet)

SIDEBAR: How to: publish a spreadsheet online

Google Docs (docs.google.com) is a free website which allows you to create and share documents. You can share them via email, by publishing them as a webpage, or by embedding your document in another webpage, such as a blog post. This is how you share a spreadsheet:

1. Open your spreadsheet in Google Docs. You can upload a spreadsheet into Google Docs if you’ve created it elsewhere – there is a size limit, however, so if you are told the file is too big try removing unnecessary sheets or columns.
2. Look for the ‘Share’ button (currently in the top right corner) and click on it.
3. A drop-down menu should appear. Click on ‘Publish as a web page’
4. A new window should appear asking which sheets you want to publish. Select the sheet you want to publish and click ‘Start publishing’ (you should also make sure ‘Automatically republish when changes are made’ is ticked if you want the public version of the spreadsheet to update with any data you add.)
5. Now the bottom half of that window – ‘Get a link to the published data’ – should become active. In the bottom box should be a web address where you can now see the public version of your spreadsheet. If you want to share that, copy the address and test that it works in a web browser. You can now link to it from any webpage.
6. Alternatively, you can embed your spreadsheet – or part of it – in another webpage. To do this click on the first drop-down menu in this area – it will currently say ‘Web page’ – and change it to ‘HTML to embed in a page’. Now the bottom box on this window should show some HTML that begins with
7. If you want to embed just part of a spreadsheet, in the box that currently says ‘All cells’ type the range of cells you wish to show. For example, typing A1:G10 will select all the cells in your spreadsheet from A1 (the first row of column A) to G10 (the 10th row of column G). Once again, the HTML below will change so that it only displays that section of your spreadsheet.

Once again, I’d welcome any comments on things I may have missed or tips you can add. Part 5, on mashups, is now available here.

This is a draft from a book chapter on data journalism (the first, on gathering data, is here; the section on interrogating data is here). I’d really appreciate any additions or comments you can make – particularly around considerations in visualisation. A further section on visualisation tools, can be found here.

UPDATE: It has now been published in The Online Journalism Handbook.

“At their best, graphics are instruments for reasoning about quantitative information. Often the most effective way to describe, explore, and summarize a set of numbers – even a very large set – is to look at pictures of those numbers.” (Edward Tufte, The Visual Display of Quantitative Information, 2001)

Visualisation is the process of giving a graphic form to information which is often otherwise dry or impenetrable. Classic examples of visualisation include turning a table into a bar chart, or a series of percentage values into a pie chart – but the increasing power of both computer analysis and graphic design software have seen the craft of visualisation develop with increasing sophistication. In larger organisations the data journalist may work with a graphic artist to produce an infographic that visualises their story – but in smaller teams, in the initial stages of a story, or when speed is of the essence they are likely to need to use visualisation tools to give form to their data.

Broadly speaking there are two typical reasons for visualising data: to find a story; or to tell one. Quite often, it is both.

In the parking tickets story above, for example, it was the process of visualisation that tipped off Adrian Short and Guardian journalist Charles Arthur to the story – and led to further enquiries.

In most cases, however, the story will not be as immediately visible. Sometimes the data will need to be visualised in different ways before a story becomes clear. And an understanding of the strengths of different types of visualisation can be particularly useful here.

UPDATE (Dec 7, 2010): Visualisation probably needs to be extended to include humanisation and personalisation. More detail here, and to come.

Types of visualisation

Visualisation can take on a range of forms. The most familiar are those we know from maths and statistics: pie charts, for example, allow you to show how one thing is divided – for example, how a budget is spent, or how a population is distributed. They are thought to be particularly useful when the proportions represented are large (for example, above 25%), but less useful when lower percentages are involved, due to issues with perception and the ability to compare different elements.

More useful in those circumstances are bar charts or histograms. Although these look the same there are subtle differences between them: the bars in bar charts represent categories (such as different cities), whereas bars in histograms represent different values on a continuum (for instance: ages, weights or amounts). You should avoid using 3D or shadow effects in bar charts as these do not add to the information or clarity (histograms do not have gaps between bars). The advantage of both types of chart over pie charts is that users can more easily see the difference between one quantity and another. Bar charts also allow you to show change over time.

Pictograms are like bar charts but use an icon to represent quantity – so a population of 50,000 might be represented by 5 ‘person’ icons. It is not advisable to use pictograms if quantities are close together as the user will find it harder to discern the differences.

Also useful for showing change over time are line graphs. Lines are “suited for showing trend, acceleration or deceleration, and volatility, including sudden peaks or troughs” (Wong, 2010, p51). In addition, a series of lines overlaid upon each other can also quickly show if any variables change at different points or at simultaneous points, suggesting either relationships or shared causes (but by no means proving it – these should be taken as starting points for further investigation. You should also avoid plotting more than four lines in one chart for purposes of clarity).

Line graphs should not be used to show unrelated events. As Seth Godin (2009) puts it: “A graph of IQs of everyone in your kindergarten class should be a series of unrelated points, not a line graph. On the other hand, your weight loss is in fact a continuous function, so each piece of data should be attached.”

Scattergrams are similar to line graphs, showing the distribution of individual elements against two axes, but can be particularly useful in showing up ‘outliers’. Outliers are pieces of data which differ noticeably from the rest. These may be of particular interest journalistically when they show, for example, an MP claiming substantially more (or less) expenses than their peers.

A number of charts can be visualised together in what is sometimes called ‘small multiples‘, allowing the journalist or users to display a number of pie charts, line graphs or other charts alongside each other – allowing comparison, for example, between different populations.

Two increasingly popular forms of visualisation online are treemaps and bubble charts. Unlike other charts which allow you to visualise two aspects of the data (i.e. their place on each axis) bubble charts allow you to visualise three aspects of the data – the third being represented by the size of the bubble itself. A particularly good example of bubble charts in action can be seen in Hans Rosling’s TED talk on debunking third-world myths – a presentation which also demonstrates the potential of other forms of visualisation, and animation, in presenting complex information in an easy-to-understand way.

Finally, Treemaps visualise hierarchical data in a way that could be described as rectangular pie charts-within-pie charts. This is particularly useful for representing different parts of a whole and their relationship to each other, for instance, different budgets within a government.

Perhaps the best-known example of a treemap is Newsmap, created in 2004 by Marcos Weskamp. This visualises the amount of coverage given to stories by news organisations based on a feed from Google News. Weskamp explains it as follows:

“Google News automatically groups news stories with similar content and places them based on algorithmic results into clusters. In Newsmap, the size of each cell is determined by the amount of related articles that exist inside each news cluster that the Google News Aggregator presents. In that way users can quickly identify which news stories have been given the most coverage, viewing the map by region, topic or time. Through that process it still accentuates the importance of a given article.” (Weskamp, 2005)

These are just the most common forms of visualisation, but there are dozens more to explore. The Periodic Table of Visualisation is a particularly useful webpage giving an overview of the various forms.

Considerations in visualisation

Charlie Beckett makes a useful distinction between using visualisation for “rational understanding (I now get the figures) and emotional understanding (I now care about the figures and want to do something).” It is worth deciding which of the two you are aiming for.

When visualising data it is also important to ensure that any comparisons are meaningful, or like-for-like. In one visualisation of how many sales a musician needs to make to earn the minimum wage, for example, a comparison is made between sites selling albums, sites selling individual tracks, and those providing music streams. Clearly this is misleading – and was criticised for being so (Techdirt, 2010).

The Wall Street Journal Guide to Information Graphics (2010) offers a wealth of tips on elements to consider and mistakes to avoid in both visualisation and data research and is well worth reading for more on this area. Here are just a selection:

• “Choose the best data series to illustrate your point, e.g. market share vs. total revenue
• “Filter and simplify the data to deliver the essence of the data to your intended audience
• “Make numerical adjustments to the raw data to enhance your point, e.g. absolute values vs. percentage change
• “Choose the appropriate chart settings, e.g. scale, y-axis increments and baseline
• “If the raw data is insufficient to tell the story, do not add decorative elements. Instead, research additional sources and adjust data to stay on point
• “Data is only as good as its source. Getting data from reputable and impartial sources is critical. For example, data should be benchmarked against a third party to avoid bias and add credibility
• “In the research stage, a bigger data set allows more in-depth analysis. In the edit phase, it is important to assess whether all your extra information buries the main point of the story or enhancwes [it].”

Visualising large amounts of text

If you are working with text rather than numbers there are ways to visualise that as well. Word clouds, for instance, show which words are used most often in a particular document (such as a speech, bill, or manifesto) or data stream (such as an RSS feed of what people are saying on Twitter or blogs). This can be particularly useful in drawing out the themes of a politician’s speech, for example, or the reaction from people online to a particular event. They can also be used to draw comparisons – word clouds have been used in the past to compare the inaugural speeches of Barack Obama with those of Bush and Clinton; and to compare the 2010 UK election manifestos of the Labour and Conservative parties. The tag cloud is similar to the word cloud, but typically allows you to click on an individual tag (word or phrase) to see where it has been used.

There are other forms for word visualisation too, particularly around showing relationships between words – when they occur together, or how often. The terminology varies: visualisation tool ManyEyes, for example, calls these word trees and phrase nets but other tools will have different names.

Once again, I’d welcome any comments on areas I may have missed or things journalists should consider. I’ve had to split this section into two, so Part 4 continues to look at visualisation, and focuses on tools and publishing.

This is a draft from a book chapter on data journalism (the first, on gathering data, is here). I’d really appreciate any additions or comments you can make – particularly around ways of spotting stories in data, and mistakes to avoid.

UPDATE: It has now been published in The Online Journalism Handbook.

“One of the most important (and least technical) skills in understanding data is asking good questions. An appropriate question shares an interest you have in the data, tries to convey it to others, and is curiosity-oriented rather than math-oriented. Visualizing data is just like any other type of communication: success is defined by your audience’s ability to pick up on, and be excited about, your insight.” (Fry, 2008, p4)

Once you have the data you need to see if there is a story buried within it. The great advantage of computer processing is that it makes it easier to sort, filter, compare and search information in different ways to get to the heart of what – if anything – it reveals.

The first stage in this process, then, is making sure the data is in the right format to be interrogated. Quite often this will be a spreadsheet or CSV (comma-separated values) file. If your information is in a PDF you will not be able to do a great deal with it other than re-type the values into a new spreadsheet (making sure to check you have not made any errors). A Word or Powerpoint document is likely to require the same work.

If the information is already online you can sometimes ‘scrape’ it – that is, automatically copy the relevant information into a separate document. How easy this is to do depends on how structured the information is. A table in a Wikipedia entry, for example, can be ‘scraped’ into a Google spreadsheet relatively easily (Tony Hirst gives instructions on how to do this at http://ouseful.wordpress.com/2008/10/14/data-scraping-wikipedia-with-google-spreadsheets/) and an online CSV file and certain other structured data can be scraped with Yahoo! Pipes (see below for more on using Yahoo! Pipes) but most scraping will involve programming (the online tool ScraperWiki provides one environment to help you do this).

Insert: Cleaning up data

Whether you have been given data, had to scrape it, or copied it manually, you will probably need to clean it up. All sorts of things can ‘dirty’ your data, from misspellings and variations in spelling, to odd punctuation, mixtures of numbers and letters, unnecessary columns or rows, and more. Computers, for example, will see ‘New Town’, ‘Newtown’ and ‘newtown’ as three separate towns when they may be one.

This can cause problems later on when analysing your data – for example, calculations not working or results not being accurate.

Some tips for cleaning your data include:

• Use a spellchecker to check for misspellings. You will probably have to add some words to the computer’s dictionary.
• Use ‘find and replace’ (normally in the Edit menu) to remove double-spaces and other common punctuation errors. Alternatively, if you are in Excel you can create a new column and use the =TRIM() function, which will copy the contents of the cell in the brackets and remove any spaces.
• Remove duplicate entries – if you are using Excel there are a few ways to do this under the Data tab – search for duplicates in Help.

For more tips on Excel specifically see this guide: http://office.microsoft.com/en-us/excel/HA102218401033.aspx

For cleaning up very large sets of data you might want to use a data cleaning tool like Google Refine.

Spotting the story

Once your data is cleaned you can start to look for anything newsworthy in it. There are some obvious places to start: if you are dealing with numbers, for example, you can work out what the ‘average’ is (the average bonus paid to council employees, for example). Similarly, you might look for the term which appears most often (e.g. the most common reason given for arresting terrorist suspects).

However, Kaiser Fung, a statistician whose blog Junk Charts is essential reading on the field, notes the dangers in lazily reaching for the average when you want to make an editorial point:

“Averaging stamps out diversity, reducing anything to its simplest terms. In so doing, we run the risk of oversimplifying, of forgetting the variations around the average. Hitching one’s attention to these variations rather than the average is a sure sign of maturity in statistical thinking. One can, in fact, define statistics as the study of the nature of variability. How much do things change? How large are these variations? What causes them?” (Fung, 2010, p4)

So while averages and modes (a mode is the number or term which appears most often) can be interesting discoveries, they should most often be used as a starting point for more illuminating investigation – normally involving leaving your computer to make phonecalls and speak to sources.

If you are looking at data over time, you can look to see what has increased over that period, or decreased – or disappeared.

But you will need to gather further data to provide context to your figures. If, for example, more council staff are receiving bonuses, is that simply because more staff have been employed? How much is spent on wages, and how do your figures compare? If you are comparing one city with another, understand how their populations differ – not just in aggregate, but in relevant details such as age, ethnicity, life expectancy, etc. You will need to know where to access basic statistics like these – the National Statistics website is often a good place to start.

Sometimes a change in the way data is gathered or categorised can produce a dramatic change in the data itself. In one example, designer Adrian Short obtained information (via an FOI request) on parking tickets from Transport for London that showed the numbers of tickets issued against a particular offence plummeted from around 8,000 to 8 in the space of one month (Arthur, 2009). Had people suddenly stopped committing that parking offence, or was there another explanation? A quick phonecall to Transport for London revealed that traffic wardens were issued with new handsets around the same time. Guardian journalist Charles Arthur hypothesised:

“Could it be that s46 [another offence which had a steep rise at the same time] is the default on the screen to issue a new ticket, and that wardens don’t bother to change it? Whatever it is, there’s a serious problem for TfL if those aren’t all s46 offences which have been ticketed since August 2006. Because if the ticket isn’t written out to the correct offence, then the fine isn’t payable. Theoretically, TfL might have to pay back millions in traffic fines for people who have been ticketed for s46 offences when they were actually committing s25 or s30 offences.”

This particular story came about at least in part because that information was easy to visualise.

The next section covers visualisation. In the meantime, once again I’d really appreciate any additions or comments – particularly around ways of spotting stories in data, and mistakes to avoid.

The following is a draft from a book about online journalism that I’ve been working on. I’d really appreciate any additions or comments you can make – particularly around sources of data and legal considerations

The first stage in data journalism is sourcing the data itself. Often you will be seeking out data based on a particular question or hypothesis (for a good guide to forming a journalistic hypothesis see Mark Hunter’s free ebook Story-Based Inquiry (2010)). On other occasions, it may be that the release or discovery of data itself kicks off your investigation.

There are a range of sources available to the data journalist, both online and offline, public and hidden. Typical sources include:

• national and local government;
• bodies that monitor organisations (such as regulators or consumer bodies);
• scientific and academic institutions;
• health organisations;
• charities and pressure groups;
• business;
• and the media itself.

One of the best places to find UK government data online, for example, is Data.gov.uk, an initiative influenced by its US predecessor Data.gov. Data.gov.uk – launched in January 2010 with the backing of the inventor of the World Wide Web, Sir Tim Berners-Lee – effectively acts as a search engine and index for thousands of sets of data held by a range of government departments, from statistics on the re-offending of juveniles to the Agricultural Price Index. The site also hosts forums for users to discuss their use of the data, examples of applications using data, further information on how to use the data, and technical resources.

At a regional level, local authorities are also releasing information that can be used as part of data journalism projects. The quality, quantity and accessibility of this information varies enormously by council, but there is continuing pressure for improvement in this area.

There are also a number of volunteer projects, such as OpenlyLocal and Mash The State, that make local government data available in as accessible a format as possible, while the organisation MySociety operate a group of websites providing easy access to information ranging from particular politicians’ voting record (TheyWorkForYou) to local problems (FixMyStreet) and information about a particular area’s transport links and beauty (Mapumental). MySociety also runs a petitions website for Downing Street, and websites that allow people to pledge to do something if other people sign up too (PledgeBank), to find groups near you (GroupsNearYou), contact your MP (WriteToThem) or be contacted by them (HearFromYourMP).

Private companies and charities

In the private sector, a number of organisations regularly release data online, from tables and research reports published on company websites to the annual reports that are filed with bodies such as Companies House. Also worth looking at is the web project Companies Open House, which seeks to make company information more easily accessible.

The Charity Commission is an excellent source of information on registered charities, who must file accounts and annual reports with the organisation. The commission also conducts occasional research into the sector.

Regulators, researchers and the media

NHS foundation trusts likewise must file reports to their regulator, Monitor. And you will find similar regulators in other areas such as the Financial Services Authority, Ofcom, Ofwat, Ofqual, the General Medical Council, the General Social Care Council and the Pensions Regulator to name just a few.

For academic and scientific research there are hundreds of specialist journals. Most have online search facilities which will provide access to summaries. To get access to the full paper you will probably need to use the library of a university which has a subscription. For access to a journal on midwifery, for example, your best bet is to give a quick call to the nearest university which teaches courses in that field. Although university libraries increasingly limit access to students, you can request a special pass. For access to the data on which research is based it is likely you will need to contact the author.

Media organisations such as The Guardian and the New York Times publish ‘datablogs’ that regularly release sets of data produced or acquired by investigations, ranging from scientific information about global warming to lists of Oscar winners. These can be a rich source of material for the data journalist, and a great starting point for the beginner as they are often ‘cleaner’ than data from elsewhere.

The Guardian and the New York Times websites are also among an increasing number of web platforms generally which are making their own data available via APIs (Application Programming Interfaces). Typically, websites which offer this access are social networking sites (such as Flickr and Twitter).

Accessing this data typically requires a level of technical ability, but can be particularly useful in measuring activity across social networks (for example sharing and publishing). Even if you don’t have that technical ability, understanding the possibilities can be extremely useful when working with web developers on a data journalism project (see the part of this chapter on mashups for more information on APIs).

Using search engines to find data

If you are using a search engine to find the data you are looking for, you should familiarise yourself with the advanced search facility, where you can often specify the format of the file you are looking for. Searching specifically for spreadsheets (files ending in .xls), for example, is likely to get you to data more quickly. Similarly, official reports can often be found more effectively by searching for PDF format, while Powerpoint presentations (.ppt) will sometimes contain useful tables of data. You can also include ‘XML’ or ‘RDF’ in your search terms if you think your data may be in those or other formats.

Advanced search also allows you to specify the type of website you are searching – those ending in .gov.uk (government), .org and .org.uk (charities), .ac.uk (educational establishments), .nhs, .police.uk and .mod (Ministry of Defence) are just some that will be particularly relevant (you can also specify an individual site – for instance, that of a local council). A basic familiarity with these search techniques – for example limiting your search to spreadsheets on .gov.uk websites – can improve your results.

Live data

Another type of data to think about is live data that is not stored anywhere yet but, rather, will be produced at a particular time. A good example of this would be how newspapers are increasingly using Twitter commentary to provide context to a particular debate. Part of the Guardian’s coverage of Tony Blair’s appearance at the Chilcot Inquiry into the Iraq War, for example, used the data of thousands of Twitter updates (‘tweets’) to provide a ‘sentiment analysis’ timeline of how people reacted to particular parts of his evidence as it went on. Similar timelines have been produced for political debates and speeches to measure public reaction.

Preparation is key to live data projects – where will you get the data from, and how will you filter it? How can you visualise it most clearly? And how do you prevent it being ‘gamed’ (users intentionally skewing the results for fun or commercial or political reasons)?

Legal considerations

Whatever data you are acquiring, you will need to consider whether you have permission to republish that data. Data may be covered by copyright, or may raise issues of data protection or privacy. Even apparently anonymous information can sometimes be traced back to individual users (Barbaro & Zeller, 2006), and while government information is paid for by public money, for example, it is, strictly speaking, often covered by Crown Copyright, while organisations like Ordnance Survey and Royal Mail have been notoriously protective of geographical information and postcodes (see Brooke, 2010).

Books and FOI

Of course, there is also a rich range of data available in books that the data journalist should familiarise themselves with – from books of facts and statistics to almanacs, from the Civil Service Year Book (also online) to volumes like Who’s Who (online at ukwhoswho.com – your library may have a subscription).

Particularly useful is the data held by public bodies which can be accessed through a well-worded Freedom of Information (FOI) request. Heather Brooke’s book Your Right To Know (2007) is a key reference work in this area, and the online tool WhatDoTheyKnow is particularly useful in allowing you to submit FOI requests easily, as well as allowing you to find similar FOI requests and the responses to them.

When requesting data through an FOI request, it is always useful to specify the format that you wish the information to be supplied in – typically a spreadsheet in electronic format. A PDF or Word document, for example, will mean extra work at the next stage: interrogation.

UPDATE: Tim Davies lists a couple of further avenues along these lines:

http://unlockingservice.data.gov.uk/ provides a root for requesting data is opened up by the Data.gov.uk team. It’s not backed by the legal framework of FOI, but may play a role in data requests under the currently debated ‘Right to Data’ legislation.

IsItOpenData.org provides a useful tool for asking non-public bodies to share their data as open data, or to clarify the licensing.”

Once again – this is a draft: I’d really appreciate any additions or comments you can make – particularly around sources of data and legal considerations. Part 2 – on interrogating data – can be found here.

Here’s another draft section from the book chapter on UGC I’m currently writing I’ve written which I’d welcome your input on. I’m particularly interested in any other objectives you can think of that news organisations have for using UGC – or the strategies adopted to achieve those.

A common mistake made when first venturing into user generated content is to focus on the technology, rather than the reasons for using it. “We need to have our own social network!” someone shouts. But why? And, indeed, how do you do so successfully?

A useful framework to draw on when thinking about how you approach UGC is the POST process for social media strategy outlined by Forrester Research (Bernoff, 2007). This involves identifying:

1. People: who are your audience (or intended audience), and what social media (e.g. Facebook, blogs, Twitter, forums, etc.) do they use? Equally important, why do they use social media?
2. Objectives: what do you want to achieve through using UGC
3. Strategy: how are you going to achieve that? How will relationships with users change?
4. Technology: only when you’ve explored the first three steps can you decide which technologies to use

Some common objectives for UGC and strategies associated with those are listed below:

Can you add any more? What strategies have you used around UGC?

UPDATE: ‘Pursue the goal, not the method‘ by Chris Brogan puts this concept well in broader terms.