This is a draft from a book chapter on data journalism (part 1 looks at finding data; part 2 at interrogating data; part 3 at visualisation, and 4 at visualisation tools). I’d really appreciate any additions or comments you can make – particularly around tips and tools.

UPDATE: It has now been published in The Online Journalism Handbook.

Mashing data

Wikipedia defines a mashup particularly succinctly, as “a web page or application that uses or combines data or functionality from two or many more external sources to create a new service.” Those sources may be online spreadsheets or tables; maps; RSS feeds (which could be anything from Twitter tweets, blog posts or news articles to images, video, audio or search results); or anything else which is structured enough to ‘match’ against another source.

This ‘match’ is typically what makes a mashup. It might be matching a city mentioned in a news article against the same city in a map; or it may be matching the name of an author with that same name in the tags of a photo; or matching the search results for ‘earthquake’ from a number of different sources. The results can be useful to you as a journalist, to the user, or both.

Why make a mashup?

Mashups can be particularly useful in providing live coverage of a particular event or ongoing issue – mashing images from a protest march, for example, against a map. Creating a mashup online is not too dissimilar from how, in broadcast journalism, you might set up cameras at key points around a physical location in anticipation of an event from which you will later ‘pull’ live feeds: in a mashup you are effectively doing exactly the same thing – only in a virtual space rather than a physical one. So, instead of setting up a feed at the corner of an important junction, you might decide to pull a feed from Flickr of any images that are tagged with the words ‘protest’ and ‘anti-fascist’.

Some web developers have built entire sites that are mashups. Twazzup (twazzup.com) for example, will show you a mix of Twitter tweets, images from Flickr, news updates and websites – all based on the search term you enter. And Friendfeed (friendfeed.com) pulls in data that you and your social circle post to a range of social networking sites, and displays them in one place.

Mashups also provide a different way for users to interact with content – either by choosing how to navigate (for instance by using a map), or by inviting them to input something (for instance, a search term, or selecting a point on a slider). The Super Tuesday YouTube/Google Maps mashup, for instance, provided an at-a-glance overview of what election-related videos were being uploaded where across the US.

Finally, mashups offer an opportunity for juxtaposing different datasets to provide fresh, sometimes ongoing, insights. The MySociety/Channel 4 project Mapumental, for example, combines house price data with travel information and data on the ‘scenicness’ of different locations to provide an interactive map of a location which the user can interrogate based on their individual preferences.

Mashup tools

Like so many aspects of online journalism, the ease with which you can create a mashup has increased significantly in recent years. An increase in the number and power of online tools, combined with the increasing ‘mashability’ of websites and data, mean that journalists can now create a basic mashup through the simple procedures of drag-and-drop or copy-and-paste.

A simple RSS mashup, which combines the feeds from a number of different sources into one, for example, can now be created using an RSS aggregator such as xFruits (xfruits.com) or Jumbra (jumbra.com).

Likewise, you can mix two maps together using the website MapTube (maptube.org) which also contains a number of maps for you to play with.

And if you want to mix two sources of data into one visualisation the site DataMasher (datamasher.org) will let you do that – although you’ll have to make do with the US data that the site provides. Google Public Data Explorer (google.com/publicdata) is a similar tool which allows you to play with global data.

But perhaps the most useful tool for news mashups is Yahoo! Pipes (pipes.yahoo.com).

Yahoo! Pipes allows you to choose a source of data – it might be an RSS feed, an online spreadsheet or something that the user will input – and do a variety of things with it. Here are just some of the basic things you might do:

• Add it to other sources
• Combine it with other sources – for instance, matching images to text
• Filter it
• Count it
• Annotate it
• Translate it
• Create a gallery from the results
• Place results on a map

You could write a whole book on how to use Yahoo! Pipes – indeed, people have – so we will not cover the practicalities of using all of those features here. There are also dozens of websites and help files devoted to the site (which you should explore). Below, however, is a short tutorial to introduce you to the website and how it works – this is a good way to understand how basic mashups work, and how easily they can be created.

Mashups and APIs

Although there are a number of easy-to-use mashup creators listed above, really impressive mashups tend to be written by people with knowledge of programming languages, and use APIs. APIs (Application Programming Interface) allow websites to interact with other websites. The launch of the Google Maps API in 2005, for example, has been described as a ‘huge tipping point’ in mashup history (Duvander, 2008) as it allowed web developers to ‘mash’ countless other sources of data with maps. Since then it has become commonplace for new websites, particularly in the social media arena, to launch their own APIs in order to allow web developers to do interesting things with their feeds and data – not just mashups, but applications and services too.

If you want to develop a particularly ambitious mashup it is likely that you will need to teach yourself some programming skills, and familiarise yourself with some APIs (the APIs of Twitter, Google Maps and Flickr are good places to start).

Box-out: Anatomy of a feed

The image below from ReadWriteWeb shows the code behind a simple Twitter update. It includes information about the author, their location, whether the update was a reply to someone else, what time and where it was created, and lots more besides. Each of these values can be used by a mashup in various ways – for example, you might match the author of this tweet with the author of a blog or image; you might match its time against other things being published at that moment; or you might use their location to plot this update on a map.

While the code can be intimidating, you do not need to understand programming in order to be able to do things with it. Of course, it will help if you do…

This was the final session in my undergraduate Online Journalism module (the other classes can be found here), taught last May. It’s a relatively brief presentation, just covering some of the possibilities of mashups and RSS, and some tools. The majority of the class is taken up with students using Yahoo! Pipes to aggregate a number of feeds.

I didn’t know how students would cope with Yahoo! Pipes but, surprisingly, every one completed the task.

As a side note, this year I kicked off the module with students setting up Twitter, Delicious and Google Reader – and synchronising them, so the RSS feed from one could update another (e.g. bookmarks being published to Twitter). This seems to have built a stronger understanding of RSS in the group, which they are able to apply elsewhere (they also have widgets on their blogs pulling the RSS feeds from Twitter & Delicious; and their profile page on the news website – built by Kasper Sorensen – pulls the latest updates from their Twitter, Delicious and blog feeds).

There’s a fabulous post over at the Center for Social Media on when using copyrighted material in video comes under fair use. If the work is ‘transformative’ then there’s a strong case for fair use. Examples include:

1. Adding satirical subtitles, fan tributes, parody, critique
2. Using copyright material for illustration of example (e.g. stages in a star’s career)
3. Accidental capture – e.g. music playing in the background while someone dances (if unstaged)
4. Documenting an event or experience, e.g. presence at a concert
5. Mashups, remixes or collages that create new meaning from old material

But of course this is all under American law. My question is: how far do these same examples go under UK law? I’d love to know your experiences and interpretations.

Here’s another tutorial on the mashup platform Yahoo! Pipes, showing how you can use it to create a meta-search that will push any search term by the user through a number of search engines, and present you with a combined result (and RSS feed). A finished version of the pipe can be seen here.

This tutorial builds on a previous post I published on how to create basic mashups with Yahoo! Pipes. If you haven’t any knowledge of Pipes you should probably read that first.

How to create a custom meta-search in Yahoo! Pipes

First, you obviously need to log in to Yahoo! Pipes, and click on Create a Pipe. You’ll be taken to the Pipe editing interface: on the left will be a menu with a series of sections (User Input, Url, Operators, etc.) to choose modules from. In the centre will be the canvas where you create your pipe – and at the bottom a ‘Debugger’ area where you can see the results of any particular part of your pipe.

In the area on the left, under the ‘User Input’ section, click on the ‘Text Input‘ module and drag it onto the canvas (or you can click on the + sign for it to be placed for you).

Select the Text Input module

In the box marked ‘Prompt’ type the instruction text for users of the pipe, e.g. ‘What do you want to search for?’. If there’s a default search you want to have appear in the search box to begin with, enter it in the box marked ‘Default’.Under the ‘Url’ section, click on ‘URL Builder‘ and drag it onto the canvas.

Select the URL Builder module

Go to a search engine that offers RSS feeds for searches, e.g. Google News. Do a test search (e.g. ‘fire’) and copy the address of the RSS feed for those search results (on Google News it will be at the foot of the page)

In the ‘URL Builder’ module paste the address of that RSS feed into the box marked ‘Base’

You’ll notice that the rest of the module now expands to include all of the parameters that the URL includes. The main one we are interested in is the one that relates to our search (you can tell which one it is as it should have the topic of your search in the right hand column) – in the case of Google News, this is ‘q’.

Customising the URL Builder module

Drag a wire from the circle at the bottom of the Text Input module to the circle to the right of that search term (e.g. ‘fire’). It should turn grey and you should now be able to see a wire joining the two modules. Your search term should now be replaced by ‘text [wired]‘.

Now drag a third module onto your canvas: this time it’s under the ‘Sources’ heading, and the module is called ‘Fetch Feed‘.

The Fetch Feed module

This time drag a wire from the circle at the bottom of ‘URL Builder‘ to the circle on the right of the ‘Fetch Feed‘ module. Again, it should turn grey and the previously empty box should now read ‘text [wired]‘.

A grey wire linking the URL Builder and Fetch Feed modules

Drag another ‘URL Builder‘ module onto the canvas and repeat the process detailed above for a separate search engine. Connect your ‘Text Input‘ module to it in the same way. You may have to delete some of the parameters to make these work. A blog search on IceRocket, for example, includes the parameter ‘tab’. Deleting this (with the ‘-’ button next to it) appears to make the search work in Pipes. If you know of any similar tweaks, please let me know.

In the ‘Fetch Feed’ module click on the ‘+’ icon to add a new, empty, box. Click on the circle at the bottom of your second ‘URL Builder‘ module to the circle to the right of that empty box. Again, it should turn grey and you should now be able to see a wire joining the two modules and the box should now read ‘text [wired]‘.

You can repeat these steps for further search engines.

Once you’ve finished, from the ‘Operators’ heading on the left, drag a ‘Union‘ module onto the canvas. You need this to combine your feeds.

To combine them, drag a wire from the bottom of all ‘Fetch Feed‘ modules to separate circles at the top of ‘Union‘ at the bottom of the canvas. Unlike the other pipes this will be blue.

Wires connect the Fetch Feed modules to the Union module, which in turn is connected to the Output module

Finally, drag a pipe from the bottom of ‘Union‘ to the top of ‘Pipe Output‘ at the bottom of the canvas.

That is it. Save your pipe and click on ‘Back to My Pipes’ towards the top centre of the screen and you should see your pipe listed. Click on that pipe to run it – you should see a text entry box towards the top where you can enter your search to change the results. If you want to embed this somewhere, click on ‘Get as a Badge’ to get the code.

However, before you do that you’ll notice that results will be clustered by search engine, and also may contain duplicate results. You’ll need to click on ‘Edit Source’ and add a couple more modules to tweak your pipe and solve the problem.

The modules you will need to use are the Unique module (in the Operators section) to filter out duplicate results (probably place it between Union and Pipe Output - drag wires to remove them), and the Sort module (also in the Operators section) to order results by date (otherwise they will cluster by search engine).

I’m by no means any expert on this, but put this out there for others to build on, comment on and correct.

Many thanks to Paul Daniel who showed me much of this as part of an open MA Online Journalism session and whose blog is a great resource on Pipes.

The following is part of a chapter for a forthcoming book on online journalism. Contributions welcome.

Maps have become a familiar part of the news language online due to a number of advantages:

• They provide an easy way to grasp a story at a glance
• They allow users to drill down to relevant information local to them very quickly
• Maps can be created very easily, and added to relatively easily by non-journalists
• Maps draw on structured data, making them a very useful way to present data such as schools tables, crime statistics or petrol prices
• They can be automated, updating in response to real-time information

News organisations have used maps in a number of ways:

Mapping public data

In the US there has been a rich history of mapping crime statistics online, stimulated enormously by Adrian Holovaty’s ChicagoCrime.org, which he later developed into EveryBlock, a website which doesn’t just map crime – but also planning and liquor applications, filming, news stories, street closures and restaurant inspections. EveryBlock pulls in information from a range of sources and displays the results based on the zip code you enter, giving you a picture of everything happening local to you.

In the UK the most significant mapping of public data has been around elections. The Telegraph election maps, for example, pull from a database to provide links to specific statistics and reports. The BBC, meanwhile, have provided maps that you could change based on your own prediction of the ‘swing’ a political party might experience.

More recently, however, as there has been an increasing move towards publishing public data and increasing use of the Freedom of Information Act to obtain public data, types of data have broadened.

Working with the public

When a number of motorists started reported problems with their cars, the BBC was able to gather data from viewers and website visitors to compile a map of cases richer than any motoring organisation or transport department. That data allowed them to pinpoint the particular petrol stations where contaminated fuel was being supplied. During floods in Berkshire BBC Radio Berkshire also used maps to show the worst affected areas, and what people were saying, along with data about emergency services.

On a regional level the Manchester Evening News plots information about congestion and roadworks on its travel map; and the Hartlepool Mail has used its readers’ contributions to map pot holes and derelict areas of the town. On the less serious side, newspapers have used maps for sightings of an unusual bird, to identify where readers are living around the globe, and to map sightings of unidentified flying objects.

Increasingly news organisations are providing raw data to users for them to create their own maps. The Guardian, for example, provided travel expenses data for MPs for users to analyse. Tony Hirst, publisher of the blog OUseful, visualised that data on a map in a way that made it easy to tell at a glance which MPs were claiming more for travel than other MPs who lived nearby. The newspaper has a Datablog which regularly releases data for users to visualise in different ways (The Times also have one, called Times Labs Blog, which at the time of writing did not release data; the New York Times does release data in its Visualization Lab).

Maps and mashups

Maps work particularly well when combined with another service that includes geospatial data – for instance, latitude and longitude, or a postcode or placename. During the US election a number of these ‘mashups’ appeared showing, for example, Twitter tweets or YouTube videos about the election displayed on a map. During the Beijing Olympics BBC Sport used similar technology to display tweets, blog posts and photos on a map of the Olympic village.

Image maps

In addition to traditional cartographic maps, it is possible to use images of anything as a ‘map’ that you navigate in the same way with your mouse, clicking on particular areas to bring up relevant detail. Examples have included the route of a race or river, a building’s floorplan, a timeline, line or bar chart, or even a group photo. In one excellent example, NPR took a picture of health lobbyists attending the hearing of a new health bill. Users could click on individual people to find out more about them – and were also invited to identify others in the picture. Similarly, The Guardian’s Interactives section often includes diagrams where you can roll over different areas to find out information about different aspects of the process or story.

A few years ago you needed to have skills in Flash or Dreamweaver to create an image map. However, a number of web-based tools (e.g. Vuvox and FineTuna) have since been launched that allow you to create these more easily.

Geotagging and the semantic web

Both the rise in mapping and a rise in people accessing news on mobile phones has created a demand for ‘geotagged’ (or geocoded) news. Geotagging a news article means adding geographical information to it – usually, latitude and longitude – in a way that makes it easy for search engines and news distribution platforms to understand what area that news article refers to.

In practice this means that if you are on a mobile phone with GPS technology you can search for ‘restaurant reviews near me’ or ‘crime stories near me’. Likewise, if you were looking for a new house you could easily find stories about the local schools, or plans for new buildings. Many search engines take into account the searcher’s own location when bringing up search results – so including geotagging in news stories would also increase the likelihood of your content being found by a local searcher.

Most news organisations are exploring geotagging in some capacity – in many cases, changing their content management systems so that journalists can add such information when publishing a story. Some have used this information to launch ‘hyperlocal’ parts of their news websites that allow users to read stories specifically about a particular postcode.

At the same time, organisations like Reuters have developed technologies that add geolocation data to stories after they have been written – using semantic web technology to look for place names in the article text and disambiguate them, understanding that the ‘Birmingham’ referred to is the UK’s second city and not the place of the same name in Alabama, US.

The weakness of the latter approach is that it can only work on the information contained in the article, which may not be specific enough. Meanwhile, journalists geotagging their own articles need to be aware of the privacy and legal implications in, for example, identifying the specific house that a criminal lives in. Normally postcodes in the UK are vague enough for this not to happen – while licensing arrangements mean that Google Maps’ mapping of UK postcodes varies in accuracy.

How to involve users in mapping

There are 3 broad approaches to mapping contributions from users. The first is to process every contribution manually – taking emails, phonecalls, comments and texts and entering them into the map yourself. This has clear advantages in being able to verify the information and keep the map working properly, but obvious disadvantages in the amount of time it requires for a journalist and how long it takes for the map to be updated.

A second approach is to publish the map in editable format – that is, allow anyone to edit the map directly. This obviously has the advantage of not requiring any further work from the journalist other than checking the map regularly and correcting any mistakes. However, it does require a certain level of technical competence from users, and if you’re editing a Google Map, which doesn’t yet have any granularity of control, you will find users accidentally editing each others’ entries and the title of the map itself.

The third approach is to part-automate the process in a way that addresses the weaknesses above. You can, for example, set up a Google Map so that it displays data from a Google spreadsheet. Publishing that spreadsheet and allowing users to edit it will likely result in more contributions, fewer errors, and easier correction (if anyone vandalises the spreadsheet you can easily ‘revert’ to previous versions). You can also create a Google Form for that spreadsheet, which you can publish on a website for users to fill in by answering a few questions. This makes it even easier for users to enter information and prevents them editing others’ – although it may mean duplicate entries from different people entering the same information.

Have I left any considerations or concepts out? Any great examples that deserve highlighting? Let me know

The following is the last part of a series of responses to the government inquiry into the future of local and regional media. We will be submitting the whole – along with blog comments – to the Culture, Media and Sport Committee. This post, by Alex Lockwood, looks at:

“How to fund quality local journalism”

The bottom has fallen out of the traditional publishing business model–and with it goes the hefty dividends expected by shareholders (e.g. £48.4m in 2008 for the Trinity Mirror Group). The future of local quality journalism can only remain with the current crop of regional newspaper publishers if they radically change their expectations, and innovate.

That might not happen. If it doesn’t, they will die off, and the future of quality local journalism will take a huge – but not definitive – blow. Then the future lies with new initiatives and the local communities themselves – passionate and entrepreneurial people, only some of whom will be journalists. What about local council initiatives to publish newspapers and local information? That’s not the way to go – covered in Part 3.

But how to fund it? Here are eight suggestions for the future of local journalism funding:

1. Save the big regional publishers through a public subsidy? The culture secretary, Andy Burnham, has already ruled that out: no state subsidies for beleaguered local newspapers. In some ways, that is good. Let’s not shore up businesses that have met requirements of shareholders over those of the local community, and which have – with a few notable exceptions – failed to innovate.

2. But… as Andy Price argued on this blog yesterday, “The regional press is the only institution with enough professional journalists to really cover civic Britain successfully.” So where public money is available, e.g. through the Digital Britain programme, efficiencies in government funding are necessary. As the authors of After the Crunch, published last week, write, “The DCMS, BERR, DCSF, Treasury, DIUS between them, spend a lot of money in the name of ‘creativity’ and ‘innovation’, but much of their effort is frustrated by the lack of a coherent approach.” If quality local journalism is a public service, then what portion of the public service budget could go to newspapers? And only on the basis that they reform their structures (as suggested by @Geoffrey Gevalt).

3. That could be knitted together with a second point made in After the Crunch: that “the small-scale nature of creative industry enterprises connects more easily, and more productively with smaller-scale government.” The government could streamline legislation and funding frameworks for supporting media organisations at local levels without the baggage of outdated business models. They can work with Business Link and entrepreneurship schemes to offer many more bursaries and small business grants to new ventures that establish in their business plans a commitment to produce quality local journalism covering local democracy issues. These will most probably be started by two groups of people: those local journalists who have been made redundant, and who are deeply passionate about local democracy and community; and new entrepreneurs who can see the potential in investing in a portfolio of local media products using new, free technologies and mash ups.

4. Where regional publishers can prove they are adapting to the new media environment, individual papers or sub-regional groups (similar to what Peter Preston called for in the Observer last Sunday) could be cut out of the dying corpse of their parent company, and given subsidies to see them through the migration to a new business model.

5. Reduce costs through ditching daily print routines. Newspapers become professional news magazines published once a week but constantly updated online by continuing to grow community engagement and news as a conversation, and by investing in non-traditional ways to access information, e.g. these maps empowering social change (h/t @JoshHalliday).

6. Media organisations, both new and traditional, turn to community-owned, community-sourced local journalism.  Two-hundred years ago it was pampheteering. In 1932, it was nine interested individuals fed up with newspaper oligarchs who raised £40,000 and set up their own local paper, the Bristol Evening Post. Crowd-sourcing and crowd-funding have always been a part of the future of media. As argued for by former Northern Echo editor Peter Sands this morning on the Radio 4 Today programme.

7. Take a leaf out of new magazine membership models, as developed by numerous brands but articulated here via Alyce Alston: sell bundles of information.

8. Fund training programmes for current (recently redundant?) journalists in new technologies and entrepreneurship. This gives the next generation of media entrepreneurs preparedness for the need to adapt to rapid media change – and that means more money into projects such as Infuze at the University of Central Lancashire (h/t Laura Oliver)

What the typical local media organisation might look like?
So how about this? The future of quality local journalism is published immediately online and weekly in print, probably in magazine format.

• A small group of editors, journalists and community managers work with a network of contributors to develop feeds in a number of formats, e.g. news stories linked to local maps, for geographical and issue-based hyper-localities: all of this online, using APIs to mash together maps, local government records, planning information etc.
• A printed version provides a format for the weekend read and brings in advertising—similar to the ways the best Teesside hyper-local content gets published in weekly papers.
• The media organisation supports investigative reporting through entertainment, sport and feature copy that attracts advertising and sponsorship.
• The magazine is distributed freely around the local region.
• This local brand was set up with a government grant, including ongoing training in technology and entrepreneurship.
• The magazine is owned by the community through a crowd-funded structure (ten thousand people each pay £20 as a yearly debenture – not a subscription) and that community then have a vote on the governance and issues covered by the magazine… Want journalists to prioritise investigations into local planning decisions? Then pay for it.

Think of it as a combination of The Age of Stupid meets Spot.Us.

What other ideas are there?

As part of a group response to  the government‘s inquiry into the future of local and regional media, Paul Bradshaw looks at the role of local authorities in regional journalism. Blog comments will be submitted to the inquiry as well as the blog posts.

So. The Committee for Culture, Media and Sport want responses on “The appropriateness and effectiveness of print and electronic publishing initiatives undertaken directly by public sector bodies at the local level”

The question of what public sector bodies should be allowed to publish, how that affects local journalism, the local economy, and local democracy, is one of the most difficult to resolve – not least because it involves so many interconnected elements.

The first problem is that any discussion runs the risk of conflating a number of separate but interlinked elements:

• local councils and local democracy are not the same thing; 
• local newspapers and local journalism are also two different things.

Whatever model emerges must recognise that papers are not the only places where public discussion takes place, and print journalists are not the only people holding power to account.

We must not prop up newspapers at the expense of the opportunity to support other emerging forums of public engagement. Any question about the future of local media must acknowledge that ‘local media’ now includes any number of blogs, websites, forums, social networks and other, distributed, media. 

As local citizens increasingly receive their ‘news’ from those forms of media, and local journalists increasingly rely on those to understand the concerns of local people, the actions of public sector bodies need to be responsive and supportive of that.

The economic role

Equally, while newspapers have an important role to play in local economies, we should not ignore the growing number of independently owned local print and online publications that have the potential to provide another source of economic growth. 

In other words, just as local newspapers protested at the potential effect BBC Local might have on their markets, we should be aware of how support for local newspaper chains might undermine the efforts of less vocal, independent news operations.

Council newspapers

The same economic argument is used to criticise the increasing number of local authorities publishing newspapers of their own.

The Local Government Association recently released research claiming council magazines were “not a threat to local media” – a useful survey, but the way it is reported by the LGA demonstrates the dangers of allowing local authorities to report on their own activities.

The statistic “Almost 60 per cent of council publications contain 10 per cent or less of advertising” is framed as part of the case that local magazines are not a threat. A casual reader would swallow that. A critical writer would point out that this means a very significant 40% of council publications carry reasonably large amounts of advertising – and even those carrying less than 10% of advertising are still having an economic impact on local newspapers. Not mentioned is whether there is an increasing trend towards carrying more advertising, which anecdotally looks to be the case. 

The move into council newspapers is a move to cut out the middleman, with little obvious benefit for local citizens: for the reasons given above it is unlikely to be informing in any meaningful sense, and even less likely to hold its paymasters to account.

The financial implications are concerning: there is the drain on public funds of of publication and distribution. There is the negative economic impact of reallocating communications and marketing budgets that might otherwise go towards local media. If indeed “People deserve to know what their council tax is being spent on” then there should be restrictions on how council newspapers do that: just the facts, please. No spin, no adverts. They used to call them leaflets.

Rather than publishing pre-packaged, pre-selected information, one way local councils could make a major difference is through publishing information in formats that make it as easy as possible for users to build media of their own from, i.e. ‘mash up’. Examples of this would include:

• RSS feeds of newly published documents
• Documents ‘tagged’ with key names, places, organisations, etc.
• The ability for users to tag documents themselves
• The ability for users to comment on or annotate documents
• Full audio or video of council meetings, etc.
• Use of microformats
• Use of free platforms that support some of the above technologies, e.g. WordPress, Twitter, Delicious

For newspapers, this would provide an efficiency not just in newsgathering (a key way to help reporters find the information they need, quickly, to interrogate it and make connections), but also production and distribution (RSS feeds and tags, for example, can be easily filtered, aggregated and mashed up).

Equally, because this makes it easier for web users to interrogate information, it helps facilitate local amateur and startup media production, including those members of the local community that journalists are increasingly relying upon to do this work.

This is obviously not to say that anyone will be able to use the data in these ways, only that it makes it possible for a wider number of people than before to create media – and to distribute it. The nature of the web is such that it also becomes easier for a wider group of people to find out about that media, and to become engaged with local issues on a social level.

In fact, there’s already an open source platform available that local authorities could look at which releases information in this way – EveryBlock.

But this is not a technical solution to a social problem, but an organisational and cultural solution. It is about openness. 

Automate, aggregate and distribute

And if councils are serious about informing their citizens I think they could go further still. They could publish relevant stories alongside their council webpages. 

If a user is on the council website planning applications page, why not have a feed from local news websites and a selection of top local blogs that have relevant tags? That information is more than likely going to be more readable and informative than the council’s own version, so it is fulfilling the council’s own stated aim of ‘informing the public’ at no extra cost. It is also helping to distribute the news and drive traffic to local news websites (a virtual version of Craig McGill’s suggestion that binmen deliver the news), not to mention the possibility of newspapers selling advertising into those feeds. 

This needn’t be limited to the council website: local authorities distribute information electronically in all kinds of ways – emails to staff, information to bus stops, text messages, local digital TV – providing a future possibility of further automated distribution.

You then have a built-in incentive for local news organisations to cover local government (needless to say this should be enshrined somehow so that councils cannot hold news organisations to ransom).

Anyway, I’ve said enough. It’s a complex area – what should local authorities do?

Will 2009 be the year news organisations finally went open? Barely a month after the New York Times allowed users to build on 28 years of content with its articles API (with immediate results), The Guardian is opening up over a million articles to developers for free as part of its own ‘Open Platform‘.

TechCrunch reports:

“The Guardian is effectively letting control of its content go in order to maximise its reach – and therefore the number of eyeballs that see its brand/content – across the Web.

“the API does not prevent developers from running “commercial applications” using the API – but I’d read the fine detail first. They do say: “You can display your own ads and keep your own revenue. We will require that you join our ad network in the future.”

“… The APIs feature ‘full fat’ feeds and other content including video, audio and photo galleries. You can combine free text search and combine tags to create feeds based on XML, JSON and Atom. 

“… Some early examples include Zemanta’s Guardian topic research demo. It’s a simple app that searches the vast database of Guardian articles via its API and then uses Zemanta’s API to get links to related concepts.

“There is also Content Tagger, an application to provide user-generated tagging on guardian.co.uk content. ApiMaps.org, built by Stamen Design, is designed to crowdsource geodata about Guardian articles. Cass Sculpture Foundation is using the Open Platform to insert lists of articles from the Guardian about its sculptors into their biography and home pages.

Expect more news organisations to follow suit.

Alice Fanning's map of UK eco stories

As a new semester begins it seems a good time to finally post about how my second year journalism degree students approached the ‘interactive’ element of their portfolio way back in May (yes, everything they do is interactive, but bear with me).

For the first time I gave them an open brief in terms of what they did interactively (in previous years I asked them to produce Flash interactives). Having been taught how to create everything from audio slideshows and image maps to multimedia interactives, Google Maps and Yahoo! Pipes mashups, I was curious to see what they would pick. Would they all plump for the same option?

Apparently not. In fact, the results are the most diverse and downright fascinating I’ve ever seen from an online journalism class.

Here’s what they did:

• Kasper Sorensen created

  a map of water issues stories in Europe

• Ashley Snape created a

  map of Environmental Activity in Australasia

• Natalie Chillington created a photo slideshow of the top ten endangered animals
• Alice Fanning not only created a

  map of UK environmental protests but also a Yahoo! Pipe mashup of eco news.

• Emma Foster created an audio slideshow – ‘Tescopoly’ – and a map of eco-businesses in the UK
• Hayley Smith created a Yahoo! Pipes mashup of environmental technology news, alerts and photos
• Kat Higgs created an audio slideshow about mistreatment of animals
• Stephanie Grant created an audio slideshow to mark African Liberation Day
• Stephen Nunes set up a YouTube channel and posted 3 mini-videos looking at how environmentally friendly his accommodation was; he also included links to full audio on many of his stories for the website
• Tuuli Platner stole the show with her YouTube video song promoting the site and her reporter blog. Journalists are becoming brands, and Tuuli has understood that brilliantly.

In addition the class submitted stories from their blogs and from the Environmental News Online website; their Twitter tweets and their bookmarks; their rss reader subscriptions and their comments on other blogs.

You could say I’m quite demanding like that.

What was interesting to see was students commenting on each others’ blogs about what they were planning to do – particularly useful when students were reporting on areas that crossed disciplines and other correspondents could see additional leads or angles.

In other words, the blogs became a team management and communication tool, as well as a publishing one.

Many students tried a number of things – a Pipes mashup; an audio slideshow – before settling on what they eventually submitted. They bookmarked, they twittered, and they blogged. They linked. They commented – one noted how her blog hits spiked after she posted a comment on another blog.

I was pretty pleased, all in. The vast scope of possibilities in online journalism is impossible to teach in ten weeks, so I tried to focus on broader issues: experimentation; possibilities. That gave them scope to latch onto what stimulated them, or what the story suited. I only hope they take it further as they begin their final year…

Recently my attention has been drawn to the Dutch news website www.en.nl. Wilbert Baan, interaction designer for the Dutch newspaper de Volkskrant, told me he wants to see “what we can do with news, social networks, wikis and more.

“I think you might like the experiment we are doing,” he wrote.

And bloody hell was he right.

The first thing that strikes you about the site is the bar chart across the top of the page, replacing the traditional masthead. This is a newsriver:

Down the outside column is a list of articles from the past hour:

That’s culture shift number 1.

At the bottom of the page you will find recent images, social bookmark sites, most commented articles from the past 24 hours, most important and most viewed.

Culture shift number 2 is the list of incoming links to this article – something built into the very fabric of blogs (pingback) but so far either anathema to mainstream publishers (“send our readers elsewhere?”), or difficult with current content management systems.

And with one simple move the site demonstrates it’s part of the conversation.

The ‘most important’ list is also worth looking at. How did they decide what was “most important?” I asked.

“We are using around ten variables to decide what’s important news. The variables we’re using right now are pageviews, visits from external websites, unique referrers to an article, comments, votes (4 options) and the press agency urgency variable (3 options; normal, high, very high).

“By showing it next to the most viewed we can easily see how it works and adjust the settings to make it better. It’s not perfect yet, but it already works remarkably well.

“We could extend this even further (tags, edits, tag removals) or skip some. All the variables are connected to points, we can set a default amount of points to a variable and define or redefine the value for the website.

“We also made a tag sniffer at http://www.skitch.com/wilbertbaan/8733/en-tag-sniffing – it scans the text on certain names and auto tags the article.”

Wilbert’s next step is building a community that can contribute to make this website better with ideas or criticism. The newspaper is already conducting a conversation with readers on a NING social network where users can contribute new ideas and discuss the website (in Dutch), but clearly this is just the start.

“For example we could connect a popular social network to the website and use what your network reads to alter the presentation of the news. Or make section pages, or a frontpage?

And all this is possible because of a Holovaty-esque focus on the power of databases.

“The most important object is the database,” he writes on his blog. “We designed the database from a view that almost everything is possible with the data. We store a lot of information that might be valuable in the future. This allows us to experiment freely with the design and think up new features. The database is the most valuable asset of a news organization.”
And this means they can do “Almost everything. We can make mash-ups, feeds, aggregated pages. Hook in to social networks, extend the wiki functionality, and more. Technically everything is possible.”

Keep an eye on this one.

UPDATE: Wilbert writes: “We have added feeds for every tag, latest news and breaking news. We have also added a personal feed that can be created by selecting the tags you like or don’t like. Very rudimentary, but it is a first experiment with personalization (My feed: http://en.nl/en/my_rss.php?editorId=3) and you can take it anywhere you want.

“With these feeds we are encouraging developers to experiment with news sorting and make their own interface or mash-up.”

Read more posts about future newspapers here