I’ve been writing about Yahoo! Pipes for some time, and am consistently surprised that there aren’t more books on the tool. Pipes Tutorial – an ebook currently priced at $14.95 – is clearly aiming to address that gap.

The book has a simple structure: it is, in a nutshell, a tour around the various ‘modules’ that you combine to make a pipe.

Some of these will pull information from elsewhere – RSS feeds, CSV spreadsheets, Flickr, Google Base, Yahoo! Local and Yahoo! Search, or entire webpages.

Some allow the user to input something themselves – for example, a search phrase, or a number to limit the type of results given.

And others do things with all the above – combining them, splitting them, filtering, converting, translating, counting, truncating, and so on.

When combined, this makes for some powerful possibilities – unfortunately, its one-dimensional structure means that this book doesn’t show enough of them.

Modules in isolation

While the book offers a good introduction into the functionality of the various parts of Yahoo! Pipes, it rarely demonstrates how those can be combined. Typically, tutorial books will take you through a project that utilises the power of the tools covered, but Pipes Tutorial lacks this vital element. Sometimes modules will be combined in the book but this is mainly done because that is the only way to show how a single module works, rather than for any broader pedagogical objective.

At other times a module is explained in isolation and it is not explained how the results might actually be used. The Fetch Page module, for example – which is extremely useful for scraping content from a webpage – is explained without reference to how to publish the results, only a passing mention that the reader will have to use ‘other modules’ to assign data to types, and that Regex will be needed to clean it up.

Regex itself – possibly one of the most useful parts of Yahoo! Pipes – is cursorily tackled, and the reader pointed to resources elsewhere. The same applies to YQL – the language that allows you to interrogate data sources. Likewise, the Web Service module which allows you to connect with an API, isn’t illustrated with any practical guidance on how to use it.

The book makes no mention of the ability to clone pipes published by others on Yahoo! Pipes, and misses a big opportunity to provide links to working pipes that the user can clone and play with themselves – or indeed any online support that I can see other than a blog that currently has 2 instructional posts.

Despite all the above omissions, the lack of similar books mean this is still a useful resource for aspiring data journalists. It provides an insight into the possibilities of Pipes, even if it doesn’t quite take you through how to exploit those.

PS: If you’ve read any other books on Yahoo! Pipes (including this one) let me know whether they’re any use.

PHP Freelancer

This is a draft from a book chapter on data journalism (part 1 looks at finding data; part 2 at interrogating data; part 3 at visualisation, and 4 at visualisation tools). I’d really appreciate any additions or comments you can make – particularly around tips and tools.

UPDATE: It has now been published in The Online Journalism Handbook.

Mashing data

Wikipedia defines a mashup particularly succinctly, as “a web page or application that uses or combines data or functionality from two or many more external sources to create a new service.” Those sources may be online spreadsheets or tables; maps; RSS feeds (which could be anything from Twitter tweets, blog posts or news articles to images, video, audio or search results); or anything else which is structured enough to ‘match’ against another source.

This ‘match’ is typically what makes a mashup. It might be matching a city mentioned in a news article against the same city in a map; or it may be matching the name of an author with that same name in the tags of a photo; or matching the search results for ‘earthquake’ from a number of different sources. The results can be useful to you as a journalist, to the user, or both.

Why make a mashup?

Mashups can be particularly useful in providing live coverage of a particular event or ongoing issue – mashing images from a protest march, for example, against a map. Creating a mashup online is not too dissimilar from how, in broadcast journalism, you might set up cameras at key points around a physical location in anticipation of an event from which you will later ‘pull’ live feeds: in a mashup you are effectively doing exactly the same thing – only in a virtual space rather than a physical one. So, instead of setting up a feed at the corner of an important junction, you might decide to pull a feed from Flickr of any images that are tagged with the words ‘protest’ and ‘anti-fascist’.

Some web developers have built entire sites that are mashups. Twazzup (twazzup.com) for example, will show you a mix of Twitter tweets, images from Flickr, news updates and websites – all based on the search term you enter. And Friendfeed (friendfeed.com) pulls in data that you and your social circle post to a range of social networking sites, and displays them in one place.

Mashups also provide a different way for users to interact with content – either by choosing how to navigate (for instance by using a map), or by inviting them to input something (for instance, a search term, or selecting a point on a slider). The Super Tuesday YouTube/Google Maps mashup, for instance, provided an at-a-glance overview of what election-related videos were being uploaded where across the US.

Finally, mashups offer an opportunity for juxtaposing different datasets to provide fresh, sometimes ongoing, insights. The MySociety/Channel 4 project Mapumental, for example, combines house price data with travel information and data on the ‘scenicness’ of different locations to provide an interactive map of a location which the user can interrogate based on their individual preferences.

Mashup tools

Like so many aspects of online journalism, the ease with which you can create a mashup has increased significantly in recent years. An increase in the number and power of online tools, combined with the increasing ‘mashability’ of websites and data, mean that journalists can now create a basic mashup through the simple procedures of drag-and-drop or copy-and-paste.

A simple RSS mashup, which combines the feeds from a number of different sources into one, for example, can now be created using an RSS aggregator such as xFruits (xfruits.com) or Jumbra (jumbra.com).

Likewise, you can mix two maps together using the website MapTube (maptube.org) which also contains a number of maps for you to play with.

And if you want to mix two sources of data into one visualisation the site DataMasher (datamasher.org) will let you do that – although you’ll have to make do with the US data that the site provides. Google Public Data Explorer (google.com/publicdata) is a similar tool which allows you to play with global data.

But perhaps the most useful tool for news mashups is Yahoo! Pipes (pipes.yahoo.com).

Yahoo! Pipes allows you to choose a source of data – it might be an RSS feed, an online spreadsheet or something that the user will input – and do a variety of things with it. Here are just some of the basic things you might do:

• Add it to other sources
• Combine it with other sources – for instance, matching images to text
• Filter it
• Count it
• Annotate it
• Translate it
• Create a gallery from the results
• Place results on a map

You could write a whole book on how to use Yahoo! Pipes – indeed, people have – so we will not cover the practicalities of using all of those features here. There are also dozens of websites and help files devoted to the site (which you should explore). Below, however, is a short tutorial to introduce you to the website and how it works – this is a good way to understand how basic mashups work, and how easily they can be created.

Mashups and APIs

Although there are a number of easy-to-use mashup creators listed above, really impressive mashups tend to be written by people with knowledge of programming languages, and use APIs. APIs (Application Programming Interface) allow websites to interact with other websites. The launch of the Google Maps API in 2005, for example, has been described as a ‘huge tipping point’ in mashup history (Duvander, 2008) as it allowed web developers to ‘mash’ countless other sources of data with maps. Since then it has become commonplace for new websites, particularly in the social media arena, to launch their own APIs in order to allow web developers to do interesting things with their feeds and data – not just mashups, but applications and services too.

If you want to develop a particularly ambitious mashup it is likely that you will need to teach yourself some programming skills, and familiarise yourself with some APIs (the APIs of Twitter, Google Maps and Flickr are good places to start).

Box-out: Anatomy of a feed

The image below from ReadWriteWeb shows the code behind a simple Twitter update. It includes information about the author, their location, whether the update was a reply to someone else, what time and where it was created, and lots more besides. Each of these values can be used by a mashup in various ways – for example, you might match the author of this tweet with the author of a blog or image; you might match its time against other things being published at that moment; or you might use their location to plot this update on a map.

While the code can be intimidating, you do not need to understand programming in order to be able to do things with it. Of course, it will help if you do…

PHP Freelancer

This is a draft from a book chapter on data journalism (the first, on gathering data, is here). I’d really appreciate any additions or comments you can make – particularly around ways of spotting stories in data, and mistakes to avoid.

UPDATE: It has now been published in The Online Journalism Handbook.

“One of the most important (and least technical) skills in understanding data is asking good questions. An appropriate question shares an interest you have in the data, tries to convey it to others, and is curiosity-oriented rather than math-oriented. Visualizing data is just like any other type of communication: success is defined by your audience’s ability to pick up on, and be excited about, your insight.” (Fry, 2008, p4)

Once you have the data you need to see if there is a story buried within it. The great advantage of computer processing is that it makes it easier to sort, filter, compare and search information in different ways to get to the heart of what – if anything – it reveals.

The first stage in this process, then, is making sure the data is in the right format to be interrogated. Quite often this will be a spreadsheet or CSV (comma-separated values) file. If your information is in a PDF you will not be able to do a great deal with it other than re-type the values into a new spreadsheet (making sure to check you have not made any errors). A Word or Powerpoint document is likely to require the same work.

If the information is already online you can sometimes ‘scrape’ it – that is, automatically copy the relevant information into a separate document. How easy this is to do depends on how structured the information is. A table in a Wikipedia entry, for example, can be ‘scraped’ into a Google spreadsheet relatively easily (Tony Hirst gives instructions on how to do this at http://ouseful.wordpress.com/2008/10/14/data-scraping-wikipedia-with-google-spreadsheets/) and an online CSV file and certain other structured data can be scraped with Yahoo! Pipes (see below for more on using Yahoo! Pipes) but most scraping will involve programming (the online tool ScraperWiki provides one environment to help you do this).

Insert: Cleaning up data

Whether you have been given data, had to scrape it, or copied it manually, you will probably need to clean it up. All sorts of things can ‘dirty’ your data, from misspellings and variations in spelling, to odd punctuation, mixtures of numbers and letters, unnecessary columns or rows, and more. Computers, for example, will see ‘New Town’, ‘Newtown’ and ‘newtown’ as three separate towns when they may be one.

This can cause problems later on when analysing your data – for example, calculations not working or results not being accurate.

Some tips for cleaning your data include:

• Use a spellchecker to check for misspellings. You will probably have to add some words to the computer’s dictionary.
• Use ‘find and replace’ (normally in the Edit menu) to remove double-spaces and other common punctuation errors. Alternatively, if you are in Excel you can create a new column and use the =TRIM() function, which will copy the contents of the cell in the brackets and remove any spaces.
• Remove duplicate entries – if you are using Excel there are a few ways to do this under the Data tab – search for duplicates in Help.

For more tips on Excel specifically see this guide: http://office.microsoft.com/en-us/excel/HA102218401033.aspx

For cleaning up very large sets of data you might want to use a data cleaning tool like Google Refine.

Spotting the story

Once your data is cleaned you can start to look for anything newsworthy in it. There are some obvious places to start: if you are dealing with numbers, for example, you can work out what the ‘average’ is (the average bonus paid to council employees, for example). Similarly, you might look for the term which appears most often (e.g. the most common reason given for arresting terrorist suspects).

However, Kaiser Fung, a statistician whose blog Junk Charts is essential reading on the field, notes the dangers in lazily reaching for the average when you want to make an editorial point:

“Averaging stamps out diversity, reducing anything to its simplest terms. In so doing, we run the risk of oversimplifying, of forgetting the variations around the average. Hitching one’s attention to these variations rather than the average is a sure sign of maturity in statistical thinking. One can, in fact, define statistics as the study of the nature of variability. How much do things change? How large are these variations? What causes them?” (Fung, 2010, p4)

So while averages and modes (a mode is the number or term which appears most often) can be interesting discoveries, they should most often be used as a starting point for more illuminating investigation – normally involving leaving your computer to make phonecalls and speak to sources.

If you are looking at data over time, you can look to see what has increased over that period, or decreased – or disappeared.

But you will need to gather further data to provide context to your figures. If, for example, more council staff are receiving bonuses, is that simply because more staff have been employed? How much is spent on wages, and how do your figures compare? If you are comparing one city with another, understand how their populations differ – not just in aggregate, but in relevant details such as age, ethnicity, life expectancy, etc. You will need to know where to access basic statistics like these – the National Statistics website is often a good place to start.

Sometimes a change in the way data is gathered or categorised can produce a dramatic change in the data itself. In one example, designer Adrian Short obtained information (via an FOI request) on parking tickets from Transport for London that showed the numbers of tickets issued against a particular offence plummeted from around 8,000 to 8 in the space of one month (Arthur, 2009). Had people suddenly stopped committing that parking offence, or was there another explanation? A quick phonecall to Transport for London revealed that traffic wardens were issued with new handsets around the same time. Guardian journalist Charles Arthur hypothesised:

“Could it be that s46 [another offence which had a steep rise at the same time] is the default on the screen to issue a new ticket, and that wardens don’t bother to change it? Whatever it is, there’s a serious problem for TfL if those aren’t all s46 offences which have been ticketed since August 2006. Because if the ticket isn’t written out to the correct offence, then the fine isn’t payable. Theoretically, TfL might have to pay back millions in traffic fines for people who have been ticketed for s46 offences when they were actually committing s25 or s30 offences.”

This particular story came about at least in part because that information was easy to visualise.

The next section covers visualisation. In the meantime, once again I’d really appreciate any additions or comments – particularly around ways of spotting stories in data, and mistakes to avoid.

PHP Freelancer

This was the final session in my undergraduate Online Journalism module (the other classes can be found here), taught last May. It’s a relatively brief presentation, just covering some of the possibilities of mashups and RSS, and some tools. The majority of the class is taken up with students using Yahoo! Pipes to aggregate a number of feeds.

I didn’t know how students would cope with Yahoo! Pipes but, surprisingly, every one completed the task.

As a side note, this year I kicked off the module with students setting up Twitter, Delicious and Google Reader – and synchronising them, so the RSS feed from one could update another (e.g. bookmarks being published to Twitter). This seems to have built a stronger understanding of RSS in the group, which they are able to apply elsewhere (they also have widgets on their blogs pulling the RSS feeds from Twitter & Delicious; and their profile page on the news website – built by Kasper Sorensen – pulls the latest updates from their Twitter, Delicious and blog feeds).

PHP Freelancer

Here’s another tutorial on the mashup platform Yahoo! Pipes, showing how you can use it to create a meta-search that will push any search term by the user through a number of search engines, and present you with a combined result (and RSS feed). A finished version of the pipe can be seen here.

This tutorial builds on a previous post I published on how to create basic mashups with Yahoo! Pipes. If you haven’t any knowledge of Pipes you should probably read that first.

How to create a custom meta-search in Yahoo! Pipes

First, you obviously need to log in to Yahoo! Pipes, and click on Create a Pipe. You’ll be taken to the Pipe editing interface: on the left will be a menu with a series of sections (User Input, Url, Operators, etc.) to choose modules from. In the centre will be the canvas where you create your pipe – and at the bottom a ‘Debugger’ area where you can see the results of any particular part of your pipe.

In the area on the left, under the ‘User Input’ section, click on the ‘Text Input‘ module and drag it onto the canvas (or you can click on the + sign for it to be placed for you).

In the box marked ‘Prompt’ type the instruction text for users of the pipe, e.g. ‘What do you want to search for?’. If there’s a default search you want to have appear in the search box to begin with, enter it in the box marked ‘Default’.Under the ‘Url’ section, click on ‘URL Builder‘ and drag it onto the canvas.

Go to a search engine that offers RSS feeds for searches, e.g. Google News. Do a test search (e.g. ‘fire’) and copy the address of the RSS feed for those search results (on Google News it will be at the foot of the page)

In the ‘URL Builder’ module paste the address of that RSS feed into the box marked ‘Base’

You’ll notice that the rest of the module now expands to include all of the parameters that the URL includes. The main one we are interested in is the one that relates to our search (you can tell which one it is as it should have the topic of your search in the right hand column) – in the case of Google News, this is ‘q’.

Drag a wire from the circle at the bottom of the Text Input module to the circle to the right of that search term (e.g. ‘fire’). It should turn grey and you should now be able to see a wire joining the two modules. Your search term should now be replaced by ‘text [wired]‘.

Now drag a third module onto your canvas: this time it’s under the ‘Sources’ heading, and the module is called ‘Fetch Feed‘.

This time drag a wire from the circle at the bottom of ‘URL Builder‘ to the circle on the right of the ‘Fetch Feed‘ module. Again, it should turn grey and the previously empty box should now read ‘text [wired]‘.

Drag another ‘URL Builder‘ module onto the canvas and repeat the process detailed above for a separate search engine. Connect your ‘Text Input‘ module to it in the same way. You may have to delete some of the parameters to make these work. A blog search on IceRocket, for example, includes the parameter ‘tab’. Deleting this (with the ‘-’ button next to it) appears to make the search work in Pipes. If you know of any similar tweaks, please let me know.

In the ‘Fetch Feed’ module click on the ‘+’ icon to add a new, empty, box. Click on the circle at the bottom of your second ‘URL Builder‘ module to the circle to the right of that empty box. Again, it should turn grey and you should now be able to see a wire joining the two modules and the box should now read ‘text [wired]‘.

You can repeat these steps for further search engines.

Once you’ve finished, from the ‘Operators’ heading on the left, drag a ‘Union‘ module onto the canvas. You need this to combine your feeds.

To combine them, drag a wire from the bottom of all ‘Fetch Feed‘ modules to separate circles at the top of ‘Union‘ at the bottom of the canvas. Unlike the other pipes this will be blue.

Finally, drag a pipe from the bottom of ‘Union‘ to the top of ‘Pipe Output‘ at the bottom of the canvas.

That is it. Save your pipe and click on ‘Back to My Pipes’ towards the top centre of the screen and you should see your pipe listed. Click on that pipe to run it – you should see a text entry box towards the top where you can enter your search to change the results. If you want to embed this somewhere, click on ‘Get as a Badge’ to get the code.

However, before you do that you’ll notice that results will be clustered by search engine, and also may contain duplicate results. You’ll need to click on ‘Edit Source’ and add a couple more modules to tweak your pipe and solve the problem.

The modules you will need to use are the Unique module (in the Operators section) to filter out duplicate results (probably place it between Union and Pipe Output - drag wires to remove them), and the Sort module (also in the Operators section) to order results by date (otherwise they will cluster by search engine).

I’m by no means any expert on this, but put this out there for others to build on, comment on and correct.

Many thanks to Paul Daniel who showed me much of this as part of an open MA Online Journalism session and whose blog is a great resource on Pipes.

PHP Freelancer

I was recently given a task where my job was to create a calendar holding around 50 events. Each event also needed to be mapped, and have a corresponding blog post.

Mapping calendar entries made me think, if this could be used for other stuff than simply putting events on a map, – which is quite useful in it’s own way. I thought it would be cool if you could create an interactive map-timeline, controlled dynamically by a (shared)calendar.

Yahoo Pipes by default uses Yahoo Maps, which is great when it comes to narratives. As you can see from the map below (If you don’t see it, click here), each entry has a little arrow that let’s you navigate from marker to marker in a specific order. Each marker also has a number indicating it’s place in a sequence. This is nothing more than entries in a Google Calender with time/date stamps, geo info and a description, mapped automatically using Yahoo Pipes.

{“pipe_id”:”ed13a198a2a83050dd4ace10d12eae16″,”_btype”:”map”,”pipe_params”:{“Curl”:”http://kaspersorensen.com/wp-content/uploads/files/icalyahoopipes.ics”}}

Here’s how you do it.

1. Create a Google Calendar

Simply go to your Google Calendar and create or import a new calendar. You can do this from the settings page under calendars.

2. Make it public

You need to make the calendar public, otherwise Yahoo Pipes won’t have access to it. You can do this while you create it, or afterwards by ticking the box ‘Make this calendar public’ from the sharing settings on your specific calendar. To access the settings for a specific calendar, you click the little arrow in the box on the left hand side that contains your calendars (My Calendars).

3. Create events

Now you simply start adding events to your calendar. Specify what happened, where it happened, when and add the description. You don’t have to add the entries chronologically, they will be sorted by date/time automatically.

4. Feed the iCal file to the Pipe

Go to your calendar settings page, not the general Calendar settings, but the settings for your specific calendar. You will see a section called ‘Calendar address’ with three buttons. Click the green ICAL button and copy the link that pops up. Now go to Mapping Google Calendar Events Pipe and paste it into the ‘Calendar iCal URL’ field and hit ‘Run Pipe’. – Your events are now mapped.

5. Embed on your website

To embed the timeline/map on your website, simply select ‘Get as badge’ just above the map. This will allow you to insert it on your blog or website.

I’m sure there are ways to make this more stable. So if you know how to optimize the pipe, please feel free to do so and let me know.

As Google Maps is already a part of Google Calendar, you would think that there was a nifty way to quickly put a whole calendar on a map, but no. And after failing to use what looked like a saviour, I bumped into a post by Tony Hurst on how to display Google Calendar events on a Google Map. Unfortunately it turns out that the XML feed Tony uses, only parses the 25 most recent calendar entries.

Google Calendar releases their event-entries in iCal format which contains all events. And with a little customization of Tony’s pipe, I managed to come up with a way to map all events from a calendar.

I think this could be potentially useful for developing stories, especially if you can collaborate on the calendar. You end up with data that can be used for nearly anything, not just maps. And if locations aren’t relevant for the story, you could simply take your iCal file and make a normal timeline.

PHP Freelancer

How widely is Yahoo! Pipes used in newsrooms? Could it be better used? Is there a way us Piping Journos could exchange best practice, ideas, and support?

I’d like to bring journalists using Yahoo! Pipes together, so I’ve created a little Twitter group. Hope you can join, and we can find some ways to help each other out.

If you’re not using Pipes or would still like a primer, here’s one I prepared earlier.

PHP Freelancer

As a new semester begins it seems a good time to finally post about how my second year journalism degree students approached the ‘interactive’ element of their portfolio way back in May (yes, everything they do is interactive, but bear with me).

For the first time I gave them an open brief in terms of what they did interactively (in previous years I asked them to produce Flash interactives). Having been taught how to create everything from audio slideshows and image maps to multimedia interactives, Google Maps and Yahoo! Pipes mashups, I was curious to see what they would pick. Would they all plump for the same option?

Apparently not. In fact, the results are the most diverse and downright fascinating I’ve ever seen from an online journalism class.

Here’s what they did:

• Kasper Sorensen created

  a map of water issues stories in Europe

• Ashley Snape created a

  map of Environmental Activity in Australasia

• Natalie Chillington created a photo slideshow of the top ten endangered animals
• Alice Fanning not only created a

  map of UK environmental protests but also a Yahoo! Pipe mashup of eco news.

• Emma Foster created an audio slideshow – ‘Tescopoly’ – and a map of eco-businesses in the UK
• Hayley Smith created a Yahoo! Pipes mashup of environmental technology news, alerts and photos
• Kat Higgs created an audio slideshow about mistreatment of animals
• Stephanie Grant created an audio slideshow to mark African Liberation Day
• Stephen Nunes set up a YouTube channel and posted 3 mini-videos looking at how environmentally friendly his accommodation was; he also included links to full audio on many of his stories for the website
• Tuuli Platner stole the show with her YouTube video song promoting the site and her reporter blog. Journalists are becoming brands, and Tuuli has understood that brilliantly.

In addition the class submitted stories from their blogs and from the Environmental News Online website; their Twitter tweets and their bookmarks; their rss reader subscriptions and their comments on other blogs.

You could say I’m quite demanding like that.

What was interesting to see was students commenting on each others’ blogs about what they were planning to do – particularly useful when students were reporting on areas that crossed disciplines and other correspondents could see additional leads or angles.

In other words, the blogs became a team management and communication tool, as well as a publishing one.

Many students tried a number of things – a Pipes mashup; an audio slideshow – before settling on what they eventually submitted. They bookmarked, they twittered, and they blogged. They linked. They commented – one noted how her blog hits spiked after she posted a comment on another blog.

I was pretty pleased, all in. The vast scope of possibilities in online journalism is impossible to teach in ten weeks, so I tried to focus on broader issues: experimentation; possibilities. That gave them scope to latch onto what stimulated them, or what the story suited. I only hope they take it further as they begin their final year…

PHP Freelancer

It’s nice when you host some training and something of use comes out of it. Alison Gow, who recently attended my Social Media for Breaking News training, has used it to build a Yahoo Pipe. It “filters all the latest news, photos and quality blog posts from the world of Fashion for the Girls Behaving Stylishly team to place on their blog as a widget, and to help them spot trends quickly without having to trawl the web.”

Her post is worth reading if you’re interested in doing it yourself, littered as it is with useful red arrow-laden screengrabs.

PHP Freelancer