Tag Archives: Yahoo! Pipes

A case study in following a field online: setting up feeds on CCGs

Over at Help Me Investigate Health I’ve just published a bunch of 20 places to keep up to date with clinical commissioning. It’s an example of something I’ve written about previously – setting up an online network infrastructure as a journalist. And below, I explain the process behind it:

Following CCGs across local newspapers and blogs

If you’re going to start scrutinising a field, it’s very useful to be kept up to date with developments in that field:

  • Concerns raised in one local newspaper may be checked elsewhere;
  • Specialist magazines may provide guides to jargon or processes that helps save you a lot of time;
  • Politicians might raise concerns and get answers;
  • And expert bloggers can provide leads and questions that you might want to follow up.

Rather than checking a list of websites on the off chance that one has been updated, a much more efficient way to keep up to date on what’s happening is to use a free RSS readerContinue reading

Data for journalists: understanding XML and RSS

If you are working with data chances are that sooner or later you will come across XML – or if you don’t, then, well, you should do. Really.

There are some very useful resources in XML format – and in RSS, which is based on XML – from ongoing feeds and static reference files to XML that is provided in response to a question that you ask. All of that is for future posts – this post attempts to explain how XML is relevant to journalism, and how it is made up.

What is XML?

XML is a language which is used for describing information, which makes it particularly relevant to journalists – especially when it comes to interrogating large sets of data.

If you wanted to know how many doctors were privately educated, or what the most common score was in the Premiership last season, or which documents were authored by a particular civil servant, then XML may be useful to you. Continue reading

Review: Yahoo! Pipes tutorial ebook

Pipes Tutorial ebook

I’ve been writing about Yahoo! Pipes for some time, and am consistently surprised that there aren’t more books on the tool. Pipes Tutorial – an ebook currently priced at $14.95 – is clearly aiming to address that gap.

The book has a simple structure: it is, in a nutshell, a tour around the various ‘modules’ that you combine to make a pipe.

Some of these will pull information from elsewhere – RSS feeds, CSV spreadsheets, Flickr, Google Base, Yahoo! Local and Yahoo! Search, or entire webpages.

Some allow the user to input something themselves – for example, a search phrase, or a number to limit the type of results given.

And others do things with all the above – combining them, splitting them, filtering, converting, translating, counting, truncating, and so on.

When combined, this makes for some powerful possibilities – unfortunately, its one-dimensional structure means that this book doesn’t show enough of them.

Modules in isolation

While the book offers a good introduction into the functionality of the various parts of Yahoo! Pipes, it rarely demonstrates how those can be combined. Typically, tutorial books will take you through a project that utilises the power of the tools covered, but Pipes Tutorial lacks this vital element. Sometimes modules will be combined in the book but this is mainly done because that is the only way to show how a single module works, rather than for any broader pedagogical objective.

At other times a module is explained in isolation and it is not explained how the results might actually be used. The Fetch Page module, for example – which is extremely useful for scraping content from a webpage – is explained without reference to how to publish the results, only a passing mention that the reader will have to use ‘other modules’ to assign data to types, and that Regex will be needed to clean it up.

Continue reading

Data journalism pt5: Mashing data (comments wanted)

This is a draft from a book chapter on data journalism (part 1 looks at finding data; part 2 at interrogating datapart 3 at visualisation, and 4 at visualisation tools). I’d really appreciate any additions or comments you can make – particularly around tips and tools.

UPDATE: It has now been published in The Online Journalism Handbook.

Mashing data

Wikipedia defines a mashup particularly succinctly, as “a web page or application that uses or combines data or functionality from two or many more external sources to create a new service.” Those sources may be online spreadsheets or tables; maps; RSS feeds (which could be anything from Twitter tweets, blog posts or news articles to images, video, audio or search results); or anything else which is structured enough to ‘match’ against another source.

This ‘match’ is typically what makes a mashup. It might be matching a city mentioned in a news article against the same city in a map; or it may be matching the name of an author with that same name in the tags of a photo; or matching the search results for ‘earthquake’ from a number of different sources. The results can be useful to you as a journalist, to the user, or both.

Why make a mashup?

Mashups can be particularly useful in providing live coverage of a particular event or ongoing issue – mashing images from a protest march, for example, against a map. Creating a mashup online is not too dissimilar from how, in broadcast journalism, you might set up cameras at key points around a physical location in anticipation of an event from which you will later ‘pull’ live feeds: in a mashup you are effectively doing exactly the same thing – only in a virtual space rather than a physical one. So, instead of setting up a feed at the corner of an important junction, you might decide to pull a feed from Flickr of any images that are tagged with the words ‘protest’ and ‘anti-fascist’. Continue reading

Data journalism pt2: Interrogating data

This is a draft from a book chapter on data journalism (the first, on gathering data, is here). I’d really appreciate any additions or comments you can make – particularly around ways of spotting stories in data, and mistakes to avoid.

UPDATE: It has now been published in The Online Journalism Handbook.

“One of the most important (and least technical) skills in understanding data is asking good questions. An appropriate question shares an interest you have in the data, tries to convey it to others, and is curiosity-oriented rather than math-oriented. Visualizing data is just like any other type of communication: success is defined by your audience’s ability to pick up on, and be excited about, your insight.” (Fry, 2008, p4)

Once you have the data you need to see if there is a story buried within it. The great advantage of computer processing is that it makes it easier to sort, filter, compare and search information in different ways to get to the heart of what – if anything – it reveals. Continue reading

Online Journalism lesson #10: RSS and mashups

This was the final session in my undergraduate Online Journalism module (the other classes can be found here), taught last May. It’s a relatively brief presentation, just covering some of the possibilities of mashups and RSS, and some tools. The majority of the class is taken up with students using Yahoo! Pipes to aggregate a number of feeds.

I didn’t know how students would cope with Yahoo! Pipes but, surprisingly, every one completed the task.

As a side note, this year I kicked off the module with students setting up Twitter, Delicious and Google Reader – and synchronising them, so the RSS feed from one could update another (e.g. bookmarks being published to Twitter). This seems to have built a stronger understanding of RSS in the group, which they are able to apply elsewhere (they also have widgets on their blogs pulling the RSS feeds from Twitter & Delicious; and their profile page on the news website – built by Kasper Sorensen – pulls the latest updates from their Twitter, Delicious and blog feeds).

How to create a custom meta-search in Yahoo! Pipes

Here’s another tutorial on the mashup platform Yahoo! Pipes, showing how you can use it to create a meta-search that will push any search term by the user through a number of search engines, and present you with a combined result (and RSS feed). A finished version of the pipe can be seen here.

This tutorial builds on a previous post I published on how to create basic mashups with Yahoo! Pipes. If you haven’t any knowledge of Pipes you should probably read that first.

How to create a custom meta-search in Yahoo! Pipes

First, you obviously need to log in to Yahoo! Pipes, and click on Create a Pipe. You’ll be taken to the Pipe editing interface: on the left will be a menu with a series of sections (User Input, Url, Operators, etc.) to choose modules from. In the centre will be the canvas where you create your pipe – and at the bottom a ‘Debugger’ area where you can see the results of any particular part of your pipe.

In the area on the left, under the ‘User Input’ section, click on the ‘Text Input‘ module and drag it onto the canvas (or you can click on the + sign for it to be placed for you).

Select the Text Input module

Select the Text Input module

In the box marked ‘Prompt’ type the instruction text for users of the pipe, e.g. ‘What do you want to search for?’. If there’s a default search you want to have appear in the search box to begin with, enter it in the box marked ‘Default’. Continue reading