Tag Archives: Yahoo! Pipes

The drawn out death of Yahoo! Pipes and the steady rise of IFTTT

Yahoo logo

So Yahoo! Pipes will be ‘retired’ in a couple of months. It was a seminal tool for its time, helping people like me demonstrate and explore the potential of RSS, APIs and automation without having to become programmers first.

But times move on. I’d stopped using Pipes years ago: I’d caught the programming bug and wanted to do more; and while Pipes fell into disrepair dozens of other tools were springing up showing how similar things could be done.

And top of the crop has been IFTTT.

IFTTT (If This Then That) was Pipes without the pipes. And although it lacked some of the functionality and control of Pipes (searching multiple sites all at once was a particular favourite), it more than made up for that with simplicity of interface and an ever-growing list of services it supported.

The proposition was a clear one: if you’re having to anything more than once, get IFTTT to do it for you. Cross-posting from WordPress to Facebook? Automate it. Saving all the tweets using a particular hashtag? Automate it.

When I first started using IFTTT there were around a dozen ‘channels’ you could connect: mainly social media and blogging services like Twitter, Facebook and WordPress – and of course, any RSS feed.

Now there are dozens and dozens of channels: not just newer social media platforms including Instagram and Pinterest, but email and phonecall triggers, iOS reminders and notifications, wearable tech like Nike+, Android Wear and Fitbit, connected home devices like WeMo light switches and Honeywell thermostats, connected car devices like Dash and Mojio, and triggers from your mobile phone including its location and text messages.

As the world has become more connected, so IFTTT has grown in potential. It is also one of the clearest demonstrations of how ‘visible’ we are when we connect to the web: sharing everything from the temperature of our house and the speed of our car to the position of our phone, and our partner’s phone.

Like Pipes you can set all sorts of filters, and you can browse, search and adapt ‘recipes’ by other users, including ‘notify me of a zombie apocalypse‘.

Of course, it’s a gateway drug to programming. By demonstrating one of the most basic concepts in programming – the ‘if’ – and how much time you can save with just that, it makes you want to do more. And it opens you up to the world of possibilities that comes with working with developers.

You may never have used Pipes, but it’s probably a good time to play with IFTTT.

Tony Hirst has written about the announcement and how that reflects changes in the culture of web companies. Adam Tinworth has pointed out that many organisations relying on pipes created by departed staff may find that things stop working.Ghacks Technology News outlines alternatives ClickScriptQuadrigram, and Superpipes

A case study in following a field online: setting up feeds on CCGs

Over at Help Me Investigate Health I’ve just published a bunch of 20 places to keep up to date with clinical commissioning. It’s an example of something I’ve written about previously – setting up an online network infrastructure as a journalist. And below, I explain the process behind it:

Following CCGs across local newspapers and blogs

If you’re going to start scrutinising a field, it’s very useful to be kept up to date with developments in that field:

  • Concerns raised in one local newspaper may be checked elsewhere;
  • Specialist magazines may provide guides to jargon or processes that helps save you a lot of time;
  • Politicians might raise concerns and get answers;
  • And expert bloggers can provide leads and questions that you might want to follow up.

Rather than checking a list of websites on the off chance that one has been updated, a much more efficient way to keep up to date on what’s happening is to use a free RSS readerContinue reading

Data for journalists: understanding XML and RSS

If you are working with data chances are that sooner or later you will come across XML – or if you don’t, then, well, you should do. Really.

There are some very useful resources in XML format – and in RSS, which is based on XML – from ongoing feeds and static reference files to XML that is provided in response to a question that you ask. All of that is for future posts – this post attempts to explain how XML is relevant to journalism, and how it is made up.

What is XML?

XML is a language which is used for describing information, which makes it particularly relevant to journalists – especially when it comes to interrogating large sets of data.

If you wanted to know how many doctors were privately educated, or what the most common score was in the Premiership last season, or which documents were authored by a particular civil servant, then XML may be useful to you. Continue reading

Review: Yahoo! Pipes tutorial ebook

Pipes Tutorial ebook

I’ve been writing about Yahoo! Pipes for some time, and am consistently surprised that there aren’t more books on the tool. Pipes Tutorial – an ebook currently priced at $14.95 – is clearly aiming to address that gap.

The book has a simple structure: it is, in a nutshell, a tour around the various ‘modules’ that you combine to make a pipe.

Some of these will pull information from elsewhere – RSS feeds, CSV spreadsheets, Flickr, Google Base, Yahoo! Local and Yahoo! Search, or entire webpages.

Some allow the user to input something themselves – for example, a search phrase, or a number to limit the type of results given.

And others do things with all the above – combining them, splitting them, filtering, converting, translating, counting, truncating, and so on.

When combined, this makes for some powerful possibilities – unfortunately, its one-dimensional structure means that this book doesn’t show enough of them.

Modules in isolation

While the book offers a good introduction into the functionality of the various parts of Yahoo! Pipes, it rarely demonstrates how those can be combined. Typically, tutorial books will take you through a project that utilises the power of the tools covered, but Pipes Tutorial lacks this vital element. Sometimes modules will be combined in the book but this is mainly done because that is the only way to show how a single module works, rather than for any broader pedagogical objective.

At other times a module is explained in isolation and it is not explained how the results might actually be used. The Fetch Page module, for example – which is extremely useful for scraping content from a webpage – is explained without reference to how to publish the results, only a passing mention that the reader will have to use ‘other modules’ to assign data to types, and that Regex will be needed to clean it up.

Continue reading

Data journalism pt5: Mashing data (comments wanted)

This is a draft from a book chapter on data journalism (part 1 looks at finding data; part 2 at interrogating datapart 3 at visualisation, and 4 at visualisation tools). I’d really appreciate any additions or comments you can make – particularly around tips and tools.

UPDATE: It has now been published in The Online Journalism Handbook.

Mashing data

Wikipedia defines a mashup particularly succinctly, as “a web page or application that uses or combines data or functionality from two or many more external sources to create a new service.” Those sources may be online spreadsheets or tables; maps; RSS feeds (which could be anything from Twitter tweets, blog posts or news articles to images, video, audio or search results); or anything else which is structured enough to ‘match’ against another source.

This ‘match’ is typically what makes a mashup. It might be matching a city mentioned in a news article against the same city in a map; or it may be matching the name of an author with that same name in the tags of a photo; or matching the search results for ‘earthquake’ from a number of different sources. The results can be useful to you as a journalist, to the user, or both.

Why make a mashup?

Mashups can be particularly useful in providing live coverage of a particular event or ongoing issue – mashing images from a protest march, for example, against a map. Creating a mashup online is not too dissimilar from how, in broadcast journalism, you might set up cameras at key points around a physical location in anticipation of an event from which you will later ‘pull’ live feeds: in a mashup you are effectively doing exactly the same thing – only in a virtual space rather than a physical one. So, instead of setting up a feed at the corner of an important junction, you might decide to pull a feed from Flickr of any images that are tagged with the words ‘protest’ and ‘anti-fascist’. Continue reading

Data journalism pt2: Interrogating data

This is a draft from a book chapter on data journalism (the first, on gathering data, is here). I’d really appreciate any additions or comments you can make – particularly around ways of spotting stories in data, and mistakes to avoid.

UPDATE: It has now been published in The Online Journalism Handbook.

“One of the most important (and least technical) skills in understanding data is asking good questions. An appropriate question shares an interest you have in the data, tries to convey it to others, and is curiosity-oriented rather than math-oriented. Visualizing data is just like any other type of communication: success is defined by your audience’s ability to pick up on, and be excited about, your insight.” (Fry, 2008, p4)

Once you have the data you need to see if there is a story buried within it. The great advantage of computer processing is that it makes it easier to sort, filter, compare and search information in different ways to get to the heart of what – if anything – it reveals. Continue reading

Online Journalism lesson #10: RSS and mashups

This was the final session in my undergraduate Online Journalism module (the other classes can be found here), taught last May. It’s a relatively brief presentation, just covering some of the possibilities of mashups and RSS, and some tools. The majority of the class is taken up with students using Yahoo! Pipes to aggregate a number of feeds.

I didn’t know how students would cope with Yahoo! Pipes but, surprisingly, every one completed the task.

As a side note, this year I kicked off the module with students setting up Twitter, Delicious and Google Reader – and synchronising them, so the RSS feed from one could update another (e.g. bookmarks being published to Twitter). This seems to have built a stronger understanding of RSS in the group, which they are able to apply elsewhere (they also have widgets on their blogs pulling the RSS feeds from Twitter & Delicious; and their profile page on the news website – built by Kasper Sorensen – pulls the latest updates from their Twitter, Delicious and blog feeds).