Tag Archives: Yahoo! Pipes

The drawn out death of Yahoo! Pipes and the steady rise of IFTTT

2 Replies

Yahoo logo

So Yahoo! Pipes will be ‘retired’ in a couple of months. It was a seminal tool for its time, helping people like me demonstrate and explore the potential of RSS, APIs and automation without having to become programmers first.

But times move on. I’d stopped using Pipes years ago: I’d caught the programming bug and wanted to do more; and while Pipes fell into disrepair dozens of other tools were springing up showing how similar things could be done.

And top of the crop has been IFTTT.

IFTTT (If This Then That) was Pipes without the pipes. And although it lacked some of the functionality and control of Pipes (searching multiple sites all at once was a particular favourite), it more than made up for that with simplicity of interface and an ever-growing list of services it supported.

The proposition was a clear one: if you’re having to anything more than once, get IFTTT to do it for you. Cross-posting from WordPress to Facebook? Automate it. Saving all the tweets using a particular hashtag? Automate it.

When I first started using IFTTT there were around a dozen ‘channels’ you could connect: mainly social media and blogging services like Twitter, Facebook and WordPress – and of course, any RSS feed.

Now there are dozens and dozens of channels: not just newer social media platforms including Instagram and Pinterest, but email and phonecall triggers, iOS reminders and notifications, wearable tech like Nike+, Android Wear and Fitbit, connected home devices like WeMo light switches and Honeywell thermostats, connected car devices like Dash and Mojio, and triggers from your mobile phone including its location and text messages.

As the world has become more connected, so IFTTT has grown in potential. It is also one of the clearest demonstrations of how ‘visible’ we are when we connect to the web: sharing everything from the temperature of our house and the speed of our car to the position of our phone, and our partner’s phone.

Like Pipes you can set all sorts of filters, and you can browse, search and adapt ‘recipes’ by other users, including ‘notify me of a zombie apocalypse‘.

Of course, it’s a gateway drug to programming. By demonstrating one of the most basic concepts in programming – the ‘if’ – and how much time you can save with just that, it makes you want to do more. And it opens you up to the world of possibilities that comes with working with developers.

You may never have used Pipes, but it’s probably a good time to play with IFTTT.

Tony Hirst has written about the announcement and how that reflects changes in the culture of web companies. Adam Tinworth has pointed out that many organisations relying on pipes created by departed staff may find that things stop working.Ghacks Technology News outlines alternatives ClickScript, Quadrigram, and Superpipes. 

A case study in following a field online: setting up feeds on CCGs

1 Reply

Over at Help Me Investigate Health I’ve just published a bunch of 20 places to keep up to date with clinical commissioning. It’s an example of something I’ve written about previously – setting up an online network infrastructure as a journalist. And below, I explain the process behind it:

Following CCGs across local newspapers and blogs

If you’re going to start scrutinising a field, it’s very useful to be kept up to date with developments in that field:

  • Concerns raised in one local newspaper may be checked elsewhere;
  • Specialist magazines may provide guides to jargon or processes that helps save you a lot of time;
  • Politicians might raise concerns and get answers;
  • And expert bloggers can provide leads and questions that you might want to follow up.

Rather than checking a list of websites on the off chance that one has been updated, a much more efficient way to keep up to date on what’s happening is to use a free RSS reader.  Continue reading →

Data for journalists: understanding XML and RSS

10 Replies

If you are working with data chances are that sooner or later you will come across XML – or if you don’t, then, well, you should do. Really.

There are some very useful resources in XML format – and in RSS, which is based on XML – from ongoing feeds and static reference files to XML that is provided in response to a question that you ask. All of that is for future posts – this post attempts to explain how XML is relevant to journalism, and how it is made up.

What is XML?

XML is a language which is used for describing information, which makes it particularly relevant to journalists – especially when it comes to interrogating large sets of data.

If you wanted to know how many doctors were privately educated, or what the most common score was in the Premiership last season, or which documents were authored by a particular civil servant, then XML may be useful to you. Continue reading →

Review: Yahoo! Pipes tutorial ebook

3 Replies

Pipes Tutorial ebook

I’ve been writing about Yahoo! Pipes for some time, and am consistently surprised that there aren’t more books on the tool. Pipes Tutorial – an ebook currently priced at $14.95 – is clearly aiming to address that gap.

The book has a simple structure: it is, in a nutshell, a tour around the various ‘modules’ that you combine to make a pipe.

Some of these will pull information from elsewhere – RSS feeds, CSV spreadsheets, Flickr, Google Base, Yahoo! Local and Yahoo! Search, or entire webpages.

Some allow the user to input something themselves – for example, a search phrase, or a number to limit the type of results given.

And others do things with all the above – combining them, splitting them, filtering, converting, translating, counting, truncating, and so on.

When combined, this makes for some powerful possibilities – unfortunately, its one-dimensional structure means that this book doesn’t show enough of them.

Modules in isolation

While the book offers a good introduction into the functionality of the various parts of Yahoo! Pipes, it rarely demonstrates how those can be combined. Typically, tutorial books will take you through a project that utilises the power of the tools covered, but Pipes Tutorial lacks this vital element. Sometimes modules will be combined in the book but this is mainly done because that is the only way to show how a single module works, rather than for any broader pedagogical objective.

At other times a module is explained in isolation and it is not explained how the results might actually be used. The Fetch Page module, for example – which is extremely useful for scraping content from a webpage – is explained without reference to how to publish the results, only a passing mention that the reader will have to use ‘other modules’ to assign data to types, and that Regex will be needed to clean it up.

Continue reading →

Data journalism pt5: Mashing data (comments wanted)

7 Replies

This is a draft from a book chapter on data journalism (part 1 looks at finding data; part 2 at interrogating data; part 3 at visualisation, and 4 at visualisation tools). I’d really appreciate any additions or comments you can make – particularly around tips and tools.

UPDATE: It has now been published in The Online Journalism Handbook.

Mashing data

Wikipedia defines a mashup particularly succinctly, as “a web page or application that uses or combines data or functionality from two or many more external sources to create a new service.” Those sources may be online spreadsheets or tables; maps; RSS feeds (which could be anything from Twitter tweets, blog posts or news articles to images, video, audio or search results); or anything else which is structured enough to ‘match’ against another source.

This ‘match’ is typically what makes a mashup. It might be matching a city mentioned in a news article against the same city in a map; or it may be matching the name of an author with that same name in the tags of a photo; or matching the search results for ‘earthquake’ from a number of different sources. The results can be useful to you as a journalist, to the user, or both.

Why make a mashup?

Mashups can be particularly useful in providing live coverage of a particular event or ongoing issue – mashing images from a protest march, for example, against a map. Creating a mashup online is not too dissimilar from how, in broadcast journalism, you might set up cameras at key points around a physical location in anticipation of an event from which you will later ‘pull’ live feeds: in a mashup you are effectively doing exactly the same thing – only in a virtual space rather than a physical one. So, instead of setting up a feed at the corner of an important junction, you might decide to pull a feed from Flickr of any images that are tagged with the words ‘protest’ and ‘anti-fascist’. Continue reading →

Data journalism pt2: Interrogating data

18 Replies

This is a draft from a book chapter on data journalism (the first, on gathering data, is here). I’d really appreciate any additions or comments you can make – particularly around ways of spotting stories in data, and mistakes to avoid.

UPDATE: It has now been published in The Online Journalism Handbook.

“One of the most important (and least technical) skills in understanding data is asking good questions. An appropriate question shares an interest you have in the data, tries to convey it to others, and is curiosity-oriented rather than math-oriented. Visualizing data is just like any other type of communication: success is defined by your audience’s ability to pick up on, and be excited about, your insight.” (Fry, 2008, p4)

Once you have the data you need to see if there is a story buried within it. The great advantage of computer processing is that it makes it easier to sort, filter, compare and search information in different ways to get to the heart of what – if anything – it reveals. Continue reading →

Online Journalism lesson #10: RSS and mashups

7 Replies

This was the final session in my undergraduate Online Journalism module (the other classes can be found here), taught last May. It’s a relatively brief presentation, just covering some of the possibilities of mashups and RSS, and some tools. The majority of the class is taken up with students using Yahoo! Pipes to aggregate a number of feeds.

I didn’t know how students would cope with Yahoo! Pipes but, surprisingly, every one completed the task.

As a side note, this year I kicked off the module with students setting up Twitter, Delicious and Google Reader – and synchronising them, so the RSS feed from one could update another (e.g. bookmarks being published to Twitter). This seems to have built a stronger understanding of RSS in the group, which they are able to apply elsewhere (they also have widgets on their blogs pulling the RSS feeds from Twitter & Delicious; and their profile page on the news website – built by Kasper Sorensen – pulls the latest updates from their Twitter, Delicious and blog feeds).

How to create a custom meta-search in Yahoo! Pipes

4 Replies

Here’s another tutorial on the mashup platform Yahoo! Pipes, showing how you can use it to create a meta-search that will push any search term by the user through a number of search engines, and present you with a combined result (and RSS feed). A finished version of the pipe can be seen here.

This tutorial builds on a previous post I published on how to create basic mashups with Yahoo! Pipes. If you haven’t any knowledge of Pipes you should probably read that first.

How to create a custom meta-search in Yahoo! Pipes

First, you obviously need to log in to Yahoo! Pipes, and click on Create a Pipe. You’ll be taken to the Pipe editing interface: on the left will be a menu with a series of sections (User Input, Url, Operators, etc.) to choose modules from. In the centre will be the canvas where you create your pipe – and at the bottom a ‘Debugger’ area where you can see the results of any particular part of your pipe.

In the area on the left, under the ‘User Input’ section, click on the ‘Text Input‘ module and drag it onto the canvas (or you can click on the + sign for it to be placed for you).

Select the Text Input module

Select the Text Input module

In the box marked ‘Prompt’ type the instruction text for users of the pipe, e.g. ‘What do you want to search for?’. If there’s a default search you want to have appear in the search box to begin with, enter it in the box marked ‘Default’. Continue reading →

How to make interactive geographical timelines using Google Calendar and Yahoo Pipes

102 Replies

I was recently given a task where my job was to create a calendar holding around 50 events. Each event also needed to be mapped, and have a corresponding blog post.

Mapping calendar entries made me think, if this could be used for other stuff than simply putting events on a map, – which is quite useful in it’s own way. I thought it would be cool if you could create an interactive map-timeline, controlled dynamically by a (shared)calendar.

Yahoo Pipes by default uses Yahoo Maps, which is great when it comes to narratives. As you can see from the map below (If you don’t see it, click here), each entry has a little arrow that let’s you navigate from marker to marker in a specific order. Each marker also has a number indicating it’s place in a sequence. This is nothing more than entries in a Google Calender with time/date stamps, geo info and a description, mapped automatically using Yahoo Pipes.

{“pipe_id”:”ed13a198a2a83050dd4ace10d12eae16″,”_btype”:”map”,”pipe_params”:{“Curl”:”http://kaspersorensen.com/wp-content/uploads/files/icalyahoopipes.ics”}}

Here’s how you do it.

1. Create a Google Calendar

Simply go to your Google Calendar and create or import a new calendar. You can do this from the settings page under calendars.

2. Make it public

You need to make the calendar public, otherwise Yahoo Pipes won’t have access to it. You can do this while you create it, or afterwards by ticking the box ‘Make this calendar public’ from the sharing settings on your specific calendar. To access the settings for a specific calendar, you click the little arrow in the box on the left hand side that contains your calendars (My Calendars).

3. Create events

Now you simply start adding events to your calendar. Specify what happened, where it happened, when and add the description. You don’t have to add the entries chronologically, they will be sorted by date/time automatically.

4. Feed the iCal file to the Pipe

Go to your calendar settings page, not the general Calendar settings, but the settings for your specific calendar. You will see a section called ‘Calendar address’ with three buttons. Click the green ICAL button and copy the link that pops up. Now go to Mapping Google Calendar Events Pipe and paste it into the ‘Calendar iCal URL’ field and hit ‘Run Pipe’. – Your events are now mapped.

5. Embed on your website

To embed the timeline/map on your website, simply select ‘Get as badge’ just above the map. This will allow you to insert it on your blog or website.

I’m sure there are ways to make this more stable. So if you know how to optimize the pipe, please feel free to do so and let me know.

As Google Maps is already a part of Google Calendar, you would think that there was a nifty way to quickly put a whole calendar on a map, but no. And after failing to use what looked like a saviour, I bumped into a post by Tony Hurst on how to display Google Calendar events on a Google Map. Unfortunately it turns out that the XML feed Tony uses, only parses the 25 most recent calendar entries.

Google Calendar releases their event-entries in iCal format which contains all events. And with a little customization of Tony’s pipe, I managed to come up with a way to map all events from a calendar.

I think this could be potentially useful for developing stories, especially if you can collaborate on the calendar. You end up with data that can be used for nearly anything, not just maps. And if locations aren’t relevant for the story, you could simply take your iCal file and make a normal timeline.

Are you a journalist using Yahoo! Pipes?

Leave a reply

How widely is Yahoo! Pipes used in newsrooms? Could it be better used? Is there a way us Piping Journos could exchange best practice, ideas, and support?

I’d like to bring journalists using Yahoo! Pipes together, so I’ve created a little Twitter group. Hope you can join, and we can find some ways to help each other out.

If you’re not using Pipes or would still like a primer, here’s one I prepared earlier.