Online Journalism Blog

Here’s another tutorial on the mashup platform Yahoo! Pipes, showing how you can use it to create a meta-search that will push any search term by the user through a number of search engines, and present you with a combined result (and RSS feed). A finished version of the pipe can be seen here.

This tutorial builds on a previous post I published on how to create basic mashups with Yahoo! Pipes. If you haven’t any knowledge of Pipes you should probably read that first.

How to create a custom meta-search in Yahoo! Pipes

First, you obviously need to log in to Yahoo! Pipes, and click on Create a Pipe. You’ll be taken to the Pipe editing interface: on the left will be a menu with a series of sections (User Input, Url, Operators, etc.) to choose modules from. In the centre will be the canvas where you create your pipe – and at the bottom a ‘Debugger’ area where you can see the results of any particular part of your pipe.

In the area on the left, under the ‘User Input’ section, click on the ‘Text Input‘ module and drag it onto the canvas (or you can click on the + sign for it to be placed for you).

Select the Text Input module

In the box marked ‘Prompt’ type the instruction text for users of the pipe, e.g. ‘What do you want to search for?’. If there’s a default search you want to have appear in the search box to begin with, enter it in the box marked ‘Default’.Under the ‘Url’ section, click on ‘URL Builder‘ and drag it onto the canvas.

Select the URL Builder module

Go to a search engine that offers RSS feeds for searches, e.g. Google News. Do a test search (e.g. ‘fire’) and copy the address of the RSS feed for those search results (on Google News it will be at the foot of the page)

In the ‘URL Builder’ module paste the address of that RSS feed into the box marked ‘Base’

You’ll notice that the rest of the module now expands to include all of the parameters that the URL includes. The main one we are interested in is the one that relates to our search (you can tell which one it is as it should have the topic of your search in the right hand column) – in the case of Google News, this is ‘q’.

Customising the URL Builder module

Drag a wire from the circle at the bottom of the Text Input module to the circle to the right of that search term (e.g. ‘fire’). It should turn grey and you should now be able to see a wire joining the two modules. Your search term should now be replaced by ‘text [wired]‘.

Now drag a third module onto your canvas: this time it’s under the ‘Sources’ heading, and the module is called ‘Fetch Feed‘.

The Fetch Feed module

This time drag a wire from the circle at the bottom of ‘URL Builder‘ to the circle on the right of the ‘Fetch Feed‘ module. Again, it should turn grey and the previously empty box should now read ‘text [wired]‘.

A grey wire linking the URL Builder and Fetch Feed modules

Drag another ‘URL Builder‘ module onto the canvas and repeat the process detailed above for a separate search engine. Connect your ‘Text Input‘ module to it in the same way. You may have to delete some of the parameters to make these work. A blog search on IceRocket, for example, includes the parameter ‘tab’. Deleting this (with the ‘-’ button next to it) appears to make the search work in Pipes. If you know of any similar tweaks, please let me know.

In the ‘Fetch Feed’ module click on the ‘+’ icon to add a new, empty, box. Click on the circle at the bottom of your second ‘URL Builder‘ module to the circle to the right of that empty box. Again, it should turn grey and you should now be able to see a wire joining the two modules and the box should now read ‘text [wired]‘.

You can repeat these steps for further search engines.

Once you’ve finished, from the ‘Operators’ heading on the left, drag a ‘Union‘ module onto the canvas. You need this to combine your feeds.

To combine them, drag a wire from the bottom of all ‘Fetch Feed‘ modules to separate circles at the top of ‘Union‘ at the bottom of the canvas. Unlike the other pipes this will be blue.

Wires connect the Fetch Feed modules to the Union module, which in turn is connected to the Output module

Finally, drag a pipe from the bottom of ‘Union‘ to the top of ‘Pipe Output‘ at the bottom of the canvas.

That is it. Save your pipe and click on ‘Back to My Pipes’ towards the top centre of the screen and you should see your pipe listed. Click on that pipe to run it – you should see a text entry box towards the top where you can enter your search to change the results. If you want to embed this somewhere, click on ‘Get as a Badge’ to get the code.

However, before you do that you’ll notice that results will be clustered by search engine, and also may contain duplicate results. You’ll need to click on ‘Edit Source’ and add a couple more modules to tweak your pipe and solve the problem.

The modules you will need to use are the Unique module (in the Operators section) to filter out duplicate results (probably place it between Union and Pipe Output - drag wires to remove them), and the Sort module (also in the Operators section) to order results by date (otherwise they will cluster by search engine).

I’m by no means any expert on this, but put this out there for others to build on, comment on and correct.

Many thanks to Paul Daniel who showed me much of this as part of an open MA Online Journalism session and whose blog is a great resource on Pipes.