Here’s a solution to a problem that aspiring data journalists have come up against time and time again: how to quickly create a searchable interface to your dataset.

freeDive is quick and – if you can follow the wizard‘s instructions – easy too.

If you have a dataset that you want people to be able to search or filter, publish it on Google Docs, copy part of the URL into freeDive when it asks you for it, then decide which fields you want people to be able to see, and how they can search, or filter their search.

At the end you’re given some script which is the most likely point of struggle for some users. If you use a WordPress.com blog then it won’t work – and even a self-hosted WordPress blog will require the installation and activation of a plugin (helpfully linked on the freeDive site) and following those instructions (if you don’t know how to use custom fields check out the explanations here and here).

If you’ve tried it on Blogger or another platform, let me know about it.

Meanwhile, here’s a test of the interface in action – the technology is still in ‘alpha’ so expect some bugginess.

PHP Freelancer

I’ve never been fond of the search engine on Twitter, not the one on search.twitter.com anyway. I have found the ones build on it’s API much friendlier and more intuitive, such as Twitterfall and the integration in Tweetdeck. But none of them work for finding old tweets. Google is not much help either, unless you know how to create your own search engine.

Friendfeed aggregates and stores all the activity that is fed into the system. Most FF users bring in their Twitter feed, in effect storing all their tweets. It works a little bit like Google Reader, once it’s there, it will always be there, even if the original is deleted.

The advanced search features of Friendfeed makes it a pretty good twitter search alternative. It even supports real-time, so you can make your own twitter news monitors.

Searching old tweets

Twitter only keeps tweets in it’s search database for a few weeks, after that they disappear. They’re still available on the web, just not searchable from Twitter (or any thrid party app). That’s great if you just want the real-time view, but not practical when looking for an exact tweet a few weeks old.

I needed to find this tweet from Paul Bradshaw for a presentation, but it was long gone from the internal database. I knew that Paul is using Friendfeed, not actively but he’s sharing his tweets there, so I did this search (Bingo, no. 2 from the top). Here’s the equivalent twitter search which is no help.

From any Friendfeed page, you simple select advanced search at the top, fill in the blanks and you got it. Here’s how mine was filled in.

Real-time “noise” filtering

Some hashtags can get ugly, real quick. There’s no way to filter out the high quality tweets either. People can favorite tweets, but you can’t search them, so no way to filter. When news breaks, there will be a few quality tweets in the beginning, people will retweet the most important. But people quickly starts talking about the event which brings no real value to the table, other than twitter-chatter. Eyewitness accounts and other useful information is lost in the stream because people have no way of marking important tweets for later retrieval (search).

On Friendfeed, people have the option of liking entries, and the advanced search let’s you filter items based on likes or comments. You can now rely on the FF community to mark the important stuff and cut through all (some of) the noise.

Here’s an example of a search that filters all tweets with the #iranelection hashtag, and shows only tweets that has 3 or more likes.

Other uses

There are many other ways to search Friendfeed and you can filter for all services like facebook, blogs etc. You can save searches and use them as filters. I have several live searches saved in Friendfeed. Here’s an example of a search that gives me all twitter entries from my friends with one or more comments.

Friendfeed suffers from the fact that it’s userbase is not as big as Twitter’s, but the ‘real’ real-time search more than makes up for that in my opinion. What I mean by real, is that items are published automatically from all services. If you bring in your Flickr, comments and blog activity to Friendfeed, they will publish automatically. Twitter doesn’t do that, you have to actively share the link after you have uploaded to Flickr, made a comment somewhere or updated your blog.

PHP Freelancer

It’s often said that Twitter’s big advantage over Google is its ability to allow you to conduct ‘real time search’ – if an event is happening right now, you don’t search Google, you search Twitter.

But today Google has announced a series of features that, while still not offering real time search, take it just that bit closer. For me it is the most significant change to Google’s core service in years. 

Here’s the video:

This week, while talking to my students about the ability to search by date in Google, the computer assisted reporting blogger Murray Dick mentioned how unreliable the feature was, so I wouldn’t get too excited. 

What is new, however, is the ‘recent search’ facility, which brings up results from the past hour or two.

Perhaps most significantly, the intuitive nature of the controls makes it much easier to find relevant information – and has obvious uses if you are a journalist only interested in recent reports. I tried it with an event I attended today and the results were clearly better than a generic search.

The ‘Timeline’ feature is particularly useful – you can now drill down to a specific period, allowing you to do the opposite – rule out recent reports – if, for example, you’re looking for background on someone currently in the news.

Also useful is the horrendously multicoloured ‘Wonder Wheel’ which will generate related terms that you can browse through.

So, a search for ‘Paul Bradshaw’ brings up online journalism, journalism blog, Press Gazette and Birmingham City University – an obvious way to get an overview of the areas and companies a person is connected with, and drill down to information about those. 

‘More text’ will give expanded descriptions of results; you can choose to include images from the pages; and it is easier to switch to searches of video, forums and reviews. 

Also worth noting is that Google appear to now be supporting microformats: this means you can filter reviews by sentiment, and more easily disambiguate people in searches. 

It also means you should be looking at using microformats for your own content, of course… 

If you can see any other applications of Search Options, or find any features I’ve not spotted, let me know.

UPDATE: TechCrunch compare Google’s performance against Twitter Search, FriendFeed, OneRiot, Tweetmeme and Scoopler (which I love) in this real-time search-off.

UPDATE 2: ZDNet spots the addition of public comments on search results. Very interesting.

PHP Freelancer

Many of the services that are being developed as part of the ‘semantic web’ are necessarily works in progress, but they all contribute to extending the success of this burgeoning area of technology. There are plenty more popping up all the time, but for the purposes of this post I have loosely grouped some prominent sites into specialities – social networking, search and browsing – before briefly explaining their uses.

BROWSING

OpenCalais is a way to tag people, places, facts and events in pre-existing content to increase its value and accessibility. It makes use of RDF to annotate content intelligently and automatically so that it can be used in more meaningful ways. Developed by Thomson Reuters, the service now has a preview tool that can take any document and provide a display of the results of tagging and linking the semantic data. It provides an immediate and useful example of the way the technology works and is fun to play around with. OpenCalais is also available as a WordPress plugin which uses the service for auto tagging posts and archives with the correct themes.

headup is a Firefox plugin that enables semantic capabilities within any web page. Extra data is displayed fully in context as the service just alerts the user with a ‘+’ symbol when there is something else of interest to them. On encountering data about a band, headup might highlight the latest YouTube videos, tour dates and official blog-posts next to their name. This data can all be viewed without ever navigating away from the original page. Impressively headup’s semantic engine promises to provide a personalised service by retrieving information that specifically interests the individual user. You can watch a demonstration video here.

SEARCH

SemantiFind claims to return more relevant results than traditional search engines, yet users can still continue using them as it is compatible with Google, Yahoo and Microsoft Live Search. You have to download and install a free browser plug-in, but SemantiFind results are displayed alongside normal search engine results, offering some familiarity. You can watch a demonstration video here.

Powerset is among those services applying natural language processing to the web and Wikipedia already benefits from its approach. Powerset displays an interface alongside the Wiki itself so users can navigate quickly and seamlessly using the keywords, themes and sections which have been stripped out of the original article. You can watch a demonstration video here.

iGlue is a search engine that tries to identify and manage entities, not keywords. The service finds relevant information even if the given element appears in a form different from that used in the original search. It understands that corresponding words can sometimes be substituted. You can see a demonstration of the technology here.

SOCIAL NETWORKING

Twine seems to be the pre-eminent ‘semantic web’ service when it comes to social networking. It acts as a means of collecting and sharing all kinds of online content, learning more about you as you fill it up and link to other content. Twine aims to build on the principles of developing communities of interest. You can even interact with Nova Spivack, the site’s creator, and see what things have captured his attention.

It is clear that all of the services – whether targeted at browsing, search or social networking – foster more advances in the field and my final post on the ‘semantic web’ explores the revolutionary uses of these new technologys for the benefit of journalism. My previous post called ‘The next step to the ‘semantic web” can be found here.

PHP Freelancer

There are billions of pages of unsorted and unclassified information online, which make up millions of terabytes of data with almost no organisation.  It is not necessarily true that some of this information is valuable whilst some is worthless, that’s just a judgement for who desires it.  At the moment, the most common way to access any information is through the hegemonic search engines which act as an entry point.

Yet, despite Google’s dominace of the market and culture, the methodology of search still isn’t satisfactory.  Leading technologists see the next stage of development coming, where computers will become capable of effectively analysing and understanding data rather than just presenting it to us.  Search engine optimisation will eventually be replaced by the ‘semantic web’.

Correctly tagging the mass of available data to provide a clear sense of meaning is the best way of achieving this according to Nova Spivack, founder of Twine, one of the leading sites in this field.  He says:

“This next generation is actually based on enriching the meaning, enriching the structure.  The reason we want to do this is so that software can understand the web like humans can understand the web. Because the semantic web is not for humans, it is for machines.”

Undertaking this task will revolutionise the way we utilise the internet, creating intelligent interaction and impacting on the way the web is perceived in popular culture. Vint Cerf, one of the driving forces behind the creation of the internet, says:

‘I don’t believe that we will see arising out of the current internet…conscious artificial intelligence, but we will probably see the system become easier to interact with – for example, voice interaction is becoming increasingly easy to accomplish. I’m almost certain you’ll see products emerging that will allow you to orally interact with the network – ask for something, demand something, or command something and have [it] happen.

“We may feel that this system is more intelligent because we are interacting with it in ways that don’t require us to point, click and type. The semantic web idea will make the internet seem more intelligent because we are extracting knowledge that other people put into it in a way that looks pretty intelligent.”

So the aim of the ‘semantic web’ is to allow data to be accessed and shared effectively by wider communities, yet processed automatically by computer.  In order for this to happen there needs to be a simple system to catagorise data so it can be easily located and organised.

Much progress has been made in this infrastructure, particularly in the development of the new languages – Resource Description Framework (RDF) and Web Ontology Language (OWL) – by the World Wide Web Consortium .  The languages are used to annotate code, representing ‘knowledge’ which will enable applications to use them more intelligently.

At the moment HTML is limited to describing static content, documents and the links between them. However RDF, OWL, and XML can describe arbitrary things such as people, events or objects.  It is layered on top of HTML and consists of a subject, a predicate, and an object. For example: “Jeremy Paxman” <subject> belongs to <predicate> journalists <object>.

These descriptions allow increased meaning behind the static content, demonstrating the structure of the knowledge behind it.  In this way a machine can process knowledge itself instead of text, using a process similar to human reasoning.  This should result in more meaningful results being returned in searches and perhaps even allow for increased automation when it comes to research by computers.

The success (or failure) of these experimental technologies will motivate further research and development, not only from within the industry but also academia.  It is certain their efforts will influence the future development of information technology.  In a further post I will explore the services currently being forged and in a final post on the ‘semantic web’ I will tackle the revolutionary uses this new technology has for journalism.

However the last word here must go to Tim Berners-Lee, the internet pioneer who says:

“A ‘semantic web’ has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines. The ‘intelligent agents’ people have touted for ages will finally materialise.”

PHP Freelancer

Which comes first? A newspaper’s agenda or its readers’ interest in those subjects? The search suggestions at Express.co.uk give a revealing insight into either what its readers are searching for or what the Express wants them to be interested in.

The screenshot below, first published on this blog, is a photomontage of the search box on the Express site. Every time you reload the Express search page, a different ‘example search’ is shown. The list seems to suggest a certain editorial agenda …

The 15 examples shown here are: anorexia, abortion, muslims, big ben, recipe, bnp, friends provident, apprentice, floods, care homes, immigration, binge drinking, lottery, Madeleine McCann, Liverpool FC. But there are many more to be found by reloading the search page (none of them terribly cheerful).

For more on search at the Express, check out this analysis of its top-10 searches box, revealing an ongoing obsession with the McCanns.

PHP Freelancer

This is something of a bonus following on from my previous beginner’s guide to Twitter. I didn’t have time to deliver this on Monday, but the following is a quick outline of various strategies for finding people of interest on Twitter.

PHP Freelancer

Malcolm Coles asks a reasonable question of Johnston Press Digital Publishing: have they looked at their own pages in Google News? It seems a reliance on javascript is making their pages invisible to search engines.

PHP Freelancer

That’s the question bumping around my mind after reading this post at SEObook.com:

“if you are not an AdWords advertiser, are not in universal search verticals (like news and video), and are not wikipedia, then you don’t have many organic search results that you can rank for on the first page.”

The image makes it clearer:

In some ways, blogs are better placed than ordinary websites, as Google may be indexing your blog as part of its news search. But that isn’t particularly comforting. The wider move towards mainstream results that keep you within Google doesn’t look particularly healthy. 

Here’s what SEObook suggests:

• If your site is fairly close to what it takes to be considered in some of Google’s verticals – like Google news, then consider upping your game a bit and submitting an inclusion request.
• Try to make some video content. Not good for everyone, but most sites could use some, and the competitive bar with video is much lower than it is with text – though I wouldn’t expect it to stay that way for more than a couple years.
• If you have some top rankings that are bouncing around consider focusing on promoting that content again – when stratification occurs you are going to be better off focusing on owning a few ideas rather than being average to slightly above average at many. Top ranked sites also benefit from self-reinforcing rankings. Read up on cumulative advantage if you have not yet done so.
• Usage data (and/or brand searches) may become a big part of future algorithms. Get ready for that by reading about BrowseRank then invest in advertising, branding, and user experience.

The only upside? Google may be making itself less relevant, and more open to competition.

PHP Freelancer