The rather lovely DocumentCloud – a tool that allows journalists to share, annotate, connect and organise documents – has finally emerged from its closet and made itself available to public searches.

This means that anyone can now search the powerful database (some tips here) of newsworthy documents. If you want to add your own, however, you still need approval.

If you do end up on this list you’ll find it’s quite a powerful tool, with quick conversion of PDFs into text files, analytic tools and semantic tagging (so you can connect all documents with a particular person, or organisation) among its best features. The site is open source and has an API too.

I asked Program Director Amanda B Hickman what she’s learned on the project so far. Her response suggests that documents have a particular appeal for online readers:

“If we’ve learned anything, it is that people really love documents. It is pretty clear that when there’s something interesting going on in the news, plenty of people want to dig a little deeper. When Arizona Republic posted an annotated version of that state’s new immigration law, it got more traffic than their weekly entertainment round up. WNYC told us that the page listing the indictments in last week’s mob roundup was still getting more traffic than any other single news story even a week later.

“These were big news documents, to be sure, but it still seems pretty clear that people do want to dig deeper and explore the documents behind the news, which is great for us and great for news.”

PHP Freelancer

Here’s another data analysis tool which is worth keeping an eye on. Extractiv “lets you transform unstructured web content into highly-structured semantic data.” Eyes glazing over? Okay, over to ReadWriteWeb:

“To test Extractive, I gave the company a collection of more than 500 web domains for the top geolocation blogs online and asked its technology to sort for all appearances of the word “ESRI.” (The name of the leading vendor in the geolocation market.)

“The resulting output included structured cells describing some person, place or thing, some type of relationship it had with the word ESRI and the URL where the words appeared together. It was thus sortable and ready for my analysis.

“The task was partially completed before being rate limited due to my submitting so many links from the same domain. More than 125,000 pages were analyzed, 762 documents were found that included my keyword ESRI and about 400 relations were discovered (including duplicates). What kinds of patterns of relations will I discover by sorting all this data in a spreadsheet or otherwise? I can’t wait to find out.”

What that means in even plainer language is that Extractiv will crawl thousands of webpages to identify relationships and attributes for a particular subject.

This has obvious applications for investigative journalists: give the software a name (of a person or company, for example) and a set of base domains (such as news websites, specialist publications and blogs, industry sites, etc.) and set it going. At the end you’ll have a broad picture of what other organisations and people have been connected with that person or company. Relationships you can ask it to identify include relationships, ownership, former names, telephone numbers, companies worked for, worked with, and job positions.

It won’t answer your questions, but it will suggest some avenues of enquiry, and potential sources of information. And all within an hour.

Time and cost

ReadWriteWeb reports that the process above took around an hour “and would have cost me less than $1, after a $99 monthly subscription fee. The next level of subscription would have been performed faster and with more simultaneous processes running at a base rate of $250 per month.”

As they say, the tool represents “commodity level, DIY analysis of bulk data produced by user generated or other content, sortable for pattern detection and soon, Extractiv says, sentiment analysis.”

Which is nice.

PHP Freelancer

Tonight I had the pleasure of chairing an extremely informative panel discussion on data and the future of journalism at the first London Linked Data Meetup. On the panel were:

• Martin Belam (Information Architect, The Guardian; blogger, Currybet)
• John O’Donovan (Chief Architect, BBC News Online)
• Dan Brickley (Friend of a Friend project; VU University, Amsterdam; SpyPixel Ltd; ex-W3C)
• Leigh Dodds (Talis)

What follows is a series of notes from the discussion, which I hope are of some use.

For a primer on Linked Data there is A Skim-Read Introduction to Linked Data; Linked Data: The Story So Far PDF) by Tom Heath, Christian Bizer and Berners-Lee; and this TED video by Sir Tim Berners-Lee (who was on the panel before this one).

To set some brief context, I talked about how 2009 was, for me, a key year in data and journalism – largely because it has been a year of crisis in both publishing and government. The seminal point in all of this has been the MPs’ expenses story, which both demonstrated the power of data in journalism, and the need for transparency from government – for example, the government appointment of Sir Tim Berners-Lee, seeking developers to suggest things to do with public data, and the imminent launch of Data.gov.uk around the same issue.

Even before then the New York Times and Guardian both launched APIs at the beginning of the year, MSN Local and the BBC have both been working with Wikipedia and we’ve seen the launch of a number of startups and mashups around data including Timetric, Verifiable, BeVocal, OpenlyLocal, MashTheState, the open source release of Everyblock, and Mapumental.

Q: What are the implications of paywalls for Linked Data?

The general view was that Linked Data – specifically standards like RDF – would allow users and organisations to access information about content even if they couldn’t access the content itself. To give a concrete example, rather than linking to a ‘wall’ that simply requires payment, it would be clearer what the content beyond that wall related to (e.g. key people, organisations, author, etc.)

Leigh Dodds felt that using standards like RDF would allow organisations to more effectively package content in commercially attractive ways, e.g. ‘everything about this organisation’.

Q: What can bloggers do to tap into the potential of Linked Data?

This drew some blank responses, but Leigh Dodds was most forthright, arguing that the onus lay with developers to do things that would make it easier for bloggers to, for example, visualise data. He also pointed out that currently if someone does something with data it is not possible to track that back to the source and that better tools would allow, effectively, an equivalent of pingback for data included in charts (e.g. the person who created the data would know that it had been used, as could others).

Q: Given that the problem for publishing lies in advertising rather than content, how can Linked Data help solve that?

Dan Brickley suggested that OAuth technologies (where you use a single login identity for multiple sites that contains information about your social connections, rather than creating a new ‘identity’ for each) would allow users to specify more specifically how they experience content, for instance: ‘I only want to see article comments by users who are also my Facebook and Twitter friends.’

The same technology would allow for more personalised, and therefore more lucrative, advertising.

John O’Donovan felt the same could be said about content itself – more accurate data about content would allow for more specific selling of advertising.

Martin Belam quoted James Cridland on radio: “[The different operators] agree on technology but compete on content”. The same was true of advertising but the advertising and news industries needed to be more active in defining common standards.

Leigh Dodds pointed out that semantic data was already being used by companies serving advertising.

Other notes

I asked members of the audience who they felt were the heroes and villains of Linked Data in the news industry. The Guardian and BBC came out well – The Daily Mail were named as repeat offenders who would simply refer to “a study” and not say which, nor link to it.

Martin Belam pointed out that The Guardian is increasingly asking itself ‘How will that look through an API’ when producing content, representing a key shift in editorial thinking. If users of the platform are swallowing up significant bandwidth or driving significant traffic then that would probably warrant talking to them about more formal relationships (either customer-provider or partners).

A number of references were made to the problem of provenance – being able to identify where a statement came from. Dan Brickley specifically spoke of the problem with identifying the source of Twitter retweets.

Dan also felt that the problem of journalists not linking would be solved by technology. In conversation previously, he also talked of “subject-based linking” and the impact of SKOS and linked data style identifiers. He saw a problem in that, while new articles might link to older reports on the same issue, older reports were not updated with links to the new updates. Tagging individual articles was problematic in that you then had the equivalent of an overflowing inbox.

(I’ve invited all 4 participants to correct any errors and add anything I’ve missed)

Finally, here’s a bit of video from the very last question addressed in the discussion (filmed with thanks by @countculture):

Linked Data London 090909 from Paul Bradshaw on Vimeo.

PHP Freelancer

Elsevier, the Dutch scientific publisher, has announced details of their grandly titled Article of the Future project.  Their prototypes, published at http://beta.cell.com, are the result of what Emilie Marcus, Editor in Chief, Cell Press called,

“…a challenge to redesign from scratch how to most effectively structure and present the content of a traditional scientific article in an online environment.”

Prototypes
Several things strike me about the prototypes — and let’s bear in mind that these are prototypes, and so are likely to change based on feedback from users in the scientific community and elsewhere; but also that they are published prototypes, and so by definition are completely open for comment — the most obvious being their remarkable lack of futuristic qualities.  Instead, the prototypes resemble an enthusiastic bash at a multimedia-infused online encyclopaedia circa 1997, when multimedia was still a buzzword, or such as you might have found on a CD-ROM magazine cover mounted giveaway around the same time.

Now, I know relatively little about the scientific journal community, or how they consume their publications online.  But I imagine they will be intrigued though underwhelmed by an international publisher of scientific journals presenting them with these prototypes as the future of their work online.

Design
There is a lack of nuance, uniformity and overall cohesion in the article designs which immediately make me feel uneasy.  From a scientific journal I expect highly structured, easily accessible data.  The inconsistency in design and layout here means I have to work harder to know where I am in the document and what exactly I’m looking at.  Additionally, the wide text columns, combined with poor linespacing, lack of subheadings and lack of general hierarchical structure within the text contribute to overall poor readability.  I’d also suggest that the tabbed element leaves little room for expansion should the number of article sections increase.  There is also too much emphasis on distracting design elements like pop-ups and modal dialogues to present small snippets of data which would have been far more suitably presented within their referencing context.

Information architecture
There is a poor overall data structure, and to much repetition of content elements such as the figures.  Although this data structure undoubtedly encourages an initial period of browsing and discovery it also precludes an easy, linear reading of the content and lengthens the user journey considerably.

Development
From a development point of view it is important to note that these prototypes are completely reliant on JavaScript to present all of their content.  Without JavaScript the browser will only render the first tab’s content visibly.  Needless to say this is unacceptable on the modern web.  Separation of content, presentation and behaviour were long ago accepted as the minimum standard for a modern, user-friendly and broadly accessible web page.  Any web pages being designed now by international publishers should display this characteristic, especially this which claim to represent the future of the medium.  Similarly, these prototypes both have an obnoxious amount of inline JavaScript, making machine readability and text-browser readability a major pain, verging on the unusable.

Data
Perhaps the most important point of all — where’s the data?  XML and specifications like RDF are capable of representing structured scientific data online in such a way that it can be consumed by browsers, applications and other user agents we haven’t yet developed.  Scientific research data would seem to be a perfect candidate to display the power of the semantic web, and yet Elsevier appears to have shown a complete ignorance of its availability, its ability to open up data to a wider audience and to normalise data and allow consumers of data and information to use it in the manner most appropriate to them.

Maintenance
Are there authoring tools for creation and maintenance of these articles?  Is there a published schema and validator to ensure consistency between producers?  Which user needs does this article format address?  Will Elsevier publish their user research so we can better understand the problems users have with current scientific paper formats?  The whole format seems not to have been thought through properly.  Why not work with the W3C and other organisations dedicated to the description and ongoing development and maintenance of online document specifications.  Use the power of groups already working in this space instead of reinventing the wheel, badly.

Innovation?
Jan Aalbersberg, Vice President of Content Innovation for Elsevier Science & Technology Journal Publishing, says “We are confident that these tools will enhance the presentation of scientific results and improve the interpretation and speed of results analysis. They are central to driving innovation in scientific publishing and represent our investment in the future of research, enabling scientists all over the world to access, interpret, and create better science more efficiently.”  But I say that these “Articles of the Future” are not tools, and they are no more innovative than using a page layout application to alter the appearance of some printed matter.  Hyperlinking and the ability to add media files to a page have existed since the web was created, and these articles add nothing more to that basic paradigm of linked data files.  There are some nods towards current trends, with a comment feature and social bookmarking links.  But overall the feel is clunky, lacking research and distinctly amateur.

Articles of the Future?
Finally, there are two noteworthy features which may tell us more about the project, its origins and its ultimate destination than the prototypes themselves.  Firstly, the copyright notice, which proclaims the pages to be copyright of Elsevier 2008.  Are these prototypes already at least 6 months old?  Marcus herself notes,

“The rapid pace of technological advancements means this will undoubtedly be an evolving design,”

but is this an indication of the kind of turnaround we can expect for amendments and improvements to the article’s format?  Secondly, the entire article can be downloaded as a PDF, in its original published format, and which is perfect for printing, and I suspect that, excepting casual browsers, this will be how most users choose to consume their scientific research and analysis.

I think this area of publishing is indeed long overdue a complete overhaul of its staid online publishing practices, and any move to define a new specification for doing so should be welcomed.  Even the otherwise impressive nature.com only goes so far in its presentation of research papers, and there is much room for improvement.  But when the result is as underwhelming, cumbersome and shortsighted as this, I despair.

PHP Freelancer

Many of the services that are being developed as part of the ‘semantic web’ are necessarily works in progress, but they all contribute to extending the success of this burgeoning area of technology. There are plenty more popping up all the time, but for the purposes of this post I have loosely grouped some prominent sites into specialities – social networking, search and browsing – before briefly explaining their uses.

BROWSING

OpenCalais is a way to tag people, places, facts and events in pre-existing content to increase its value and accessibility. It makes use of RDF to annotate content intelligently and automatically so that it can be used in more meaningful ways. Developed by Thomson Reuters, the service now has a preview tool that can take any document and provide a display of the results of tagging and linking the semantic data. It provides an immediate and useful example of the way the technology works and is fun to play around with. OpenCalais is also available as a WordPress plugin which uses the service for auto tagging posts and archives with the correct themes.

headup is a Firefox plugin that enables semantic capabilities within any web page. Extra data is displayed fully in context as the service just alerts the user with a ‘+’ symbol when there is something else of interest to them. On encountering data about a band, headup might highlight the latest YouTube videos, tour dates and official blog-posts next to their name. This data can all be viewed without ever navigating away from the original page. Impressively headup’s semantic engine promises to provide a personalised service by retrieving information that specifically interests the individual user. You can watch a demonstration video here.

SEARCH

SemantiFind claims to return more relevant results than traditional search engines, yet users can still continue using them as it is compatible with Google, Yahoo and Microsoft Live Search. You have to download and install a free browser plug-in, but SemantiFind results are displayed alongside normal search engine results, offering some familiarity. You can watch a demonstration video here.

Powerset is among those services applying natural language processing to the web and Wikipedia already benefits from its approach. Powerset displays an interface alongside the Wiki itself so users can navigate quickly and seamlessly using the keywords, themes and sections which have been stripped out of the original article. You can watch a demonstration video here.

iGlue is a search engine that tries to identify and manage entities, not keywords. The service finds relevant information even if the given element appears in a form different from that used in the original search. It understands that corresponding words can sometimes be substituted. You can see a demonstration of the technology here.

SOCIAL NETWORKING

Twine seems to be the pre-eminent ‘semantic web’ service when it comes to social networking. It acts as a means of collecting and sharing all kinds of online content, learning more about you as you fill it up and link to other content. Twine aims to build on the principles of developing communities of interest. You can even interact with Nova Spivack, the site’s creator, and see what things have captured his attention.

It is clear that all of the services – whether targeted at browsing, search or social networking – foster more advances in the field and my final post on the ‘semantic web’ explores the revolutionary uses of these new technologys for the benefit of journalism. My previous post called ‘The next step to the ‘semantic web” can be found here.

PHP Freelancer

There are billions of pages of unsorted and unclassified information online, which make up millions of terabytes of data with almost no organisation.  It is not necessarily true that some of this information is valuable whilst some is worthless, that’s just a judgement for who desires it.  At the moment, the most common way to access any information is through the hegemonic search engines which act as an entry point.

Yet, despite Google’s dominace of the market and culture, the methodology of search still isn’t satisfactory.  Leading technologists see the next stage of development coming, where computers will become capable of effectively analysing and understanding data rather than just presenting it to us.  Search engine optimisation will eventually be replaced by the ‘semantic web’.

Correctly tagging the mass of available data to provide a clear sense of meaning is the best way of achieving this according to Nova Spivack, founder of Twine, one of the leading sites in this field.  He says:

“This next generation is actually based on enriching the meaning, enriching the structure.  The reason we want to do this is so that software can understand the web like humans can understand the web. Because the semantic web is not for humans, it is for machines.”

Undertaking this task will revolutionise the way we utilise the internet, creating intelligent interaction and impacting on the way the web is perceived in popular culture. Vint Cerf, one of the driving forces behind the creation of the internet, says:

‘I don’t believe that we will see arising out of the current internet…conscious artificial intelligence, but we will probably see the system become easier to interact with – for example, voice interaction is becoming increasingly easy to accomplish. I’m almost certain you’ll see products emerging that will allow you to orally interact with the network – ask for something, demand something, or command something and have [it] happen.

“We may feel that this system is more intelligent because we are interacting with it in ways that don’t require us to point, click and type. The semantic web idea will make the internet seem more intelligent because we are extracting knowledge that other people put into it in a way that looks pretty intelligent.”

So the aim of the ‘semantic web’ is to allow data to be accessed and shared effectively by wider communities, yet processed automatically by computer.  In order for this to happen there needs to be a simple system to catagorise data so it can be easily located and organised.

Much progress has been made in this infrastructure, particularly in the development of the new languages – Resource Description Framework (RDF) and Web Ontology Language (OWL) – by the World Wide Web Consortium .  The languages are used to annotate code, representing ‘knowledge’ which will enable applications to use them more intelligently.

At the moment HTML is limited to describing static content, documents and the links between them. However RDF, OWL, and XML can describe arbitrary things such as people, events or objects.  It is layered on top of HTML and consists of a subject, a predicate, and an object. For example: “Jeremy Paxman” <subject> belongs to <predicate> journalists <object>.

These descriptions allow increased meaning behind the static content, demonstrating the structure of the knowledge behind it.  In this way a machine can process knowledge itself instead of text, using a process similar to human reasoning.  This should result in more meaningful results being returned in searches and perhaps even allow for increased automation when it comes to research by computers.

The success (or failure) of these experimental technologies will motivate further research and development, not only from within the industry but also academia.  It is certain their efforts will influence the future development of information technology.  In a further post I will explore the services currently being forged and in a final post on the ‘semantic web’ I will tackle the revolutionary uses this new technology has for journalism.

However the last word here must go to Tim Berners-Lee, the internet pioneer who says:

“A ‘semantic web’ has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines. The ‘intelligent agents’ people have touted for ages will finally materialise.”

PHP Freelancer

Reuters recently published a report entitled: ‘What’s Happening to Our News: An investigation into the likely impact of the digital revolution on the economics of news publishing in the UK‘. In it author Andrew Currah provides an overview of the situation facing UK publishers, and 3 broad suggestions as to ways forward – namely, kitemarks, public support, and digital literacy education.

The kitemark idea seems to have stirred up the most fuss. In the first of a series of email exchanges I asked Currah how he saw this making any difference to consumption of newspapers, and how it could work in practice. This is his response:

Yes, the kitemark idea has triggered quite a response… Unfortunately, as the discussion online suggests, the term has implied to many a top-down, centralised system of certification which would lead to some form of
‘apartheid’ between bloggers and journalists.

That was certainly not our intended message. The report simply wanted to foreground the idea of digital labelling as a means of improving transparency in online news coverage.

All we meant by a kitemark was a symbol (expressed visually, and electronically as metadata) to convey to audiences, bloggers, journalists and others that a piece of news content had been intelligently labelled with relevant information and that it is open to derivative checking/use… similar in a sense to the Creative Commons ‘mark’ that travels with media content across the web.

Our report only touched upon this project of labelling, which the Media Standards Trust are busy working on. For a more detailed discussion, see the post by Martin Moore or read about the related efforts of http://www.newscredit.org

So, in summary, we are in favour of an open source, voluntary, bottom-up system of tagging NOT an archaic, top-down system of certification dividing amateurs and professionals. We did not envision participation in such an initiative as a precursor to public funding – though intelligent labelling and linking to external sites could, for example, be far more developed at the BBC.

In terms of value, by intelligently labelling the news all sorts of valuable derivative uses might be enabled (e.g. helping users to filter content by criteria or triangulate stories). It might also help to avoid the failures of purely algorithmic search approaches to news (e.g. the fiasco surrounding the publication of an outdated United Airlines story on Google News in August last year – triggered, in part, due to the lack of any embedded metadata about the story’s publication date):

Is this similar to the ideas that Tim Berners- Lee is working on in his Knight-funded project?

Yes – absolutely. This is something we only briefly touch in the report. We’re hoping to spend more time looking at this approach in follow-on research. I think the initiative being developed by Tim Berners-Lee and the Media Standards Trust has a great chance of improving transparency, especially when tagging and labelling technologies are seamlessly integrated into the workflow of the newsroom.

I can see how something around metadata could help users find original journalism, but how do you see this kitemark keeping journalism alive in a business sense?

Whether this would realistically boost the economics of news is difficult to answer. But on the basis of our research, it seems that a more transparent, systematic way of tagging the news could help publishers in a variety of ways…

For example, by developing a more comprehensive network of tags connecting stories, themes and content that might, in theory, keep people on a site for longer – in turn, strengthening ad revenues. It might also perpetuate the value and profile of a story long after it was published.

Metadata is also the key to techniques such as search engine optimisation, social media marketing, and the like, all of which are about attracting more attention around the content for longer. It would also provide a system for displaying stories in new formats, such as digital maps.

When or whether all of this will translate into enough ad revenues to keep publishers afloat is an open question; investing in the systems and training to make this archival linking possible is another hurdle.

An alternative approach might be to buck the trend towards free by introducing new forms of online paid subscription, to provide access to a premium, searchable and fully digitized archive of all back content. Metadata would also be a key step in that direction.

The discussion continues in the comments

NB: The Freeman’s Journal has an excellent critical overview of the report, with responses from Currah in the comments.

PHP Freelancer

This month’s Carnival of Journalism looks forward to new media developments in the coming year. Here are my no doubt misguided and naive predictions:

2009 will not be the year of the mobile web

Every year we make end of year predictions that the coming year will finally see the mobile web hit the mainstream. In many ways, it already has. But any expectations of there being some significant spread in 2009 will be scuppered by the credit crunch: users will be increasingly reluctant to spend money on a smart phone as the purse strings tighten. We’re not all going to be carrying around iPhones.

On the plus side, as a result of that slowdown we can expect mobile service providers to become more competitive in their data rates and packages, so that those who do have smart phones will have more reason to take out a mobile web package.

We can also expect to see increasing numbers of retailers offering free wifi to attract customers, as Pret A Manger have done, or government investment in wifi clouds to stimulate growth. So those who do access the web on the move – not just mobile phones but laptops and ipods – could start to do so more.

2009 will not be the year of the semantic web

The semantic web holds enormous promise for journalism, but it’s still early days and even the best products are far from mass market. I don’t expect that to change any time soon. However…

In 2009 Google will look more vulnerable than ever

Google has been fiddling with its successful formula, trying to keep users within its verticals and getting greedy for user data. It is weakest on local search and semantic search and both those areas should see a lot of development in 2009. In 2010, however, Google will probably simply buy the best competitors.

2009 will see social media getting lean – and mean

Social media startups who do not want to join Pownce on the scrapheap will stop developing extra features, trim others, and focus on their core business. Oh, and they’ll be under increasing pressure to actually start coming up with business models too, which means more advertising (if they can sell it), more e-commerce, and less stuff for free. All of which will mean less innovation, fewer users and startups without deep pockets joining Pownce on the scrapheap.

2009 will see a lot of thinking and little action

All those redundant journalists, publishers, developers, and estate agents will have plenty of time to reflect on how their industries are changing, to play around with online tools, meet people online and offline, and come up with ideas on where to go next.

They’ll be doing this in an environment where funds are beginning to appear that enable them to act on those. In the UK at least there is £1billion from NESTA, £50m from 4iP, £1m from the Scottish government and various other pots of money aimed at maintaining economic growth.

So by 2010, when the bids have been put in, funds released, and pilots completed, we should see some very interesting new media indeed.

PHP Freelancer

Semantic journalism is a vision for the future of journalism. As the writer works on her article, her computer would gather data on the matter, from pictures to other articles to assessing global opinion trends. It would read through the Wikipedia pages of a given theme and summarize key concepts. A semantic algorithm would bring a selection of the most authoritative people on a subject.

The journalist is left with what she does best: checking and analyzing the data.

That means avoiding the pitfalls of redundant news content. That means escaping the trap of writing about topics without having a clue of what’s at stake. That means interviewing people who do things rather than those who talk about it.

This article is the first of a 4-part series. We’ll explore semantic hacks for newsgathering, writing and publishing in the coming weeks.

Part 1: Semantics today: What revolution?

Semantic journalism is closely related to the semantic web. The latter is a tidal wave redesigning the web since the early 2000′s, the motto of which is to make a webpage readable for machines. XML and RDF are the key words, Tim Berners-Lee the guru.

Now, having machines precisely understand the meaning of a story is another matter. Querying a database in natural language has been done since the 1970′s. Concretely, it means typing ‘What is the temperature in London?’ and seeing the machine display ‘20°C’.

But since the 1970′s, little has improved. Put simply, the computer reads the sentence, identifies a few words, their syntactical function and runs through a database to pick relevant information. Each word is given a meaning from the multiple senses it can carry.

In the example above, the computer can tell that ‘temperature’ is not referring to Sean Paul’s hit from the sentence’s structure. Then, it asks the database containing weather-related data for the current temperature in London.

Semantics rapid evolution has to do with Moore’s law and its army of escorting laws, all of which say that it’s getting cheaper to store and access data. Semantic applications can add more meanings to each word. Eventually, a semantic app will know that Temperature is also a 1937 movie. With a large enough database, it can store an almost infinite amount of temperature-related data.

However, when Sean Paul says that he ‘got the right temperature fi shelter you from the storm’, a computer will have a hard time understanding that there’s no actual shelter and no storm, no matter how many databases it commands. The key is to know that it’s a lush R&B song.

Some researchers argue that the traditional approach will not solve the semantic conundrum, no matter how much processing power is unleashed. Instead of a stratified method, where the program identifies the grammatical syntax, then the different possible meanings of each word, they favor a ‘what’s going on’ approach (they call it dynamic sense building, as opposed to compositional sense computing, in the words of semanticist Bernard Victorri).

In a paper, Daniel Kayser (full disclosure: that’s my dad) and Farid Nouioua explain that when a computer reads the sentence The truck in front of me braked suddenly, the key to extracting meaning isn’t in any of the words, but in knowing what is not said.

The semantic field for the word ‘car’, according to Sabine Ploux’s very cool semantic altlas

What the sentence actually means does not come by putting together the sense (as found in a dictionary) of each of its words. You need to know a lot about ordinary driving situations to grasp what any reader would find easily (e.g. the risk of accident was high). The knowledge required is not to be found in any dictionary or encyclopaedia, as thick as it might be. They argue that sense doesn’t come from what’s written, but from what’s assumed and left unwritten.

Semantics did not dramatically improve over the last decade. Automated summaries, for instance, a problem that has kept semanticists busy for the past 40 years, are still not expected for a distant future. Worse, it’s hard to see any technological lock that could, if broken, propel semantics into a higher gear.

In the coming weeks, the Online Journalism Blog team will test all kind of semantics apps that could help journalists. We’ll try to separate semantic snake-oil from genuinely innovative apps and discuss the value semantics can add. Stay tuned!

By Nicolas Kayser-Bril

PHP Freelancer