Searching is the most popular activity online after email. It is the prism through which we experience a significant proportion of the world’s information – from news and information about our community, through to health information, commerce, and just about anything that has a presence online.

Search Engine Society takes a critical look at search engines, how they work, the techniques used to manipulate them – from gaining better rankings to censorship, and the implications for privacy and democracy.

Chapter one looks at the development and workings of search engines, from the once-essential directories of Yahoo! and the citation-based algorithms of Google that now dominate the search landscape, through to lesser-known players such as social bookmarking service Delicious which relies on user-generated ‘folksonomies’ to organise material, and specialised regional and ‘vertical’ search engines like the French language Voila or the genetic materials search engine The Bioinformatic Harvester. This is situated within a wider discussion of information retrieval histories from the Library of Babylon onwards – and touches on recent moves into geospatial, mobile, social and semantic search.

Balancing that focus on technology, the following chapter focuses on users, looking at how people search. Search behaviours vary widely between users and between searches – Halavais discusses research that showed how many users simply add ‘.com’ to a word as the start of their search, while others use a ‘shopping mall’ approach of going direct to the likes of Wikipedia and the Internet Movie Database (which also contain search facilities). Using a search engine, Halavais argues, is only one method of search, and search is “not only an iterative process, but one that is rarely linear and requires seeking out the concepts that surround a problem or question. In other words, the query and search strategy is likely to change as more information becomes available.”

Search as ‘re-finding’

Halavais also emphasises the importance of ‘re-finding’ – “not as a sub-set of finding, but the other way around” – indeed, this is the basis of social bookmarking services like Delicious and Digg that allow the user to store and label (‘tag’) webpages for later retrieval, as well as searching for webpages that have been given similar tags by other users.

Power law distribution patterns famously recur throughout the web and in the third chapter Halavais looks at how this affects search results. With Google’s rankings relying so strongly on how many links point to a particular page, it is important to look at how those links are distributed. The fact that highly linked pages are likely to attract ever more links – what Huberman calls “preferential attachment” – leads to the “chunky” nature of the web – in concrete terms the dominance of websites like those of the BBC and Guardian; a quality which, Halavais argues, Google’s PageRank technology ‘calcifies’.

But when Google tweaks its search engine algorithms to attempt to improve results, it can have enormous consequences for organisations dependent on their rankings in search results. Halavais uses the example of Skyfacet.com and Answers.com which saw sales and visits drop by 17% and 28% respectively when they dropped off the first page of related Google searches. It is as if someone moved your shop from the main high street to an industrial estate. In this context it is not surprising that search engine advertising accounts for the majority of online advertising spend.

Digital divides

Following up on those issues, the fourth chapter looks at implications for democracy on two sides: firstly, the division between winners and losers in the contest for public attention; and secondly, the division between skilled and unskilled users of search engines. Halavais is keen to highlight that division is nothing new:

“Current search engines, like communication technologies before them, contain both centralizing and diversifying potentials. These potentials affect the stories we tell ourselves as a society; and the way we produce knowledge and wisdom.”

In practice, these potentials are heavily weighted towards US sites:

“In the language of PageRank, US sites simply have more authority: more links leading to them … sites have existed longer in the United States, where much of the early growth of the internet occurred… Add to this the idea that early winners have a continuing advantage in attracting new links and traffic, and US dominance of search seems a foregone conclusion … the search engines do not merely reflect this authority, they help to reproduce it.”

Indeed, ranking systems that reinforce authority, says Halavais, are conservative in nature and comprise what Lewis Mumford, writing 40 years ago, called “authoritarian technics”.  But because of the unlimited size and reach of the internet compared to previous media technologies, it is not so simple:

“The current structure is a complex combination of a high degree of centralization at the macro-level, with a broad set of diverse divisions at the micro-level.”

Blogger as ‘search intellectual’

Interestingly, at this point Halavais introduces the blogger as a “search intellectual”, upsetting existing structures of authority on the web and acting as “a counterweight to the hegemonic culture of the search engines” in bringing otherwise overlooked material into the “circle of reputation and links that search engines tend to enforece”. The recent rise of Twitter in performing a similar role would be worth adding to that list.

Chapter 5 takes a broad look at censorship – “just another word for filtering” – while Chapter 6 looks at privacy – search engines as “databases of intentions” where even anonymised logs of what individuals are searching for can lead to people being identified. Chapter 7 revisits the rise of “sociable search” tools and folksonomy – where classification is created by a mass of users’ ‘tags’ rather than any centralised scheme, and ‘finding’ is a social act closely related to ‘sharing’.

The book closes with a roundup of the possibilities of future search and the factors that will influence that, from increasing digitisation of material to improved mapping and the possibilities of RFID tags (which makes objects a part of the web too). Semantic search – technology that understands the meaning of what you are searching for, or of relationships between objects – is the promise that lies forever ‘just over the horizon’, while sociable search offers a more likely immediate move.

As is natural, there are areas which have developed since this book was written and so are not tackled in depth – most notably real-time search. The rise of Twitter and the ability to search through what people are talking about ‘right now’ represents such serious competition to Google that it introduced the first major new features to its homepage in years. Wolfram Alpha – the “computational knowledge engine” that made newspaper front pages this year – is not even mentioned.

But those are incidental issues in what is an important book. Halavais manages to acknowledge the dominance of Google without being distracted by it, and gives due attention to non-Western tools and services not commonly seen as search tools. He avoids the pitfalls of technological determinism and manages to distinguish between top-down domination and bottom-up diversity. What emerges is a sophisticated picture of power in flux. “Search engines are interesting to the person who wants to understand the exercise of power in the information society,” Halavais writes in the his conclusion. “In an era in which knowledge is the only bankable commodity, search engines own the exchange floor.” The more readers understand this exchange floor, the better we can exchange and interrogate what information we possess.

A shorter version of this review will appear in Journalism

PHP Freelancer

Interesting research from (ahem) Yahoo highlighted in Search Engine Land suggests that there are actually areas where users don’t go straight to Google – and makes encouraging reading for local news organisations.

Conducted earlier this year among 2,500 consumers, the research looked at how consumers used search in choosing local services and providers in five vertical categories. This is how it broke down:

• Vocational Education – 67 percent of respondents used a major search engine compared to 41 percent who used local search
• Healthcare – 53 percent used insurance provider directories while 44 percent relied on a search engine
• Real Estate – 51 percent used a real estate vertical search engine versus 44 percent who used a major search engine
• Legal – 36 percent of respondents referred to a search engine while 25 percent used the internet yellow pages
• Home Contracting – 36 percent used a search engine while 26 percent used the internet yellow pages

“Consistent across categories was word of mouth. Yahoo found in every category that 75 percent of consumers relied on recommendations from friends and family during the research process. This also translated into interest in online reviews: “Yahoo’s study found that local merchants who encourage customers to post online reviews and ratings benefit from the same kind of impact seen through word-of-mouth endorsements from family and friends.””

That last point is key: content.

The post has a lot more detail on what consumers actually did – and if you want to do something about local search optimisation, this recent panel discussion should offer some useful pointers.

Now to see if news organisations and advertisers are listening…

PHP Freelancer

Every week I come across some web-based service that makes it possible to do in a few clicks what a year ago would have required anything from a day of fiddling to months of developer time. Today’s tool is one of a number offered by Dapper, a company which aims to “make it easy and possible for anyone to extract and reuse content from any website.” The tool is the Facebook Appmaker.

It took me around 30 minutes to create a Facebook app which would allow Facebook users to display a feed from my blog. You first need to create a ‘Dapp‘ (basically a custom RSS feed), and then use the Facebook Appmaker to create an app based on that. It’s basically a lot of clicking, and copying and pasting according to the instructions.

Given the ease of use, this means there’s now no excuse for news organisations not to offer Facebook apps for any aspect of their news service – including pages which do not already have RSS feeds.

And here’s the real beauty – a basic app for your news feed is only the beginning.

Because you can create custom RSS feeds from anywhere, you can create custom apps that do any number of things. Similar to Yahoo! Pipes, you can create mashups – or ‘scrape’ web content from pages that do not offer RSS feeds (there’s even a Firefox extension that will detect RSS feeds created by other people for any site you’re on).

So, for example, although Google does not offer an RSS feed of searches (Google News and Blog Search do), I created this RSS feed for the search ‘new online magazine launches’ – and then a Facebook app based on that. You could combine any number of RSS feeds, and you can add filters too. From that you can create your own Facebook app, Flash widget, Google Gadget…

Stop there. Do you understand the implications of this? You are not just publishing content any more – you are creating a bespoke news service, built on content from elsewhere, using your journalistic knowledge and editorial experience. Do you know what you have done?

You’ve become Google News.

Read more ‘Something for the weekend’ posts

PHP Freelancer

Virtual intern Natalie Chillington rounds up last week’s online journalism-related news

Google

• Google will announce a new metrics tool to measure web site audience, to rival current power players Nielsen and ComScore.

• Lots of debate over whether Google is making us stupid

WordPress

• Puffbox.com announces it will be sponsoring WordCamp UK in July,bringing together around 100 devotees of WordPress in Birmingham for aweekend of code and conversation.

Twitter

• Debate over whether you should use Twitter for live blogging at a conference.

• The same people behind the tech news site WebProNews have launched Twellow, a new search tool and directory for finding Twitter users

Social

• Facebook launched new mini-feed comments

• MySpace partners with NBC News and msnbc.com to find a citizen journalist to cover national conventions of the Democratic and Republican parties later this summer in the US.

• Nokia has acquired German social networking service Plazes,expanding it’s mobile internet services. Plazes offers a location-basedservice, with friends posting messages about their location and whatthey are doing from their mobile or PC – sounds like Twitter.

• Domestic Chinese mobile phone service provider MICAT has announcedan agreement with social-networking website, MySpace. Together, theywill be launching “MySpace MICAT,” a social platform for Chinese mobile internet users.

• Facebook introduces Chinese-language versions of its social networking website, marking its first move into the world’s largest internet market.

Web

• Interactive map shows the aftermath of Parkersberg tornado

• YouTube is home to nearly half of web clips watched in the UK

• Magazine publisher IPC Media buys gaming website Mousebreaker in an effort to increase its access to an audience of young male internet users.

• Yahoo to announce the results of a major staff reorganisation as internet firm tries to reassure investors and staff that it can improve its performance and competitiveness.

• The Internet Corporation for Assigned Names and Numbers (ICANN) was due to vote on new internet domain names this week. It is thougth that from early 2009, users will be able to acquire generic addresses such as .love

Media

• Debate over ‘Pay-per-view’ future of online journalism

• House of Lords report says growth of news sites is harming investment in news gathering

• Sky News Online prepares to re-launch with a new look

• CNET is also getting a new look

• The Associated Press will begin rolling out details of its new pricing to members this week, a plan that will return up to $21 million to its U.S. member newspapers.

• Debate over the usefulness of blogging to journalists continues

• BBC launches new revamped online catch-up service, iPlayer with integrated radio shows.

• Trinity Mirror launches a series of branded YouTube channels for the video content of its regional newspapers.

• ITN has signed a ‘six figure’ deal with Al Jazeerato make 800,000 hours of archived video content available to thebroadcaster. As part of the deal, the network and production companieswill have access to Channel 4, Reuters, Granada and ITN.

• The Independant hires the former head of the prime minister’s digital PR operation Jimmy Leach, to lead online development.

• Some journalists still opposed/scared of the internet

Other

• Freelance journalist Sean Langan released after being kidnapped by a group associated with the Taliban in a three-month ordeal.

PHP Freelancer

Yahoo! have released a family of Reputation patterns:

“They don’t tell you how to lay out a page or where to put an interactive widget. Instead, they address how to design a reputation system for your social software.”

Why is this important? The patterns are a wonderful resource for any news organisation looking to plan a community element in which reputation performs a role. In my experience, reputation systems are pretty important in encouraging users to keep coming back to your online community – you could argue, for instance, that the number of friends in Facebook or followers in Twitter is one simple example. Plurk more explicitly uses ‘karma’, as does (in a much better way) Slashdot (for more on Slashdot and karma systems I thoroughly recommend Gatewatching by Axel Bruns).

Yahoo say these are “the first of several collections of social-design related patterns that we’re working on,” so worth keeping an eye on what comes next.

PHP Freelancer

Allison White has written this wonderful roundup of last week’s news for the OJB. But now she’s got a job. Persuade her to do this again in the comments…

Google

-Announced no desire to create content and will respect copyright.

It added face-blur technology to its Street View mapping serivce to protect privacy. Also speculation from Groves Media on whether this technology is more of a threat to civil liberties than CCTV.

Microsoft

-Looking to limit the kinds of computers that can use their low-cost OS, making them poor computers even if they could be better and still be as cheap.

-The Microsoft/Yahoo struggle is still ongoing – rumor has it that Carl Ichan is buying up stocks in Yahoo to replace board members
who will sell to Microsoft. Seems he is replacing board members but his plans are not evident

Twitter

-Poynter did a good roundup of the Twitter coverage of the Chinese earthquake. Also gave news of situation and stories in Burma.

-Twitterfone is a new app for mobiles that allows users to call and leave a voice message, which is translated to text and then posted on Twitter. Has issues like knowing how to end the call.

-Funny meme on Twitter is finishing this sentence: “When Obama wins, …” Mine was “When Obama wins, college will be free and pizza will be subsidized”

Tech

-AP trying to get a button on the iPhone

-Effort underway to make clickstream tracking opt-in only in US. Marketers say it will be the death of their industry, but will most likely be like opting out of telemarketers.

-RIAA and MPAA still insist on using DRM on content bought to limit customer .Microsoft may also be planning to use a copyright ‘cop’ on the Zune for NBC. EA backs down on using DRM after user outcry at the idea

-Wikipedia to become a print edition, some debate about if the authors should get paid but Techdirt says it is non-controversial and they should not be paid.

-The FBI backs off of the Internet Archive – they wanted records of who used it now journalists can rest more at ease

-In the case against Ray Niro, writer of Troll Tracker blog, tried to prove he was not a “real” journalist and could not claim rights as one because he wasn’t trained, was bias and that he was a corporate mouthpiece. Defense took apart the claim of what it really means to be a journalist.

Nimbuzz moving in on Skype’s territory by making an app for both mobile and PC

Comcast bought Plaxo Inc., the social networking site. It turns online address books into online networks.

Other

Bush gave his first online interview Tuesday

Adrian Monck moved to WordPress http://adrianmonck.com/

Social

Quarter of social net users are using their mobiles to be social online in the UK.

Social Networks boom in Europe, projected to reach 107.4 million people in 2012. UK expected to be largest with 27.1 million projected.

How marketers are using social media and how often

Employers continue to ban Facebook and other social media sites at work without recognizing the possible benefits from these sites.

A new social site for health issues is iMedix, and although in beta is pretty poor in the community department. The people on it now are rather ‘happy-go-lucky’ as is some of the content which may not be appropriate for the kind of users it wants to attract (people with health problems)

-Social Networking Sites are now allowing people to basically export their data to other sites: MySpace with a few other partnered sites, Facebook to anyone, and Google to anyone. Although Facebook is closing access to journalists and is looking to take off its 5000 friends limit.

Web

Two sites, Newstrust and Newscred, have set up a system to rate the credibility of Web sites. Buzz Machine says it won’t work because there is just too much bad stuff and too easily discredited good stuff.

LiveNewsCamera.com allows viewers to see the news as it’s happening, uncut. Pope speeches, Iraqi parliament hearings, and the like.

PRWeb now allows users to search for their press releases, although this may bring fewer balanced results in a search

Harry McCracken from PC World has left to start his own tech web site.

A lot of talk going on about Seesmic: consensus seems to be that it is rough now with great potential and it is hard to imbed on some sites.

MSNBC has created a new widget to scroll news and also two games to see main news headlines. Not sure if they’re effective, but they’re trying

Media

Time Warner may be looking to unload that do not transfer well to web (i.e. Coffee table mags)

Using video to truly enhance a story – not having it be a stand alone piece but a short snipit filled with things print can’t show, like emotion.

The BBC will be using the Parliamentary channel coverage to instead cover the Olympics

From Thoughts of Nigel: 3 ways China should handle Western media – give access to reporters, don’t freak out on negative coverage, and work on building a worldwide reputation

ReinventingClassifieds.com is looking to create a new business model to keep the classified section alive

Dutch free dailies are increasing circulation and so are free Italian papers

Irish paper group, River Media, appears to be in trouble and is looking for $16 million in help.

Guatemalan journalist, Jorge Merida Perez, shot dead in his home. He worked for the Prensa Libre.

From Adam Tinworth: The possibility of journalists being paid for increasing defined traffic online is being looked at, although still skeptically

The Liverpool Daily Post liveblogged the making of the paper Tuesday

The INMA drops “newspaper” from their title and adds “newsmedia”

Cablevision buys Newsweek over Rupert Murdoch, but it is also speculated Murdoch might end up with it in a few years anyways

Upcoming book Super Media has some chapters free online and discusses how journalism can save itself and the world through network journalism

Daily Mail site just pulled Flickr photos to write an article about fat pets without asking permission or attributing credit.

Boston NOW closed

Andy Bull writes on the blandness of sports journalism with the double speak and untruthfulness of players, teams and journalists

Wall Street Journal hires on chief technology officer, Sarabjit “Ruby” Walia

But London Evening just cut their media reporter, which is just the beginning of changes there they stated

PHP Freelancer