With local elections looming this is a great 20-30 minute project for any journalist wanting to create an interactive Google map of council ward boundaries.

For this you will need:

• A Google account with Docs
• A Scraperwiki account
• Access to webspace to host an html file

Firstly we want to scrape the council ward geometry data held by MaPit by mysociety.org and spit it out into a CSV file format that is compatible with Google’s mapping tools.

Getting the ID for the council ward

Go to the MaPit homepage and use the postcode search for a point in the town/city you want the ward data for. In the example I’ve searched using a Preston postcode.

Then in the results page find the council you want data for and note down the id number next to it.

Adapting a scraper to scrape that council ward’s geometry

Now login to your Scraperwiki and visit this page for reclosedev’s council ward scraper. Click ‘Copy’ and you’ll be taken to a code editor page.

On Line 10 of the code you should see:

10: PARENT_ID = 2366  # Preston City Council

Change the 2366 value to the MaPit id of your council and change Preston City Council (anything after # is a comment and isn’t important to the code but it is useful to keep track of what you’re scraping).

Once you’d done this hit ‘Save Scraper’  the ‘Back to scraper overview’. This will take you to your own Scraperwiki page where the scraper is saved. It would be useful at this point to click on the pen symbol next to the scraper name and rename it ‘Your Council Ward geometry data’.

Then click RUN (or CTRL+R) to run your scraper, and wait a while for it to complete (usually no more than a couple of minutes).

When it has completed running, click Back to scraper overview (upper right) and on the section titled This scraper’s datastore click the swdata tab and you should see something similar to this:

The first column contains the shape/geometry data of the council ward and the third column contains its name. Does it look right? Then hit download (in the upper right of this image) and choose CSV as the format.

CSV or comma separate values is an open table format readable by Excel, OpenOffice and Google Docs and any text editor.

We’re going to use Google Fusion Tables to convert the data to a map.

Mapping the data

Head over to your Google Docs account (or Drive if you’ve been switched over) and hit Create > Table. When it asks you to import a new table choose and upload the CSV file you downloaded. On the next page you’ll be asked to specify the columns to import, leave this page as default and click Next.

On the next page name your table and attribute the data to the MaPit. Click Finish. We’re nearly there!

Under the Visualize menu item click Map and you’ll probably see a red blob similar to this.

If you want to make it prettier you can style this clicking ‘Configure styles’  and changing the settings for ‘Polygons‘.

When you’re happy with the settings for that click inside one of your council wards. You should get a window like this:

Not all that informative is it? To make it a bit more useful click ‘Configure info window’ then select the ‘Custom’ tab, delete what’s there and enter this code.

<div class='googft-info-window' style='font-family: sans-serif'>
 <h2><a href="http://mapit.mysociety.org/area/{id}.html" target="_blank">{name}</a></h2><br>
 </div>

Which will give you a link to the MapIt within the information box:

The final step is to embed this map in your webpage.

First click ‘Share‘ in the top right corner of the table page (not the Google+ sharebox) and change to either Public or Unlisted. Then hit File > About and note down the Numeric ID.

Go to the FusionTables LayerWizard and enter this id in the first box. For location column select ‘kml‘ then hit ‘Put layer on map‘.

Optional: Click ‘Add a search feature‘ and ‘select based search‘. For ‘Select Label‘ enter Search by ward name and for ‘Column to query‘ select ‘name’. Then add. What this allows you to do is to easily select the ward you want to view.

Zoom in to the Preview map to how you want the map displayed. Then copy and paste the code generated below into an html editor or plain text editor of your choice.

Replace lines 4-6

<style>
 #map-canvas { width:500px; height:400px; }
 </style>

with this

<style type="text/css">
 html { height: 100% }
 body { height: 100%; margin: 0; padding: 0 }
 #map-canvas { height: 100% }
 </style>

Then cut line 43

<div id="map-canvas"></div>

and paste it at the end of the page between </div> and </body> like so…

Save the file with the .html extension and try opening it in your browser. It should look a little like this.

Upload this page to your webspace. Now you can either link to it or embed it in an iframe. This sample code should work for most purposes.

<iframe id="ifrm" src="http://your.map.page/here.html" width="NUMBEROFPIXELS" height="NUMBER OF PIXELS">Your browser does not support iframes.</iframe>

Ta da!

This is really just scratching the surface of what Scraperwiki and Fusion Tables can do but I hope it served as an easy-ish introduction to them both. If this tutorial did not work for you or if you have any questions then leave a comment and I’ll help out as much I can.

PHP Freelancer

I’ve contributed to a “free, open-source book that aims to help journalists to use data to improve the news” – and it will be has now been published online tomorrow (Saturday 28th April)

The Data Journalism Handbook was coordinated by the European Journalism Centre and the Open Knowledge Foundation (in particular Liliana Bounegru), and includes contributions from:

“Dozens of data journalism’s leading advocates and best practitioners – including from Australian Broadcasting Corporation, the BBC, the Chicago Tribune, Deutsche Welle, the Guardian, the Financial Times, Helsingin Sanomat, La Nacion, the New York Times, ProPublica, the Washington Post, the Texas Tribune, Verdens Gang, Wales Online, Zeit Online and many others.”

The book will be is available for download at datajournalismhandbook.org under a Creative Commons Attribution ShareAlike License. There will also be a printed and e-book version published by O’Reilly Media.

PHP Freelancer

Following my previous posts on the network journalist and community manager roles as part of an investigation team, this post expands on the first steps a student journalist can take in filling the data journalist role.

1: Brainstorm data that might be relevant to your investigation or field

Before you begin digging for data, it’s worth mapping out the territory you’re working in. Some key questions to ask include:

• Who measures or monitors your field? For example:
• regulators and inspectors
• charities (try searching by keyword on the Charity Commission or OpenCharities)
• campaigning groups
• central government (the department and/or agency responsible, e.g. Ministry of Justice, BIS, etc. – there may also be specific ministers)
• local government (local authorities or primary care trusts, police force, etc.)
• select committees (browse parliamentary research indexed here, or try a specific search)
• general statistical/audit bodies such as ONS or the Audit Commission.
• Where is spending recorded? This might be at both a local and national level.
• What are the key things that might be measured in your field? For example, in prisons they might be interested in reoffending, or overcrowding, or staffing.
• Can you find historical data?
• What data do you need to provide basic context? e.g.
• Where – addresses for all institutions in your field (e.g. schools, prisons, etc.)
• Codes – often these are used instead of institution or area names
• Who – names of those responsible for particular aspects of your field
• Demographics – the distribution of age, gender, ethnicity, industries, wealth, property or other elements may be important to your work
• Politics – who is in charge in each area (local authority and local MP)
• How could you collate data that doesn’t exist? E.g. public awareness of something; or how the policies of different bodies compare, etc.

Sometimes the simplest and quickest way to find out these things is to pick up the phone and speak to someone in a relevant organisation and ask them: what information is collected about your field, and by whom?

You can also make content from this process of research: post a guide to how your field is regulated and measured (and what information isn’t); who’s who in your field - the regulators, monitors, politicians and bodies that all have a hand in keeping it on track.

2. Learn advanced techniques to obtain that data

Once you’ve mapped it all out you can start to prioritise the datasets that are most relevant to your particular investigation. You may need to use different techniques to get hold of these, including:

• Advanced search techniques (limit by filetype:, site:, etc.)
• Simply picking up the phone to call the relevant department (try to get as much detailed data as possible rather than aggregate, i.e. very general, figures)
• Using FOI requests

• Converting PDFs into spreadsheets
• Scraping

Again, you can make content from this process, for example: “How we found…” or “Why we’re asking the MoJ for…” (with a link to the FOI request) or “Get the data” (here’s how to publish data online)

The flow chart below (from this previous post) helps guide you to the relevant techniques for your data:

3. Pull out the parts of data relevant to your field/investigation

For example:

• If the data covers every region, pull out the parts that apply to your locality, or how that compares to other areas (space), or to previous data (time)
• Look at the particular issue(s) that interests you in the data, e.g. a particular crime out of many, or a particular indicator. How does that compare across space (regions) or time?

4. Add value to the data

Here are just some suggestions. You can use one or many:

• Combine datasets - e.g. one may have school ratings; another may have the addresses of all schools, or their local authority
• Convert data – this amounts to much the same thing, but for example: postcodes are more useful when converted into lat/long coordinates (likewise easting and northing)
• Find out how the data was collected and/or measured (put simply: pick up the phone and ask)
• Get an independent expert perspective on the data
• Compare the data with official claims or spin - does it really back those claims up?

• Compare the data with reports from elsewhere – is anything missing?
• Unpick jargon and definitions (here’s an example of James Ball unpicking different work experience schemes)
• Add a search and filter interface

Any of these provide useful opportunities for posting new content with the new contextual information (e.g. “How the data on X was gathered“) or new combined data (“Now with QOF data“) or the issues that they raise (“Why schools data may be worthless“).

5. Communicate the story in the data

I’ve written separately about the different ways of communicating data stories, so you can read that here. In short, human case studies are helpful, and visualisation is often useful.

And it’s at this point that you can also link to the further detail provided in all the content you’ve written in the previous 4 steps: How you got the data, the wider context, the specific data that’s of interest, the more detailed expert analysis or background, and so on.

PHP Freelancer

Data is so central to the decision-making that shapes our countries, jobs and even personal lives that an increasing amount of data journalism involves scrutinising the problems with the very data itself. Here’s an illustrative list of when bad data becomes the story – and the lessons they can teach data journalists:

Deaths in police custody unrecorded

This investigation by the Bureau of Investigative Journalism demonstrates an important question to ask about data: who decides what gets recorded?

In this case, the BIJ identified “a number of cases not included in the official tally of 16 ‘restraint-related’ deaths in the decade to 2009 … Some cases were not included because the person has not been officially arrested or detained.”

As they explain:

“It turns out the IPCC has a very tight definition of ‘in custody’ –  defined only as when someone has been formally arrested or detained under the mental health act. This does not include people who have died after being in contact with the police.

“There are in fact two lists. The one which includes the widely quoted list of sixteen deaths in custody only records the cases where the person has been arrested or detained under the mental health act. So, an individual who comes into contact with the police – is never arrested or detained – but nonetheless dies after being restrained, is not included in the figures.

“… But even using the IPCC’s tightly drawn definition, the Bureau has identified cases that are still missing.”

Cross-checking the official statistics against wider reports was key technique. As was using the Freedom of Information Act to request the details behind them and the details of those “ who died in circumstances where restraint was used but was not necessarily a direct cause of death”.

Cooking the books on drug-related murders

Cross-checking statistics against reports was also used in this investigation by Diego Valle-Jones into Mexican drug deaths:

“The Acteal massacre committed by paramilitary units with government backing against 45 Tzotzil Indians is missing from the vital statistics database. According to the INEGI there were only 2 deaths during December 1997 in the municipality of Chenalho, where the massacre occurred. What a silly way to avoid recording homicides! Now it is just a question of which data is less corrupt.”

Diego also used the Benford’s Law technique to identify potentially fraudulent data, which was also used to highlight relationships between dodgy company data and real world events such as the dotcom bubble and deregulation.

Poor records mean no checks

Detective Inspector Philip Shakesheff exposed a “gap between [local authority] records and police data”, reported The Sunday Times in a story headlined ‘Care home loses child 130 times‘:

“The true scale of the problem was revealed after a check of records on police computers. For every child officially recorded by local authorities as missing in 2010, another seven were unaccounted for without their absence being noted.”

Why is it important?

“The number who go missing is one of the indicators on which Ofsted judges how well children’s homes are performing and the homes have a legal duty to keep accurate records.

“However, there is evidence some homes are failing to do so. In one case, Ofsted gave a good report to a private children’s home in Worcestershire when police records showed 1,630 missing person reports in five years. Police stationed an officer at the home and pressed Ofsted to look closer. The home was downgraded to inadequate and it later closed.

“The risks of being missing from care are demonstrated by Zoe Thomsett, 17, who was Westminster council’s responsibility. It sent her to a care home in Herefordshire, where she went missing several times, the final time for three days. She had earlier been found at an address in Hereford, but because no record was kept, nobody checked the address. She died there of a drugs overdose.

“The troubled life of Dane Edgar, 14, ended with a drugs overdose at a friend’s house after he repeatedly went missing from a children’s home in Northumberland. Another 14-year-old, James Jordan, was killed when he absconded from care and was the passenger in a stolen car.”

Interests not registered

When there are no formal checks on declarations of interest, how can we rely on it? In Chile, the Ciudadano Inteligente Fundaciondecided to check the Chilean MPs’ register of assets and interests by building a database:

“No-one was analysing this data, so it was incomplete,” explained Felipe Heusser, executive president of the Fundacion. “We used technology to build a database, using a wide range of open data and mapped all the MPs’ interests. From that, we found that nearly 40% of MPs were not disclosing their assets fully.”

The organisation has now launched a database that “enables members of the public to find potential conflicts of interest by analysing the data disclosed through the members’ register of assets.”

Data laundering

Tony Hirst’s post about how dodgy data was “laundered” by Facebook in a consultants report is a good illustration of the need to ‘follow the data’.

We have some dodgy evidence, about which we’re biased, so we give it to an “independent” consultant who re-reports it, albeit with caveats, that we can then report, minus the caveats. Lovely, clean evidence. Our lobbyists can then go to a lazy policy researcher and take this scrubbed evidence, referencing it as finding in the Deloitte report, so that it can make its way into a policy briefing.”

“Things just don’t add up”

In the video below Ellen Miller of the Sunlight Foundation takes the US government to task over the inconsistencies in its transparency agenda, and the flawed data published on its USAspending.gov – so flawed that they launched the Clearspending website to automate and highlight the discrepancy between two sources of the same data:

Key budget decisions made on useless data

Sometimes data might appear to tell an astonishing story, but this turns out to be a mistake – and that mistake itself leads you to something much more newsworthy, as Channel 4′s FactCheck found when it started trying to find out if councils had been cutting spending on Sure Start children’s centres:

“That ought to be fairly straightforward, as all councils by law have to fill in something called a Section 251 workbook detailing how much they are spending on various services for young people.

“… Brent Council in north London appeared to have slashed its funding by nearly 90 per cent, something that seemed strange, as we hadn’t heard an outcry from local parents.

“The council swiftly admitted making an accounting error – to the tune of a staggering £6m.”

And they weren’t the only ones. In fact, the Department for Education  admitted the numbers were “not very accurate”:

“So to recap, these spending figures don’t actually reflect the real amount of money spent; figures from different councils are not comparable with each other; spending in one year can’t be compared usefully with other years; and the government doesn’t propose to audit the figures or correct them when they’re wrong.”

This was particularly important because the S251 form “is the document the government uses to reallocate funding from council-run schools to its flagship academies.”:

“The Local Government Association (LGA) says less than £250m should be swiped from council budgets and given to academies, while the government wants to cut more than £1bn, prompting accusations that it is overfunding its favoured schools to the detriment of thousands of other children.

“Many councils’ complaints, made plain in responses to an ongoing government consultation, hinge on DfE’s use of S251, a document it has variously described as “unaudited”, “flawed” and”not fit for purpose”.

No data is still a story

Sticking with education, the TES reports on the outcome of an FOI request on the experience of Ofsted inspectors:

“[Stephen] Ball submitted a Freedom of Information request, asking how many HMIs had experience of being a secondary head, and how many of those had led an outstanding school. The answer? Ofsted “does not hold the details”.

““Secondary heads and academy principals need to be reassured that their work is judged by people who understand its complexity,” Mr Ball said. “Training as a good head of department or a primary school leader on the framework is no longer adequate. Secondary heads don’t fear judgement, but they expect to be judged by people who have experience as well as a theoretical training. After all, a working knowledge of the highway code doesn’t qualify you to become a driving examiner.”

“… Sir Michael Wilshaw, Ofsted’s new chief inspector, has already argued publicly that raw data are a key factor in assessing a school’s performance. By not providing the facts to back up its boasts about the expertise of its inspectors, many heads will remain sceptical of the watchdog’s claims.”

Men aren’t as tall as they say they are

To round off, here’s a quirky piece of data journalism by dating site OkCupid, which looked at the height of its members and found an interesting pattern:

“The male heights on OkCupid very nearly follow the expected normal distribution—except the whole thing is shifted to the right of where it should be.

“Almost universally guys like to add a couple inches. You can also see a more subtle vanity at work: starting at roughly 5′ 8″, the top of the dotted curve tilts even further rightward. This means that guys as they get closer to six feet round up a bit more than usual, stretching for that coveted psychological benchmark.”

Do you know of any other examples of bad data forming the basis of a story? Please post a comment – I’m collecting examples.

UPDATE (April 20 2012): A useful addition from Simon Rogers: Named and shamed: the worst government annual reports explains why government department spending reports fail to support the Government’s claimed desire for an “army of armchair auditors”, with a list of the worst offenders at the end.

Also:

• This post on the lack of data on deaths from legal highs, by some of my students at City University.
• Sex trafficking: a story of data gone wrong, which is the source of the opening image for this post (by Lauren York, another student of mine)
• Chicago police crash reports are full of errors.

PHP Freelancer

OpenlyLocal are trying to scrape planning application data from across the country. They want volunteers to help write the scrapers using Scraperwiki - and are paying £75 for each one.

This is a great opportunity for journalists or journalism students looking for an excuse to write their first scraper: there are 3 sample scrapers to help you find your feet, with many more likely to appear as they are written. Hopefully, some guidance will appear too (if not, I may try to write some myself).

Add your names in the comments on Andrew’s blog post, and happy scraping!

PHP Freelancer

A must-read for any data journalist, aspiring or otherwise, is Simon Rogers’ post on The Guardian Datablog where he compares public and private sector pay.

This is a classic apples-and-oranges situation where politicians and government bodies are comparing two things that, really, are very different. Is a private school teacher really comparable to someone teaching in an unpopular school? What is the private sector equivalent of a director of public health or a social worker?

But if these issues are being discussed, journalists must try to shed some light, and Simon Rogers does a great job in unpicking the comparisons. From pay and hours worked, to qualifications and age (big differences in both), and gender and pay inequality (more women in the public sector, more lower- and higher-paid workers in the private sector), Rogers crunches all the numbers:

“[T]he proportion of low skill jobs in the private sector has increased, and the proportion of high skill jobs in the public sector increased to around 31% of all jobs by 2011, compared 26% of all private sector jobs.

“But, at the same time, people who are most highly qualified actually get paid worse in the public sector.

“… Public sector workers tend to be older … Average mean hourly earnings peak in the early 40s in both sectors. They decline slightly approaching retirement although the decline happens earlier in the private sector than in the public sector, possibly because the higher earners in the private sector are more likely to leave the labour market earlier.

“It also shows that if you’re older in the public sector, you get paid better than in the private sector.

“… [T]he bottom 5% of workers in the public sector earn less than £6.91 per hour, whereas in the private sector, 5% of workers earn less than £5.93 per hour.”

When you find yourself in an apples-and-oranges situation you can’t avoid, this is the way to do it. Any other examples?

PHP Freelancer

Here’s a solution to a problem that aspiring data journalists have come up against time and time again: how to quickly create a searchable interface to your dataset.

freeDive is quick and – if you can follow the wizard‘s instructions – easy too.

If you have a dataset that you want people to be able to search or filter, publish it on Google Docs, copy part of the URL into freeDive when it asks you for it, then decide which fields you want people to be able to see, and how they can search, or filter their search.

At the end you’re given some script which is the most likely point of struggle for some users. If you use a WordPress.com blog then it won’t work – and even a self-hosted WordPress blog will require the installation and activation of a plugin (helpfully linked on the freeDive site) and following those instructions (if you don’t know how to use custom fields check out the explanations here and here).

If you’ve tried it on Blogger or another platform, let me know about it.

Meanwhile, here’s a test of the interface in action – the technology is still in ‘alpha’ so expect some bugginess.

PHP Freelancer

Guest post by Duarte Romero

Since the start of the year the Argentinian newspaper ‘La Nación’ has been publishing ‘Nación Data’, a blog dedicated to data visualization, interactive projects and especially, all the news related with data journalism.

During this time they have been posting interviews with experts from the community, reporting popular events such as NICAR and sharing the most innovative pieces made by other newspapers.

The multimedia development manager of ‘La Nación’, Momi Peralta, pointed out that their main goal so far is to release as much data as they can.

It is important to highlight that in Argentina there is not a Freedom of Information Act, so most of the statistics and spreadsheets they use are made by their own journalists.

Peralta believes that this data journalism will help to open up information in her country:

“One of our aims [is to] produce good information in Argentina, opening new data and creating visualizations. Each [set of] data that is published means that more knowledge is released.”

Last Friday they launched a new platform, titled, simply: ‘Data’. This tool is used to publish those indices that are collected and refined by journalists and allow the possibility to download them as a spreadsheet or share on social networks.

On the dashboard you can find data about topics as different as traffic collisions, carbon dioxide emissions and the activity of the parliament.

Some are used in current stories but others are just uploaded for the interest of the public.

Peralta hopes that this project will help empower the open data movement in Argentina, so she wants to use the blog to keep in touch with the community.

She likes to compare data driven journalism with constructing a building. Data are the materials and tools are the technology, but if you want to make something you will need to order them first.

Finally, she points out that you may build things on your own but if you have contributions from other people, the result will be better.

Learning by doing

None of this would be possible if ‘La Nación’ had not made a great effort to train its workers in the main tools used in data journalism. The team that is in charge of this area attended several courses on Excel, Tableau, scraping and other techniques.

During the NICAR conference they played a video (embedded below) about this learning process to share their improvements and achievements with the community.

This enthusiasm was recognized by some of the most prestigious institutions of the online journalism community. In the last two years, ‘La Nación’ has received two ONA awards and one EPPY thanks to their innovative projects.

Furthermore, last week the Argentinian paper was chosen by the Knight-Mozilla OpenNews for a partnership aimed at driving open source innovation in news.

This award is shared with The New York Times, ProPublica and Spiegel Online and it means that ‘La Nación’ is the first paper written in Spanish to receive this prize.

Not many Spanish-language countries have developed a truly open data policy, but with more examples of media organisations working with data, the hope is that public institutions will be forced to improve their transparency and openness.

PHP Freelancer

Over the weekend Fleet Street Blues has had a bee in its bonnet about the “pretence” of data journalism and Saturday’s Guardian front page: “Half UK’s young black men out of work“.

This, says FSB, is a lie that demonstrates the ”pretence” that “‘crunching the numbers’ is somehow an an abstract, scientific, mathematical task”.

There are two problems with this: the first is that I’ve never heard a data journalist make this claim; and the second is that the ‘lie’ does not come from a data journalist (they generally don’t write headlines). It is, in short, a straw man.

The story itself is, however, perfectly valid. While FSB points to the exclusion of students, for example, The Guardian’s story mentions that early on.

It’s fair to say that those who are economically inactive should not be included in unemployment figures. Indeed, the Datablog post which expands on the data does a very good job in explaining how that activity is mentioned:

“Youth unemployment figures are always slightly odd, and as with many things in life, it’s students that get the blame.

“Students can be counted in three different ways: a full-time student doing an evening job in a bar counts as employed. A student who wants bar work but can’t get it is unemployed. A full-time student who’s not topping up his income with a job (and isn’t trying to) is economically inactive.”

Fleet Street Blues uses the raw data published by the Datablog to highlight a number of other ways of interpreting the data, all of which are interesting – and in fact, I’ll probably use them in future as an example of how the same data can tell many different stories.

More than one story

But again, this proves nothing about the ‘pretence’ of data journalism. All it proves is that there is more than one story to be found in a dataset, and that journalists will pick the one that is most newsworthy for their particular market.

In fact, not only journalists, but politicians, PR staff, marketers, scientists, lobbyists and anyone else who wants to tell a story.

It’s because of this that data journalism is not something which should be snootily written off as a “fad”. Data is important. Journalists need to be able to interrogate it and find the stories that are not being told.

That is exactly what The Guardian have done. Yes, the headline could be more accurate* – but how many times has a headline writer omitted key details due to the limitations of space (on every type of story)? And yes, as one FSB commenter points out, the inclusion of whole numbers would have added further context.

But the irony is that it’s precisely because The Guardian isn’t trying to pretend to be ‘The Only Truth’ that FSB and its commenter can interrogate the data, and that the reader can understand the subtleties in how data is gathered and classified.

If there is a pretence about data journalism, it is a wider one: a  belief in society that somehow numbers equate to truth. A belief which is exploited by politicians but which is coming – and should come – under increasing scrutiny from journalists. (The story, for example, began as a column by a Labour politician in The Guardian, fact-checked by Channel 4 News, and followed up by The Guardian’s journalists)

The more good data journalism we have, the less that anyone – including journalists – can pretend to the idea of a “scientific” process.

*(Notably, the online version includes a second headline which is clearer: “Unemployment rate for black 16 to 24-year-olds available for work now double that for white counterparts, ONS data shows”)

PHP Freelancer

There’s a persuasive argument being made by Francis Irving and Rufus Pollock in a joint blog post about the growth of data management systems – the ‘DMS’ to content management systems’ ‘CMS’:

“Just as then we wrote HTML in text files by hand and uploaded it by FTP, now we analyse data on our laptops using Excel, and share it with friends by emailing CSV files.

“But it reaches the point where using the filesystem and Outlook as your DMS stretches to breaking point. You’ll need a proper one.

“Nobody really knows what a proper one will look like yet. We’re all working on it.”

Their post lists what a DMS needs to do and the companies already trying to solve the ‘DMS problem’ from different directions: a list which includes Google Docs (“coming from the web spreadsheet direction”), the data social network BuzzData, visualisation tool Tableau, data marketplaces, operating systems, Scraperwiki, and PANDA (“making a DMS for newsrooms”)

It’s a well-drawn picture from an angle which I haven’t seen before. Certainly, a number of news organisations are trying to reduce the friction of producing content for different platforms by ‘atomising’ it in data-driven production processes (where a piece of content might be assembled and presented differently depending on the platform it is accessed through, for example), and their internal systems can probably be added to the list above.

What do you think? Is this a problem that’s being addressed in your own organisation?

PHP Freelancer