But first let me say that the #1 problem facing journalists is not the availability of mashable data, but the availability of the piece of data or dataset they WANT to investigate. For every RSS and XML feed there are hundred pieces of crucial data, hidden behind legacy systems, clumsy web front-ends, members only login windows and proprietary formats.

There is a cottage industry of websites on sex offenders and crime records. Why? because legislation made it necessary to make that data available. But what about all the important government, business and private sector data that SHOULD be public, but that don’t have the prime-time news appeal to make them so?

People in our industry (the web scrapers) have been a pariah in the web community. But there are those of us who are dying to collaborate with the journalists, academia and the public sector to put our services to positive uses. I spend a few hours of my day on hold on the phone, waiting for “permission to scrape” some public data or another.