Tinkering With Scraperwiki – The Bottom Line, OpenCorporates Reconciliation and the Google Viz API
Having got to grips with adding a basic sortable table view to a Scraperwiki view using the Google Chart Tools (Exporting and Displaying Scraperwiki Datasets Using the Google Visualisation API), I thought I’d have a look at wiring in an interactive dashboard control.
You can see the result at BBC Bottom Line programme explorer:
The page loads in the contents of a source Scraperwiki database (so only good for smallish datasets in this version) and pops them into a table. The searchbox is bound to the Synopsis column and and allows you to search for terms or phrases within the Synopsis cells, returning rows for which there is a hit.
Here’s the function that I used to set up the table and search control, bind them together and render them:
google.load('visualization', '1.1', {packages:['controls']});
google.setOnLoadCallback(drawTable);
function drawTable() {
var json_data = new google.visualization.DataTable(%(json)s, 0.6);
var json_table = new google.visualization.ChartWrapper({'chartType': 'Table','containerId':'table_div_json','options': {allowHtml: true}});
//i expected this limit on the view to work?
//json_table.setColumns([0,1,2,3,4,5,6,7])
var formatter = new google.visualization.PatternFormat('<a href="http://www.bbc.co.uk/programmes/{0}">{0}</a>');
formatter.format(json_data, [1]); // Apply formatter and set the formatted value of the first column.
formatter = new google.visualization.PatternFormat('<a href="{1}">{0}</a>');
formatter.format(json_data, [7,8]);
var stringFilter = new google.visualization.ControlWrapper({
'controlType': 'StringFilter',
'containerId': 'control1',
'options': {
'filterColumnLabel': 'Synopsis',
'matchType': 'any'
}
});
var dashboard = new google.visualization.Dashboard(document.getElementById('dashboard')).bind(stringFilter, json_table).draw(json_data);
}
The formatter is used to linkify the two URLs. However, I couldn’t get the table to hide the final column (the OpenCorporates URI) in the displayed table? (Doing something wrong, somewhere…) You can find the full code for the Scraperwiki view here.
Now you may (or may not) be wondering where the OpenCorporates ID came from. The data used to populate the table is scraped from the JSON version of the BBC programme pages for the OU co-produced business programme The Bottom Line (Bottom Line scraper). (I’ve been pondering for sometime whether there is enough content there to try to build something that might usefully support or help promote OUBS/OU business courses or link across to free OU business courses on OpenLearn…) Supplementary content items for each programme identify the name of each contributor and the company they represent in a conventional way. (Their role is also described in what looks to be a conventionally constructed text string, though I didn’t try to extract this explicitly – yet. (I’m guessing the Reuters OpenCalais API would also make light work of that?))
Having got access to the company name, I thought it might be interesting to try to get a corporate identifier back for each one using the OpenCorporates (Google Refine) Reconciliation API (Google Refine reconciliation service documentation).
Here’s a fragment from the scraper showing how to lookup a company name using the OpenCorporates reconciliation API and get the data back:
ocrecURL='http://opencorporates.com/reconcile?query='+urllib.quote_plus("".join(i for i in record['company'] if ord(i)<128))
try:
recData=simplejson.load(urllib.urlopen(ocrecURL))
except:
recData={'result':[]}
print ocrecURL,[recData]
if len(recData['result'])>0:
if recData['result'][0]['score']>=0.7:
record['ocData']=recData['result'][0]
record['ocID']=recData['result'][0]['uri']
record['ocName']=recData['result'][0]['name']
The ocrecURL is constructed from the company name, sanitised in a hack fashion. If we get any results back, we check the (relevance) score of the first one. (The results seem to be ordered in descending score order. I didn’t check to see whether this was defined or by convention.) If it seems relevant, we go with it. From a quick skim of company reconciliations, I noticed at least one false positive – Reed – but on the whole it seemed to work fairly well. (If we look up more details about the company from OpenCorporates, and get back the company URL, for example, we might be able to compare the domain with the domain given in the link on the Bottom Line page. A match would suggest quite strongly that we have got the right company…)
As @stuartbrown suggeted in a tweet, a possible next step is to link the name of each guest to a Linked Data identifier for them, for example, using DBPedia (although I wonder – is @opencorporates also minting IDs for company directors?). I also need to find some way of pulling out some proper, detailed subject tags for each episode that could be used to populate a drop down list filter control…
PS for more Google Dashboard controls, check out the Google interactive playground…
PPS see also: OpenLearn Glossary Search and OpenLearn LEarning Outcomes Search
