A couple of days ago I got an email from Jennifer Okamato of the Dallas News, who had picked up on one my mashup posts describing how to scrape tabluar data from a web page and get it onto an embeddable map (Data Scraping Wikipedia with Google Spreadsheets). She’d been looking at live crime incident data from the Dallas Police, and had managed to replicate my recipe in order to get the data into a map embedded on the Dallas News website:
But there was a problem: the data being displayed on the map wasn’t being updated reliably. I’ve always known there were cacheing delays inherent in the approach I’d described, which involves Google Spreadsheets, Yahoo Pipe, Google Maps, as well as local browsers all calling on each other an all potentially cacheing the data, but never really worried about them. But for this example, where the data was changing on a minute by minute basis, the delays were making the map display feel too out of date to be useful. What’s needed is a more real time solution.
I haven’t had chance to work on a realtime chain yet, but I have started dabbling around the edges. The first thing was to get the data from the Dallas Police website.
(You’ll notice the data includes elements relating to the time of incident, a brief description of it, its location as an address, the unit handling the call and their current status, and so on.)
A tweet resulted in a solution from @alexbilbie that uses a call to YQL (which may introduce a cacheing delay?) to scrape the table and generate a JSON feed for it, and a PHP handler script to display the data (code).
I tried the code on the OU server that ouseful.open.ac.uk works on, but as it runs PHP4, rather than the PHP5 Alex coded for, it fell over on the JSON parsing stakes. A quick Google turned up a fix in the form of a PEAR library for handling JSON, and a stub of code to invoke it in the absence of native JSON handling routines:
//JSON.php library from http://abeautifulsite.net/blog/2008/05/using-json-encode-and-json-decode-in-php4/
include("JSON.php");
// Future-friendly json_encode
if( !function_exists('json_encode') ) {
function json_encode($data) {
$json = new Services_JSON();
return( $json->encode($data) );
}
}
// Future-friendly json_decode
if( !function_exists('json_decode') ) {
function json_decode($data) {
$json = new Services_JSON();
return( $json->decode($data) );
}
}
I then started to explore ways of getting the data onto a Google Map…(I keep meaning to switch to OpenStreetMap, and I keep meaning to start using the Mapstraction library as a proxy that could in principle cope with OpenStreetMap, Google Maps, or various other mapping solutions, but I was feeling lazy, as ever, and defaulted to the Goog…). Two approaches came to mind:
– use the Google static maps API to just get the markers onto a static map. This has the advantage of being able to take a list of addresses in the image URL which then then be automatically geocoded; but it has the disadvantage of requiring a separate key area detailing the incidents associated with each marker:
– use the interactive Google web maps API to create a map and add markers to it. In order to place the markers, we need to call the Google geocoding API once for each address. Unfortunately, in a quick test, I couldn’t get the version 3 geolocation API to work, so I left this for another day (and maybe a reversion to the version 2 geolocation API, which requires a user key and which I think I’ve used successfully before… err, maybe?!;-).
So – the static maps route it is.. how does it work then? I tried a couple of routes: firstly, generating the page via a PHP script. Secondly, on the client side using a version of the JSON feed from Alex’s scraper code.
I’ll post the code at the end, but for now will concentrate on how the static image file is generated. As with the Google Charts API, it’s all in the URL.
For example, here’s a static map showing a marker on Walton Hall, Milton Keynes:
Here’s the URL:
http://maps.google.com/maps/api/staticmap?
center=Milton%20Keynes
&zoom=12&size=512×512&maptype=roadmap
&markers=color:blue|label:X|Walton%20Hall,%20Milton%20Keynes
&sensor=false
You’ll notice I haven’t had to provide latitude/longitude data – the static map API is handling the geocoding automatically from the address (though if you do have lat/long data, you can pass that instead). The URL can also carry more addresses/more markers – simply add another &markers= argument for each address. (I’m not sure what the limit is? It may be bound by the length of the URL?)
So -remember the original motivation for all this? Finding a way of getting recent crime incident data onto a map on the Dallas News website? Jennifer managed to get the original Google map onto the Dallas News page, so it seems that if she has the URL for a web page containing (just) the map, she can get it embedded in an iframe on the Dallas News website. But I think it’s unlikely that she’d be able to get Javascript embedded in the parent Dallas News page, and probably unlikely that she could get PHP scripts hosted on the site. The interactive map is obviously the preferred option, but a static map may be okay in the short term.
Looking at the crude map above, I think it could be nice to be able to use different markers (either different icons, or different colours – maybe both?) to identify the type of offence, its priority and its status. Using the static maps approach – with legend – it would be possible to colour code different incidents too, or colour or resize them if several units were in attendance? One thing I don;’t do is cluster duplicate entries (where maybe more than one unit is attending?)
It would be nice if the service was a live one, with the map refreshing every couple of minutes or so, for example by pulling a refreshed JSON feed into the page, and updating the map with new markers, and letting old markers fade over time. This would place a load on the original screenscraping script, so it’d be worth revisiting that and maybe implementing some sort of cache so that it plays nicely with the Dallas Police website (e.g. An Introduction to Compassionate Screen Scraping could well be my starter for 10). If the service was running as a production one, API rate limiting might be an issue too, particularly if the map was capable of being updated (I’m not sure what rate limiting applies to the static maps api, the Google maps API, or the Google geolocation API? In the short term (less coding) it might make sense to try to offload this to the client (i.e. let the browser call Google to geocode the markers), but a more efficient solution might be for a script on the server to geocode each location and then pass the lat/long data as part of the JSON feed.
Jennifer also mentioned getting a map together for live fire department data, which could also provide another overlay (and might be useful for identifiying major fire incidents?) In that case, it might be necessary to dither markers, so e.g. police and fire department markers didn’t sit on top of and mask each other. (Not sure how to do this in static maps, where we geocoding by address? Would maybe have to handle things logically, and use a different marker type for events attended by just police units, just fire units, or both types? If we’re going for real time, it might also be interesting to overlay recent geotagged tweets from twitter?
Anything else I’m missing? What would YOU do next?
PS if you want to see the code, here it is:
Firstly, the PHP solution [code]:
<html>
<head><title>Static Map Demo</title>
</head><body>
<?php
error_reporting(-1);
ini_set('display_errors', 'on');
include("json.php");
// Future-friendly json_encode
if( !function_exists('json_encode') ) {
function json_encode($data) {
$json = new Services_JSON();
return( $json->encode($data) );
}
}
// Future-friendly json_decode
if( !function_exists('json_decode') ) {
function json_decode($data) {
$json = new Services_JSON();
return( $json->decode($data) );
}
}
$response = file_get_contents("http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22http%3A%2F%2Fwww.dallaspolice.net%2Fmediaaccess%2FDefault.aspx%22%20and%0A%20%20%20%20%20%20xpath%3D'%2F%2F*%5B%40id%3D%22grdData_ctl01%22%5D%2Ftbody'&format=json");
$json = json_decode($response);
$reports = array();
if(isset($json->query->results))
{
$str= "<img src='http://maps.google.com/maps/api/staticmap?center=Dallas,Texas";
$str.="&zoom=10&size=512x512&maptype=roadmap";
$ul="<ul>";
$results = $json->query->results;
$i = 0;
foreach($results->tbody->tr as $tr)
{
$reports[$i]['incident_num'] = $tr->td[1]->p;
$reports[$i]['division'] = $tr->td[2]->p;
$reports[$i]['nature_of_call'] = $tr->td[3]->p;
$reports[$i]['priority'] = $tr->td[4]->p;
$reports[$i]['date_time'] = $tr->td[5]->p;
$reports[$i]['unit_num'] = $tr->td[6]->p;
$reports[$i]['block'] = $tr->td[7]->p;
$reports[$i]['location'] = $tr->td[8]->p;
$reports[$i]['beat'] = $tr->td[9]->p;
$reports[$i]['reporting_area'] = $tr->td[10]->p;
$reports[$i]['status'] = $tr->td[11]->p;
$addr=$reports[$i]['block']." ".$reports[$i]['location'];
$label=chr(65+$i);
$str.="&markers=color:blue|label:".$label."|".urlencode($addr);
$str.=urlencode(",Dallas,Texas");
$ul.="<li>".$label." - ";
$ul.=$reports[$i]['date_time'].": ".$reports[$i]['nature_of_call'];
$ul.", incident #".$reports[$i]['incident_num'];
$ul.=", unit ".$reports[$i]['unit_num']." ".$reports[$i]['status'];
$ul.=" (priority ".$reports[$i]['priority'].") - ".$reports[$i]['block']." ".$reports[$i]['location'];
$ul.="</li>";
$i++;
}
$str.="&sensor=false";
$str.="'/>";
echo $str;
$ul.="</ul>";
echo $ul;
}
?>
</body></html>
And here are a couple of JSON solutions. One that works using vanilla JSON [code], and as such needs to be respectful of browser security policies that say the JSON feed needs to be served from the same domain as the web page that’s consuming it:
<html>
<head><title>Static Map Demo - client side</title>
<script src="http://code.jquery.com/jquery-1.4.2.min.js"></script>
<script type="text/javascript">
function getData(){
var str; var msg;
str= "http://maps.google.com/maps/api/staticmap?center=Dallas,Texas";
str+="&zoom=10&size=512x512&maptype=roadmap";
$.getJSON('dallas2.php', function(data) {
$.each(data, function(i,item){
addr=item.block+" "+item.location;
label=String.fromCharCode(65+i);
str+="&markers=color:blue|label:"+label+"|"+encodeURIComponent(addr);
str+=encodeURIComponent(",Dallas,Texas");
msg=label+" - ";
msg+=item.date_time+": "+item.nature_of_call;
msg+=", incident #"+item.incident_num;
msg+=", unit "+item.unit_num+" "+item.status;
msg+=" (priority "+item.priority+") - "+item.block+" "+item.location;
$("<li>").html(msg).appendTo("#details");
})
str+="&sensor=false";
$("<img/>").attr("src", str).appendTo("#map");
});
}
</script>
</head><body onload="getData()">
<div id="map"></div>
<ul id="details"></ul>
</body></html>
And a second approach that uses JSONP [code], so the web page and the data feed can live on separate servers. What this really means is that you can grab the html page, put it on your own server (or desktop), hack around with the HTML/Javascript, and it should still work…
<html>
<head><title>Static Map Demo - client side</title>
<script src="http://code.jquery.com/jquery-1.4.2.min.js"></script>
<script type="text/javascript">
function dallasdata(json){
var str; var msg;
str= "http://maps.google.com/maps/api/staticmap?center=Dallas,Texas";
str+="&zoom=10&size=512x512&maptype=roadmap";
$.each(json, function(i,item){
addr=item.block+" "+item.location;
label=String.fromCharCode(65+i);
str+="&markers=color:blue|label:"+label+"|"+encodeURIComponent(addr);
str+=encodeURIComponent(",Dallas,Texas");
msg=label+" - ";
msg+=item.date_time+": "+item.nature_of_call;
msg+=", incident #"+item.incident_num;
msg+=", unit "+item.unit_num+" "+item.status;
msg+=" (priority "+item.priority+") - "+item.block+" "+item.location;
$("<li>").html(msg).appendTo("#details");
})
str+="&sensor=false";
$("<img/>").attr("src", str).appendTo("#map");
}
function cross_domain_JSON_call(url){
url="http://ouseful.open.ac.uk/maps/dallas2.php?c=true";
$.getJSON(
url,
function(data) { dallasdata(data); }
)
}
$(document).ready(cross_domain_JSON_call(url));
</script>
</head><body>
<div id="map"></div>
<ul id="details"></ul>
</body></html>
Online Journalism Blog 