wheresrhys

For a long time I’ve wanted to work with the Delicious API. Initially it was because the Delicious website not only had the difficult to remember del.icio.us url, but was also very badly designed. If you compared its progress – addition of new features, cleaning up of design, making use of new techniques suchas AJAX – with its web2.0 compatriots (Flickr, Digg, boris-johnson.com) it lagged way behind.

So I initially planned to build a new front-end for it, making it easier to work with your bookmarks, but before I could progress far enough in my coding abilities they completely redesigned the site; a vast improvement.

Though still not perfect. For a while I’ve found it frustrating that there is no easy way to simultaneously see the content of a bookmarked page and delete the bookmark if you deem it no longer useful, so my delicious account gradually got more and more cluttered. Well, this afternoon I decided to do something about it (and not just because I’m avoiding doing more important stuff).

But I was foiled for a long time by the laziness of the Delicious developers. My initial plan was to use javascript to get a JSON of all my bookmarks (or alternatively request one at a time) and go through them one by one, displaying the webpage in an iframe, and offering the option to discard or keep the bookmark. However, delicious only publish this data as XML which means, due to cross-domain restrictions on AJAX, you can’t just use javascript. I may be a bit hasty in pinning this on developer laziness, but I imagine creating alternate templates (because that’s all the difference between JSON and XML really) wouldn’t be too time consuming, and would greatly enhance the versatility of the API.

Anyway, I realised I would have to use a bit of PHP to get the XML and create pages from which my javascript would be able to access the data. Luckily, before I dived straight in I came across phpdelicious (which, appropriately, I have now bookmarked in Delicious) , a very easy to use php class for wrapping the Delicious API, which is very handy indeed. Less than an hour later I had built exactly what I wanted.

I reckon a few more hours development and I can make it a publicly available service.  All I need to do is include a form for other users to be able to login, and (ideally) preload websites in the iframe to speed things up (though this is problematic as some sites force the whole web page to be redirect if you try and put them in an iframe).

In previous posts I’ve covered the inherent difficulty of geocoding addresses and postcodes in the UK, specifically in order use the geocodes with Google maps. I learned a lot about the limitations of Google’s various geocoding services. To sum up the situation:

• The Google maps API’s geocoding webservice is buggy: giving more address details can give a less accurate, and in many cases plain wrong, geocode. However, the GClientGeocoder class built into Google maps fares a lot better with addresses…
• … apart from when the address entered is a postcode. Then it geocodes to the central point of the postcode sector, rather than the precise postcode location, i.e if you search for E3 4GY or E3 4NT, it will return the geocode for the centre of E3 4 in both cases.
• But, as I discovered when trying an alternative route to postcode geocoding, the Google Ajax search API, the localSearch class in particular, gives very accurate geocodes for full postcodes.

Previously I’d been geocoding large batches of addresses/postcodes, but having got the geocodes in order to plot points on the map, I wanted to add search functionality to the map, so that it would zoom in on a given address/postcode. I wanted it to be accurate for both addresses and postcodes, so using what I’d learned, I wrote a javascript function, geocodeUKAddress, which always returns the best geocode that Google can offer, so your website can be as reliable as a London cabbie (again, I can’t embed it here as javascript has a habit of breaking wordpress, though there must be some way to make it safe – I will research).

You will need to include the following in ther head of your webpage too

<script src="http://maps.google.com/maps?file=api&amp;v=2&amp;key=your_api_key_here" type="text/javascript"></script>
<script src="http://www.google.com/jsapi" type="text/javascript"></script>

I can’t take credit for the regular expression that recognises UK postcodes, but apart from that, it’s all my own work, which anyone can feel free to use. Unless you maybe work for Rupert Murdoch.

This damn economic crisis/swine flu outbreak isn’t quite that bad yet, but nevertheless there is a very limited sense where the UK is quite anarchic: geocoding addresses using Google Maps.

Having completed my download of addresses for my new Google Maps website the next stage was to geocode them so that I can plot them on the map. I had no idea how tricky it would be when I started out.

The most irritating and fundamental difficulty is that geocodes for UK postcodes are not available for free. The data is owned by the Royal Mail, and there is at least one website where you can buy access to this information (it has a free trial, but I discovered that this is just for about 10 or so geocodes). You can search by postcode on google maps, but if you put a postcode e.g. LL13 7YH into the geocoder API you’re given the geocode for LL13 7 – not accurate enough to be of any real use.

So you have to go for geocoding full addresses instead. The geocodes for these data isn’t owned by Royal Mail, but by the Ordnance Survey, and for some reason they are less restrictive about sharing the information. But there’s still a long hard slog before you can get the geocodes out of this.

Google offer a really useful turorial on geocoding addresses, and this, combined with my approach to iterating over a large number of records meant I was collecting the geocodes in no time. However, it wasn’t as peachy as it seemed.

For example, the address Llantysilio, Denbighshire, UK brings up a pretty accurate geocode for the village of Llantysilio in North Wales. However, the full address, including the postal town is Llantysilio, Llangollen, Denbighshire, UK, and this unexpectedly brings up the geocode for an address on Castle Street, right in the middle of Llangollen. So a more complete address leads to a far less accurate geocode. This is immensely problematic.

In general I was feeding in the longest possible address made up out of the data I had, so in my php script I had something like the following:

while(count($arr_address > 1) && !$str_lat)
 {
$str_address = implode(', ', $arr_address);
 attempt_geocode($url.$str_name.', '.$str_address.', '.$str_county.', UK');
 attempt_geocode($url.$str_address.', '.$str_county.', UK');
 attempt_geocode($url.$str_address.', UK');
 array_pop($arr_address);
 }

This starts with the longest, most detailed address string, and then gradually cuts the string down (possibly sacrificing accuracy in order to get a passable geocode), with attempt_geocode() exiting the loop on success.

But the fact that longer addresses can lead to incorrect geocodes meant I had to work in a way to start off with shorter addresses, and if that doesn’t get a geocode then gradually shorten them and keep trying to geocode. So I’ now have:

 attempt_geocode($url.$arr_address[0].', '.$str_county.', UK');
 attempt_geocode($url.$arr_address[0].', '.end($arr_address).', '.$str_county.', UK');
while(count($arr_address > 1) && !$str_lat)
{
 $str_address = implode(', ', $arr_address);
 attempt_geocode($url.$str_address.', '.$str_county.', UK');
 attempt_geocode($url.$str_address.', UK');
 array_pop($arr_address);
}

A long process in order to get a geocode that could still quite likely be wrong, and even if it’s basically correct might not be as accurate as a postcode; but nevertheless an improvement on what I had before.

A glimmer of hope though is that google maps itself doesn’t suffer from this issue – both address versions return the same accurate point on the map, and as someone pointed out to me on stackoverflow, google maps is in beta, so maybe teh geocoder API just hasn’t been updated to the newer, better address parser, and maybe one day reliable geocoding for free in the UK will be a reality. Also, somebody has found a way to geocode in the UK using postcodes, by hacking together the google maps and search APIs, and I may well try it, as this address geocoding malarchy leaves a lot to be desired. (*edit – turns out it’s heavily reliant on javascript so can’t be used for geocoding masses of pages without slowing down your browser.)

Finally, if this article wasn’t any help, there’s loads of geocoding links here.