wheresrhys

I can’t understand why people have a problem with the BBC. Well, I can, but I have no respect for the point of view. The general argument goes thus:

The BBC has a state-funded huge budget stifling the market, which is bad for competition, news, programming … (never mind that the BBC actually concentrates a lot on very high-quality, balanced programming, raising the bar for everyone else, and actually welcomes strong competition).

But James Murdoch, son of Voldemort Rupert, now head of the Murdoch media empire in UK, and probable heir to the global monolith that is NewsCorp… James Murdoch, one of the most influential media figures in the world, has just had a pop. To comment on a few choice pieces:

This year is the 150th anniversary of Darwin’s The Origin of Species. [...] It argued that the most dramatic evolutionary changes can occur through an entirely natural process. Darwin proved that evolution is unmanaged.[...]  The number who reject Darwin and cling to the concept of creationism is substantial.  And it crops up in some surprising places. For example, right here in the broadcasting sector in the UK.

I love/hate it when people misappropriate evolutionary theory. Darwin didn’t prove evolution is unmanaged, he merely examined evidence that evolution was happening and coined the theory of natural selection to explain how it happened. Below our Jimmy M displays his ignorance of what the theory of evolution says.

The consensus appears to be that creationism – the belief in a managed process with an omniscient authority -  is the only way to achieve successful outcomes.  There is general agreement that the natural operation of the market is inadequate, and that a better outcome can be achieved through the wisdom and activity of governments and regulators.

Now, Darwin said nothing about whether the results of natural selection are better than creationism (in fact, he’d maybe see it as a meaningless question), he only argued that natural selection is what happens. So whether creationism (what an absolutely absurd way for Jimmy to make his speech sound intelligent and relevant) or natural selection is the best way to manage the development of a system must be evaluated as a separate exercise, external to the system itself – defining what constitutes a good result and weighing up which method gets you closest to a good result. There is nothing in the theory of evolution that is anti-’creationism’ at all (other than that in the case of nature it dissolves the need for postulating a creator).

When I say this I feel like a crazy relative who everyone is a little embarrassed by and for sure is not to be taken too seriously.

No wonder.

Creationism penalises the poorest in our society with regressive taxes and policies – like the licence fee and digital switchover;

Still using the laughable creationism/evolution debate paradigm, J.Murd? OK – I’ll play along. There are plenty of examples (arguably most of the activity of government) where state control does the very opposite. Healthcare, education, rubbish collection – these are all examples where central control and collection of fees/taxes leads to all sorts of things being affordable by the poor. Without the licencing fee the money for the BBC’s high-quality output would need to come from somewhere, probably from subscription fees which surely would be at least about as high as the licence fee.

The study of evolution reminds us that  it is very difficult to predict the outcomes of events. [...]

Witness the international banana market.  In the 1950s the banana export industry faced a problem:  the then dominant Gros Michel – or ‘Big Mike’ – variety was being wiped out by a fungus called Panama Disease.  The industry took the decision to replace the entire world export crop with a supposedly disease-resistant variety called the Cavendish banana – the one we eat today.  Unfortunately it now appears that these bananas may themselves be vulnerable to a different kind of Panama Disease. Since Cavendish bananas are genetically identical sterile clones, they cannot build up any resistance.

There are important lessons here: attempts to manage natural diversity have unpredictable consequences and are more likely than not to fail over the long-term.

I do like the banana story, Jay-Mur. I find it whimsical. Though not entirely relevant. How’s about a better example: livestock have been bred over many, many generations to give better fleeces, milk and meat. Do you oppose that management of natural diversity? Of course not. Getting rid of all media channels in the world/UK except for the BBC I would of course oppose (though it’s no worse than having a Murdoch media monopoly), but that is not the situation being faced.

Fourth question.  Is this creationism good for investment?  No.  A heavily regulated environment with a  large public sector crowds out the opportunity for profit, hinders the creation of new jobs,  and dampens innovation in our sector.

This is far from true. The BBC contract out more and more work to independent companies, so there is still a lot of diversity. Because the investment comes from the BBC and not other largemedia conglomerates (let’s face it, the BBC isn’t the only bull in the TV shop) you could argue it protects diversity as the BBC has to spend money on innovation, minority interests, regional programming … diversity, in short.

We don’t even have the basics in place to protect creative work. Whether it’s shoplifting at HMV or pirating the same movie online, theft is theft. They are both crimes and should be treated accordingly.  The government dithers – dimly aware of what it has to do but afraid to do it.

Do you see other countries tackling it better, J-Mu?

As originally with news and sport, so now with the arts and drama.  Sky now offers four dedicated arts channels.  Original commissioning by channels that customers choose to pay for is expanding and will continue to do so, not just from Sky but from the likes of National Geographic, History, MTV and the Disney Channel, to name a few.  Sky alone now invests over £1 billion a year
in UK content.

Remind me – was your speech about how stifled the rest of the industry is by the BBC, or about how the rest of the industry is doing pretty well too? It’s begining to sound like you’re making a mountain out of an imaginary molehill. I would add to the above though that I’ve watched a fair bit of cable/satellite TV over the years and, while it’s difficult to define exactly what, they definitely don’t do some things like the BBC do them. I think subtlety is the lacking ingredient. Having a TV channel that doesn’t have to attract advertising for each programme definitely enables the production of programmes that aren’t so brazen about garbbing your attention. David Attenborough, Adam Curtis, The Royle Family, The Office – these might all have never come to light without an organisation that isn’t always searching for more bang in order to get more buck.

And now I’ll stop quoting as this post is getting ridiculously long, but you get the jist: Jammon has quite an axe to grind against a country whose citizen’s don’t mind paying a public body to produce a lot of good quality TV and other media because it’s unbalanced, stifling, power grabbing… sinister, even… nothing like his dad.

In previous posts I’ve covered the inherent difficulty of geocoding addresses and postcodes in the UK, specifically in order use the geocodes with Google maps. I learned a lot about the limitations of Google’s various geocoding services. To sum up the situation:

• The Google maps API’s geocoding webservice is buggy: giving more address details can give a less accurate, and in many cases plain wrong, geocode. However, the GClientGeocoder class built into Google maps fares a lot better with addresses…
• … apart from when the address entered is a postcode. Then it geocodes to the central point of the postcode sector, rather than the precise postcode location, i.e if you search for E3 4GY or E3 4NT, it will return the geocode for the centre of E3 4 in both cases.
• But, as I discovered when trying an alternative route to postcode geocoding, the Google Ajax search API, the localSearch class in particular, gives very accurate geocodes for full postcodes.

Previously I’d been geocoding large batches of addresses/postcodes, but having got the geocodes in order to plot points on the map, I wanted to add search functionality to the map, so that it would zoom in on a given address/postcode. I wanted it to be accurate for both addresses and postcodes, so using what I’d learned, I wrote a javascript function, geocodeUKAddress, which always returns the best geocode that Google can offer, so your website can be as reliable as a London cabbie (again, I can’t embed it here as javascript has a habit of breaking wordpress, though there must be some way to make it safe – I will research).

You will need to include the following in ther head of your webpage too

<script src="http://maps.google.com/maps?file=api&amp;v=2&amp;key=your_api_key_here" type="text/javascript"></script>
<script src="http://www.google.com/jsapi" type="text/javascript"></script>

I can’t take credit for the regular expression that recognises UK postcodes, but apart from that, it’s all my own work, which anyone can feel free to use. Unless you maybe work for Rupert Murdoch.

The art of geocoding addresses in the UK, as I previously explained, is a soul-destroying process, frought with inaccuracy, bugs and convoluted workarounds. And for all that work you end up with a set of points of which a great deal are probably somewhat inaccurate and at least some of which are completely wrong. UK addresses (and probably those elsewhere in the world) are complicated creatures, which Google’s geocoding engine often interprets wrongly.

Postcodes, on the other hand, are rather easier; there is a well-defined relationship between a UK postcode and its corresponding (usually pretty small) piece of the British countryside. But google’s geocoding api will only return a geocode for the postcode sector (ie will give a geocode for LL12 5 when you searched for LL12 5TH). However, someone did figure out a way of using Google’s local search API combined with google maps to geocode UK postcodes. Since he blogged about it the API has changed, so below is an outline of how to geocode a batch of postcodes in the UK using just some simple php, the current google ajax search API and a little javascript (jQuery isn’t essential, but cuts down on coding a bit). The javascript is the crucial step.

Assuming you have a database full of postcodes and id numbers, and 2 empty columns to store latitude and longitude values, this is how it’s done. (Download source geocode.zip).

1. Create a html page geocode.html with the following content:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" >

<head>
<title></title>
<meta name="description" content="" />
<meta http-equiv="content-type" content="text/html; charset=iso-8859-1" />
<link rel="stylesheet" href="" type="text/css" media="screen" />
<script type="text/javascript" src="jquery-1.3.2.js"></script>
<script src="http://www.google.com/jsapi" type="text/javascript"></script>
<script type="text/javascript" src="geocode.js"></script>
</head>
<body>
<div id="counter"></div>
</body>
</html>

(Make sure you specify the correct location for your local javascript files)

2. Create a php file (in the same directory), geocode.php, with the following rough structure (it will only be accessed via ajax, so is very stripped down):

<?php
require_once ('mysqlConnect.php'); //or other database connection details
if($_GET)
{
 //var_dump($_GET);
 update_record();
 send_new_data();
}

//gets the next record without a geocode and sends the id and postcode to the browser
function send_new_data() {
 $query = @mysql_query("SELECT id, postcode FROM geocode_table WHERE lat = '' AND postcode != '' ORDER BY id LIMIT 1");
 if(($query) &&mysql_num_rows($query)) {
  $row = mysql_fetch_array($query, MYSQL_ASSOC);
  echo $row['id'].','.$row['postcode'];
 } else {
  echo 'stop';
 }
}

//updates the last record with data sent from browser
function update_record() {
 $id = $_GET['id'];
 $lat = $_GET['lat'];
 $lng = $_GET['lng'];
 if($id > 0)
 {
  $update = "UPDATE geocode_table SET lat = '".$lat."', lng = '".$lng."' WHERE id = ".$id;
  $result = @mysql_query($update);
  if (!$result) {
   die('Invalid query: ' . mysql_error());
  }
 }
}
?>

3. Create a javascript file geocode.js, saved in the same directory again (I would paste it here but it keeps breaking wordpress)

4. Running the code

Once you’ve altered the database connection details, and SQL query to suit your setup, simply open geocode.html in your browser. A counter will tell you which record you’re on. To stop the code simply close your browser/browser tab.

How it all works

In a nutshell (ignoring the special case of starting off the loop) the code repeatedly performs the following process:

….in geocode.php, send_new_data() finds a record which has no latitude value and sends it’s id number and postcode as an ajax response to set_and_get_next(). This keeps track of the id in a global variable and sends the postcode to getPointFromPostcode(), which uses google’s local search to get a geocode. Once it’s found a geocode it passes it to set_and_get_next(), which sends it to geocode.php in an ajax request. There update_record()… well… updates the record, and send_new_data() finds a record which has no la….

Compared to my previous approach iterating a script over large sets of data, using ajax is very sleek. Similarly to a pure php script I can load from a browser, though with much of the resource intensive scripting taking place on my or google’s server. But with ajax there’s no problem with the browser timing out from time to time, or baulking at the number of times a page is requested. It’s a little harder to code, and probably less efficient… but I like it. And I’ll definitely be using my shiny new geocoded postcode data.

This damn economic crisis/swine flu outbreak isn’t quite that bad yet, but nevertheless there is a very limited sense where the UK is quite anarchic: geocoding addresses using Google Maps.

Having completed my download of addresses for my new Google Maps website the next stage was to geocode them so that I can plot them on the map. I had no idea how tricky it would be when I started out.

The most irritating and fundamental difficulty is that geocodes for UK postcodes are not available for free. The data is owned by the Royal Mail, and there is at least one website where you can buy access to this information (it has a free trial, but I discovered that this is just for about 10 or so geocodes). You can search by postcode on google maps, but if you put a postcode e.g. LL13 7YH into the geocoder API you’re given the geocode for LL13 7 – not accurate enough to be of any real use.

So you have to go for geocoding full addresses instead. The geocodes for these data isn’t owned by Royal Mail, but by the Ordnance Survey, and for some reason they are less restrictive about sharing the information. But there’s still a long hard slog before you can get the geocodes out of this.

Google offer a really useful turorial on geocoding addresses, and this, combined with my approach to iterating over a large number of records meant I was collecting the geocodes in no time. However, it wasn’t as peachy as it seemed.

For example, the address Llantysilio, Denbighshire, UK brings up a pretty accurate geocode for the village of Llantysilio in North Wales. However, the full address, including the postal town is Llantysilio, Llangollen, Denbighshire, UK, and this unexpectedly brings up the geocode for an address on Castle Street, right in the middle of Llangollen. So a more complete address leads to a far less accurate geocode. This is immensely problematic.

In general I was feeding in the longest possible address made up out of the data I had, so in my php script I had something like the following:

while(count($arr_address > 1) && !$str_lat)
 {
$str_address = implode(', ', $arr_address);
 attempt_geocode($url.$str_name.', '.$str_address.', '.$str_county.', UK');
 attempt_geocode($url.$str_address.', '.$str_county.', UK');
 attempt_geocode($url.$str_address.', UK');
 array_pop($arr_address);
 }

This starts with the longest, most detailed address string, and then gradually cuts the string down (possibly sacrificing accuracy in order to get a passable geocode), with attempt_geocode() exiting the loop on success.

But the fact that longer addresses can lead to incorrect geocodes meant I had to work in a way to start off with shorter addresses, and if that doesn’t get a geocode then gradually shorten them and keep trying to geocode. So I’ now have:

 attempt_geocode($url.$arr_address[0].', '.$str_county.', UK');
 attempt_geocode($url.$arr_address[0].', '.end($arr_address).', '.$str_county.', UK');
while(count($arr_address > 1) && !$str_lat)
{
 $str_address = implode(', ', $arr_address);
 attempt_geocode($url.$str_address.', '.$str_county.', UK');
 attempt_geocode($url.$str_address.', UK');
 array_pop($arr_address);
}

A long process in order to get a geocode that could still quite likely be wrong, and even if it’s basically correct might not be as accurate as a postcode; but nevertheless an improvement on what I had before.

A glimmer of hope though is that google maps itself doesn’t suffer from this issue – both address versions return the same accurate point on the map, and as someone pointed out to me on stackoverflow, google maps is in beta, so maybe teh geocoder API just hasn’t been updated to the newer, better address parser, and maybe one day reliable geocoding for free in the UK will be a reality. Also, somebody has found a way to geocode in the UK using postcodes, by hacking together the google maps and search APIs, and I may well try it, as this address geocoding malarchy leaves a lot to be desired. (*edit – turns out it’s heavily reliant on javascript so can’t be used for geocoding masses of pages without slowing down your browser.)

Finally, if this article wasn’t any help, there’s loads of geocoding links here.