wheresrhys

I’m redesigning this site at the moment and one thing I wanted to pay closer attention to this time around was choice of fonts. Notwithstanding the fact that web browsers were, until recently, pretty shoddy at allowing you to use the fonts you wanted to, the main reason I hadn’t paid much attention to fonts in the past was that it’s so darn hard picking one, for a number of reasons:

1. There’s so damn many of them
2. There’s no relation between names and appearance (as opposed to, say, #f60283 being guessable as a garish pink, closer to red than to mauve)
3. They’re difficult to preview and compare

So in the interests of productivity, quality and, dare I say it, working smarter, not harder, I’ve devoted some time to sorting out how I choose fonts, reduced to the following four steps:

1. Organise your fonts in themed folders
2. Find a good stand alone font previewer i.e. don’t rely on photoshop and windows’ poor tools
3. Narrow down to a short list
4. Use javascript to make quick comparisons easy

Read on if your curiosity is piqued.

1. Organising the fonts

Windows’s default font collection (and, I imagine, Apple’s too) and the collection of 2000 fonts I also use are organised alphabetically which is rubbish. Unless you have web design OCD you’re unlikely to want to narrow down your choice of font by it’s first-letter. But, on the other hand, it is difficult to categorise fonts; I’m yet to see a website do it effectively, and I’m not entirely happy with my choice. But in the end I plumped for the following: serif, sans-serif, script, stylised, symbol, stencil.

Stencil could easily fall within stylised, and stylised could easily be split into subfolders (Art Deco, Gothic, Celtic …), and I realised part way through that I should have a subfolder within symbol for fonts depicting alphabets other than Roman, but I think a minimal folder system would be serif, sans-serif and other. In the serif and sans-serif folders I included only sensible fonts you could print a book not aimed at children in, which I think is a useful distinction to make; often the hardest task when picking fonts is to find the one which is subtly different to other ordinary fonts, but somehow suits the design better than almost indistinguishable alternatives. Cutting out all the fancy fonts makes this task a lot easier.

So once you have all these folders put a copy of each font you have in one of them, and you’re ready for the next step. You don’t need to bother with installing them yet.

2. Previewing your fonts

My criteria for a good font-previewer are:

1. Choice of text to use for the preview
2. Choice of text-size and colour
3. Ability to organise fonts how you want them organised
4. Speed and ease of switching between one font and another
5. Ease of installing a font

Windows’ control panel fails to meet conditions 1 and 2, and Photoshop’s built-in font selector fails to meet 3 and 4, and they both force you to install a font before using it. The best free solution I’ve come across for Windows which meets all 5 conditions is Fast Font Preview.

3. Short-listing

Once you’ve got Fast Font Preview installed you can start picking your shortlist. Simply browse to your organised font folders, click on settings to type in your sample text, adjust the font-size to what you want, and double click on any contenders. Once the font is opened you can install with one click. While the font is open you’ll also want to jot down its official font name.

4. Use javascript to make comparisons

This is where I stop being pedantic and hopefully can be of some use. Insert the following script into the head of your web-page-in-need-of-a-font, substitute your list of font names and the id/tag name of the elements to be affected and, hey presto, a handy font-switcher should appear, making choosing exactly the right font much easier. If your web page already uses javascript then add the body of the function to your existing onload event. (NOTE: It tends to work better in Google Chrome than other browsers as Google Chrome seems to be a bit more forgiving when it comes to font names).

window.onload = function() {
 var body = document.getElementsByTagName('body')[0];
 var id = '';
 var tagName = '';
 var fonts = 'Comma,separated,list,of font,names';
 fonts = fonts.split(',');
 var elements = (id)      ? [document.getElementById(id)] :
                (tagName) ? document.getElementsByTagName(tagName):
                            [body];
 var switcher = document.createElement('DIV');
 switcher.setAttribute('style','position:absolute;top:0;right:0;background:white;z-index:200;padding:10px;');
 for(var i=0, il = fonts.length;i<il;i++){
   createButton(i);
 }
 body.appendChild(switcher);
 var buttons = switcher.childNodes
 loadFont(0);

 function createButton(pos) {
   var button = document.createElement('a');
   button.innerHTML = fonts[pos];
   button.setAttribute('style','display:block;padding:2px;');
   button.setAttribute('href','#');
   button.onclick = function() {loadFont(pos);};
   switcher.appendChild(button);
 }

 function loadFont(pos) {
   for (var i=0,il=elements.length;i<il;i++)
   {
     elements[i].style.fontFamily = fonts[pos];
   }
   for (var j=0,jl=buttons.length;j<jl;j++) {
     buttons[j].style.backgroundColor='';
   }
   buttons[pos].style.backgroundColor='yellow';
   return false;
 };
};

Well, this has to be the quickest I’ve ever gone from idea to publishable (albeit limited functionality) website.

Deletious is my new site for simultaneously viewing a page bookmarked in Delicious and deciding whether to keep or delete the bookmark. I’ve had quite a lot of fun using it the last few days, rediscovering all sorts of articles, games, tools and other long forgotten sites. As well as wasting a lot of time reacquainting myself with all these I’ve also managed to de-clutter my Delicious account; all the CSS articles from 2-3 years ago giving an introduction to topics I now know inside out are gone from my bookmarks, as are all those gimmicky websites I can’t believe i found funny at one time.

Disappointingly, I’m having problems uploading the logo to the website’s folder, but it’ll be sorted sometime soon I hope.

So please do give it a go and let me know what you think.

EDIT – There’s a bug that pops up every now and then (something to do with caching) which leads Deletious to show zero bookmarks for your account. I’ll fix the bug when I get time, but waiting a few hours seems to clear the cache (at least, it works for my account) and then you can access your bookmarks again.

A very short post to announce my third  jQuery plugin: listSplitter, which takes a long list of categorised items (where the categories can overlap) and creates a tabbed interface to show only one category at once. I haven’t done a demo yet (well, I have, buthaving teh same old problem transferring to the server as the server runs on UNIX while my laptop is Windows. I recently found out why this causes a problem, but no easy fix has presented itself) so you’ll just half to take my word for it, though before too many months have gone by I will use it as the basis for a new portfolio.

*edit: Here is a demo

And it can be used with jQuery themeroller, i.e. use the tool at jqueryui.com/themeroller, to design how it shoudl look, then after clicking “download theme” make sure you have ‘tabs’ ticked underneath widgets.

As I think I may have mentioned, the latest project I worked on I’ve been using the Zend Framework for all the server-side development. Over the last few months I’ve developed a love hate relationship with it. On the plus side it does pretty much everything I need without needing too much customisation, a few of the negatives though are:

• The quickstart in the documentation assumes way too much background information about configuration of a php app and data sources etc. They don’t seem to have considered that a reason many programmers use zend is because they don’t know too much about back-end development and want something to take care of the tricky bits for them. It took ages to get past this stumbling block, with the help of this tutorial (WARNING: The bit on connecting to the database either uses quotes when it shouldn’t or vice versa) and this website with tutorials on various zend components.
• Having said it does everything, there are lots of gaps. I’m sure a lot of thought goes into deciding what gets included and they must get all sorts of requests, but some simple standard things are missing, eg a validator to check a confirm password field. Nevertheless it is fairly easy to write extensions, but the zend documentation site should have a lot more and clearer information on this. Linked to this, there seems to have been very little thought put into building some sort of community to share plugins, unlike jQuery, for example.
• You may have sensed that I don’t have a great deal of respect for the documentation. It’s very sloppy to be frank. There are so few cross-links to different sections, and a lot of the classes contain examples which I found fairly irrelevant, covering a way of doing things unlikely to be used in a real website (eg the examples for querying your database don’t really use the Model-View-Controller structure you’re supposed to use). And, in my view, it’s way too wordy, and much of the text is just waffley clutter; far more so than better examples of documentation, such as php and google maps. The website is also very difficult to navigate.

To fix the final gripe I’ve written a greasemonkey script (my first ever one) to replace the existing documentation layout with one a little easier to negotiate.

Before greasemonkey

After greasemonkey

I’ve now written two jQuery plugins, and have a pretty good idea for a third. As you might expect, I’ve learned quite a few things along the way. When I started there was a lot I didn’t know about javascript and jQuery. There still is, but one thing that struck me is that there are a lot of great tutorials on writing jQuery plugins out there but none of them individually captures all the things that took my plugins from being nice ideas with mediocre implementations to being something I’m proud of.

So here is my definitive combination of them all, together with one or two of my own contributions (a bit scant on detail, but follow the links for more info).

1. Build a good test page

My test page for any given plugin has 3 instances of a type of element the plugin is suppose to effect. eg a list. They are marked up as follows:

<ul class="single" >...</ul>
<ul class="multiple" >...</ul>
<ul class="multiple" >...</ul>

And I run the following script in the document head

$(document).ready(function() {
  $('.single').myPlugin();
  $('.multiple').myPlugin();
});

This will make sure your plugin works when called more than once on a page, and when one instance affects more than one element. For testing your code for speed the same page with various bits of html copied lots of times does the trick.

2. Follow jQuery’s recommended tutorial

It really is very good, and covers most important points. The other items on the list are either little tweaks that I think make the code neater or, in just one or two cases (highlighted in red), things its missed out. Read the tutorial

3. Make sure your plugin won’t conflict with other javascript libraries

The most effective way to do this is to wrap your whole plugin in a function where $ is its parameter, and then call it on jQuery, like so:

(function($) {
  $.fn.pluginName = function(options) { ... };
})(jQuery);

More info

4. Make your plugin’s settings changeable, both when the plugin is called, and by tweaking the default settings

The way to do this is to set up your plugin’s defaults as a JSON which is a public property of the plugin object. So within your plugin add these lines (where options is the name of the JSON passed to the plugin as a parameter).

this.defaults = {parameter:'value', ...};
options = $.extend(defaults,options);

More info

5. Make sure your plugin keeps track of DOM elements effectively

Structure your code as follows in order to make sure your variables are accessible to the right instances of the plugin, but aren’t stored in memory longer than they need to be.

$.fn.pluginName = function(options) {
  var constructor1, constructor2, ...;//elements that are used only when the plugIn is constructing instances of itself at the start and play no role in handling later events;
  return this.each(function() {
     var tracker1, tracker2, ...;//elements that are important for events and other later changes. Also, if your plugin's constructor involves some AJAX it may be best to put all variables in here
     subFunction() {
	//this should always be able to call the variables it needs, in particular access the right DOM elements
     }
  });
};

6. Optimise, opimise, optimise

You never know how demanding the website using your plugin is going to be: how many times is it going to be called on one page; will the dom objects (eg sorting a list of items) it has to handle be a lot bigger than the ones you’ve tested it on. My crossSelect plugin was, I discovered, before version 4/5, really sluggish on lists larger than about 20 items.

So I found a great list of techniques to optimise your code, the most pertinent to writing plugins I have put below:

• Make sure your optimisations are optimisations

  To keep track of whether your optimisations are having the desired effect, run your plugin on a page full of huge elements for it to process and then use firebug to measure the time it takes.

• Don’t over-query the DOM

  jQuery is principallya tool for accessing and manipulating the DOM, but it doesn’t pay to do this too often as your code will be a lot quicker if you keep it to a minimum. Techniques for avoiding this include:

  • store DOM elements you know you will have to access a few times in variables (but be careful – see point 5)
  • Add new DOM elements as html, . So instead of

    $('selector').append('<ul>');
    for(i=0;i<limit;i++)
    {
      $('selector > ul').append('<li>'+i-th text+'<li>');
    }

    use

    string = '<ul>';
    for(i=0;i<limit;i++)
    {
      string+='<li>'+i-th text+'<li>';
    }
    $('selector').append(string);

    The improvement in speed of code is astounding.

  • Use contexts. A lot. It will both reduce code size (as shown below) and speed up the code (because jQuery only has to search a subset of the DOM). So instead of

    var theContainer = $('selector');
    $(theContainer).children('ul').children('li').aMethod();

    use

    var theContainer = $('selector');
    $('ul>li', theContainer).aMethod();

And that’s yer lot. Please let me know if you have any comments or suggested additions.

I won’t go into details, but I’ve had quite a tough month or so in the web programming side of my life. A lot of hard work and dedication leading to a very unsatisfactory ending – the first time in my adult life I’ve left a job with a bitter taste in my mouth.

So I thought I’d write this to say thank you to everyone who has, in the same time period, contributed to the small flood of comments relating to my two jQuery plugins – fullTextArea and crossSelect. Up until recently I had no idea anyone was even using them, so it really brightened the last 4 weeks to know that the free time I put into programming is appreciated (even if the same can’t be said for paid work all the time).

So thank you again for restoring my faith in the programming community. I’ll get on to incorporating the best suggestions as soon as I can.

The art of geocoding addresses in the UK, as I previously explained, is a soul-destroying process, frought with inaccuracy, bugs and convoluted workarounds. And for all that work you end up with a set of points of which a great deal are probably somewhat inaccurate and at least some of which are completely wrong. UK addresses (and probably those elsewhere in the world) are complicated creatures, which Google’s geocoding engine often interprets wrongly.

Postcodes, on the other hand, are rather easier; there is a well-defined relationship between a UK postcode and its corresponding (usually pretty small) piece of the British countryside. But google’s geocoding api will only return a geocode for the postcode sector (ie will give a geocode for LL12 5 when you searched for LL12 5TH). However, someone did figure out a way of using Google’s local search API combined with google maps to geocode UK postcodes. Since he blogged about it the API has changed, so below is an outline of how to geocode a batch of postcodes in the UK using just some simple php, the current google ajax search API and a little javascript (jQuery isn’t essential, but cuts down on coding a bit). The javascript is the crucial step.

Assuming you have a database full of postcodes and id numbers, and 2 empty columns to store latitude and longitude values, this is how it’s done. (Download source geocode.zip).

1. Create a html page geocode.html with the following content:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" >

<head>
<title></title>
<meta name="description" content="" />
<meta http-equiv="content-type" content="text/html; charset=iso-8859-1" />
<link rel="stylesheet" href="" type="text/css" media="screen" />
<script type="text/javascript" src="jquery-1.3.2.js"></script>
<script src="http://www.google.com/jsapi" type="text/javascript"></script>
<script type="text/javascript" src="geocode.js"></script>
</head>
<body>
<div id="counter"></div>
</body>
</html>

(Make sure you specify the correct location for your local javascript files)

2. Create a php file (in the same directory), geocode.php, with the following rough structure (it will only be accessed via ajax, so is very stripped down):

<?php
require_once ('mysqlConnect.php'); //or other database connection details
if($_GET)
{
 //var_dump($_GET);
 update_record();
 send_new_data();
}

//gets the next record without a geocode and sends the id and postcode to the browser
function send_new_data() {
 $query = @mysql_query("SELECT id, postcode FROM geocode_table WHERE lat = '' AND postcode != '' ORDER BY id LIMIT 1");
 if(($query) &&mysql_num_rows($query)) {
  $row = mysql_fetch_array($query, MYSQL_ASSOC);
  echo $row['id'].','.$row['postcode'];
 } else {
  echo 'stop';
 }
}

//updates the last record with data sent from browser
function update_record() {
 $id = $_GET['id'];
 $lat = $_GET['lat'];
 $lng = $_GET['lng'];
 if($id > 0)
 {
  $update = "UPDATE geocode_table SET lat = '".$lat."', lng = '".$lng."' WHERE id = ".$id;
  $result = @mysql_query($update);
  if (!$result) {
   die('Invalid query: ' . mysql_error());
  }
 }
}
?>

3. Create a javascript file geocode.js, saved in the same directory again (I would paste it here but it keeps breaking wordpress)

4. Running the code

Once you’ve altered the database connection details, and SQL query to suit your setup, simply open geocode.html in your browser. A counter will tell you which record you’re on. To stop the code simply close your browser/browser tab.

How it all works

In a nutshell (ignoring the special case of starting off the loop) the code repeatedly performs the following process:

….in geocode.php, send_new_data() finds a record which has no latitude value and sends it’s id number and postcode as an ajax response to set_and_get_next(). This keeps track of the id in a global variable and sends the postcode to getPointFromPostcode(), which uses google’s local search to get a geocode. Once it’s found a geocode it passes it to set_and_get_next(), which sends it to geocode.php in an ajax request. There update_record()… well… updates the record, and send_new_data() finds a record which has no la….

Compared to my previous approach iterating a script over large sets of data, using ajax is very sleek. Similarly to a pure php script I can load from a browser, though with much of the resource intensive scripting taking place on my or google’s server. But with ajax there’s no problem with the browser timing out from time to time, or baulking at the number of times a page is requested. It’s a little harder to code, and probably less efficient… but I like it. And I’ll definitely be using my shiny new geocoded postcode data.

No, this isn’t an artist’s impression of the UK trapped in the midst of a swine flu epidemic, but is in fact what all my geocoded points look like plotted on a map of the UK*. All 4,528 of them. Now I just have to figure out how to make this look presentable, i.e. feed the points to the map gradualy as the user zooms in. I bet this is where things will really start to get difficult.

*with apologies to Cornwall and northern Scotland

This damn economic crisis/swine flu outbreak isn’t quite that bad yet, but nevertheless there is a very limited sense where the UK is quite anarchic: geocoding addresses using Google Maps.

Having completed my download of addresses for my new Google Maps website the next stage was to geocode them so that I can plot them on the map. I had no idea how tricky it would be when I started out.

The most irritating and fundamental difficulty is that geocodes for UK postcodes are not available for free. The data is owned by the Royal Mail, and there is at least one website where you can buy access to this information (it has a free trial, but I discovered that this is just for about 10 or so geocodes). You can search by postcode on google maps, but if you put a postcode e.g. LL13 7YH into the geocoder API you’re given the geocode for LL13 7 – not accurate enough to be of any real use.

So you have to go for geocoding full addresses instead. The geocodes for these data isn’t owned by Royal Mail, but by the Ordnance Survey, and for some reason they are less restrictive about sharing the information. But there’s still a long hard slog before you can get the geocodes out of this.

Google offer a really useful turorial on geocoding addresses, and this, combined with my approach to iterating over a large number of records meant I was collecting the geocodes in no time. However, it wasn’t as peachy as it seemed.

For example, the address Llantysilio, Denbighshire, UK brings up a pretty accurate geocode for the village of Llantysilio in North Wales. However, the full address, including the postal town is Llantysilio, Llangollen, Denbighshire, UK, and this unexpectedly brings up the geocode for an address on Castle Street, right in the middle of Llangollen. So a more complete address leads to a far less accurate geocode. This is immensely problematic.

In general I was feeding in the longest possible address made up out of the data I had, so in my php script I had something like the following:

while(count($arr_address > 1) && !$str_lat)
 {
$str_address = implode(', ', $arr_address);
 attempt_geocode($url.$str_name.', '.$str_address.', '.$str_county.', UK');
 attempt_geocode($url.$str_address.', '.$str_county.', UK');
 attempt_geocode($url.$str_address.', UK');
 array_pop($arr_address);
 }

This starts with the longest, most detailed address string, and then gradually cuts the string down (possibly sacrificing accuracy in order to get a passable geocode), with attempt_geocode() exiting the loop on success.

But the fact that longer addresses can lead to incorrect geocodes meant I had to work in a way to start off with shorter addresses, and if that doesn’t get a geocode then gradually shorten them and keep trying to geocode. So I’ now have:

 attempt_geocode($url.$arr_address[0].', '.$str_county.', UK');
 attempt_geocode($url.$arr_address[0].', '.end($arr_address).', '.$str_county.', UK');
while(count($arr_address > 1) && !$str_lat)
{
 $str_address = implode(', ', $arr_address);
 attempt_geocode($url.$str_address.', '.$str_county.', UK');
 attempt_geocode($url.$str_address.', UK');
 array_pop($arr_address);
}

A long process in order to get a geocode that could still quite likely be wrong, and even if it’s basically correct might not be as accurate as a postcode; but nevertheless an improvement on what I had before.

A glimmer of hope though is that google maps itself doesn’t suffer from this issue – both address versions return the same accurate point on the map, and as someone pointed out to me on stackoverflow, google maps is in beta, so maybe teh geocoder API just hasn’t been updated to the newer, better address parser, and maybe one day reliable geocoding for free in the UK will be a reality. Also, somebody has found a way to geocode in the UK using postcodes, by hacking together the google maps and search APIs, and I may well try it, as this address geocoding malarchy leaves a lot to be desired. (*edit – turns out it’s heavily reliant on javascript so can’t be used for geocoding masses of pages without slowing down your browser.)

Finally, if this article wasn’t any help, there’s loads of geocoding links here.

Crawling websites for data using php running in a browser

I’ve had an idea for a website for almost a year now (won’t spill the beans just yet though) and today I finally started work on it. To lift the veil of secrecy a little, I’m putting information about certain places onto a Google Map, because somehow nobody has thought of doing it yet.

All that information about the places is already available on the internet, just not embed in a map, so my first step was to crawl some websites to get hold of all that information. A slight problem I had is that this required running a script to automatically trawl through all the pages. I only know PHP, which as far as I was aware, could only be run in a browser, and browsers time out after a while, so it would be impossible to just leave it running.

A little more research revealed that it is possible to run PHP scripts as stand-alone entities outside the browser, but only using something called CGI. I had no idea what CGI was, and my hosting company don’t allow you to use CGI anyway. But I did manage to find another solution.

Although a php script crawling lots of pages for data would cause a browser timeout, a script crawling just one page almost certainly wouldn’t. So what I had to do was:

1. Write a script that crawls the data of one page but…
2. … checks what page was last crawled and moves on to the next one when it starts and…
3. … tells the script to execute again once it’s finished.

1. will differ depending on your needs, but my solutions to 2. and 3., I believe, provide a good technique to crawl web pages if you only know PHP.

Solution to 2.

Each iteration will presumably write the information to a database. Provided you’re iterating over an integer (e.g the webpages are of the form http://www.thesite.com/thepage?id=theinteger) then you’ll probably be storing that integer in your database. Then the following code at the beginning of your script will advance you to the next web page to crawl.

 $last_entry_query = @mysql_query("SELECT theinteger FROM thetable ORDER BY theinteger DESC LIMIT 1");
 if(($last_entry_query) &&mysql_num_rows($last_entry_query)) {
    $last_entry_row = mysql_fetch_array($last_entry_query, MYSQL_ASSOC);
    $last_entry = $last_entry_row['int_ukcs_id'];
 } else {
    $last_entry = one less than the first entry (starts the script off);
 }
$current_page = file('http://www.thesite.com/thepage?id='.($last_entry+1));

You can stop and start the crawl whenever you like as the database will always tell you which page to crawl next.

Solution to 3.

Really simple this one. At the end of the script you need to run it again. What better way than just redirecting the browser to the same page again.

if(mysql_affected_rows() == 1){
    header('Location: http://localhost/campcrawl/index.php?id='.$last_entry);
 }

Still one little tweak though; browsers will tend to limit the number of times a page can be redirected to in order to avoid infinite loops. In Firefox, to over-ride this go to about:config  and change redirection-limit to something really big. If there is a danger of an infinite loop you can limit the number of iterations in your script with a counter or a timeout, but for me it wasn’t a  problem as just closing the browser tab stopped my script.

The reason this technique is workable is that even though it requires a browser to run the script, with today’s multi-tab browsers, and the fact that all the calculation is done on the server, mean that it doesn’t infringe on whatever else you’re using the browser for (aside from maybe occasionally having to refresh the tab running the script as sometimes it stops for seemingly no reason, but that might just be bad programming from me).

So that’s stage one nearly completed (2,000 out of about 9,000 pages trawled so far, but at this rate should be fininshed within the hour). Now on to actually doing something with the data.