2018-03-18

find.me.uk

As some of you may know, some time ago I had a bit of a play with the Open Data from the Ordnance Survey. This is free data you can play with.

One of the rather fun things is the location of every postcode in the country (well the centre of it), but there is a lot of raster map data as well, and tools to convert Eastings/Northings to/from Latitude/Longitude, and quite a few other things.

Whilst we make use of various bits of this data, I put the raster map information in to a web site called find.me.uk.

I also set up some magic DNS, e.g. you can do

dig LOC rg121qs.find.me.uk

to find the location of a postcode.

When I first made the web site, I messed about with the images, made smaller tiles, and organised them and renamed them based on E/N co-ordinates. It worked quite well, and made for a useful link to show locations. Nowhere near as good as google maps, but was a fun project.

A few months ago we had a disk issue, and because of the size of the mapping data, and the fact it is available from Ordnance Survey anyway, it was excluded from the back up. It should have been simple to restore and re-instate the old site, but turns out I did not document the re-tiling stuff very well, and it looks like the original mapping data may be a different format and filenames now, so I could not simply re-instate it.

Apparently loads of people use it though! So I have found a bit of time this weekend to re-do some bits and have another play. If you have safari then you'll find it basically works now. Other browsers, maybe not - just a blue square... I'll explain why... [update: it does now]

My idea, this time, was to try and make an image that, by construction of the URL, would be a specified size, scale, and location, to make it easy to include an image on a web page. We had this on our order page when someone selects an address, but it was not centred and was just one of my map tiles which was only like 100m square. I wanted something way more flexible that would allow embedding any size image I like, so I can link in to various of our internal web pages and so on.

However, what I did not want to do is have a back end rendering a custom image on every request, that is slow. So, I hatched a cunning plan. Make the image an SVG. You can include an SVG using an <img...> tag in a web page.

So I worked on the SVG, and the URL after https://find.me.uk/ allows for location by postcode, or by easting/nothing, and a scale (A, B, C, D, E), and image size as widthxheight, and finally a .svg on the end.

E.g. https://find.me.uk/RG121QS/B/500x700.svg makes a 500x700 SVG image centred on the postcode RG121QS. The idea is you can include in an <img...> tag.

In order to make the SVG, I use the tiles as provided by Ordnance Survey, so all I had to do was unpack the zip files you download. These are TIFF files, the smallest of which is 4000x4000 pixels. So in many cases only a single tile is needed suitably positioned so the location is in the centre. SVG makes it easy to add a semi transparent pointer and some text.

Bingo... Or so I thought...

Unfortunately I have two problems. Firstly it seems that whilst an SVG can include an <image.../> referencing a TIFF file, and that works, it does not work when that SVG is used in an <img...> tag in an HTML document. I have no idea why, and I cannot find anything by googling as to why this is. So that undermines the whole reason for doing it all this way. The other big issue is that whilst safari is happy to load a TIFF in an SVG, it seems other browsers are not (the SVG spec requires PNG, JPG and SVG at least but does not mandate TIFF). I'll have to do more experimenting on that one to confirm.

To work around the first problem I have put the SVG in an <iframe...> instead, which works, but there is no point, I could simply include the SVG in a web page and use that in an iframe. The whole <img...> thing is a bust, which is a shame.

As for TIFF, assuming that is the issue, I will need to convert to PNG or some such, and if doing that I may as well re-tile and rename everything. Except this time I'll make sure I have a proper script / code to do it, and document it. It would have the advantage of allowing more manageable tiles, such as 1000x1000. I won't go down to 100x100 as I did last time as that was overkill.

Obviously, as I'll be changing the site, this blog will be out of date at some point. It depends when I find the time to play with this project again. I may even open source and put on GitHub...

So, sorry for the delay - normal service will be resumed shortly.

P.S. It really was taking all day (well all night) to tile the images, until I discovered the trick. Convert the large tiff in to an mpc file using graphicsmagick, this is its internal cache format. Then use crop on the mpc file for each tile - it maps the mpc file to memory, so making each tile in under a second.

P.S. PNGs work in firefox and chrome where tiff did not work.

P.S. All changed to new tiles now, some more tweaks to make I expect... Looking good and nice and fast.

P.S. Seems to work in everything now, even Firefox

28 comments:

  1. Wow, lack of non-Safari (and IE) support is going to be a pain. Hope you do find time to code a workaround.

    The functionality of clicking a location and having it produce a postcode for that location seems to be gone, is this something likely to return later? And thanks for the https! just need to https speedtest2.aa.net.uk now!

    ReplyDelete
  2. https://developer.mozilla.org/en-US/docs/Web/SVG/SVG_as_an_Image

    "For security purposes, Gecko places some restrictions on SVG content when it's being used as an image:"

    This is entirely in the spirit of IMG, which is not intended for use as a catch-all embedding mechanism. If you want an iframe, use iframe.

    ReplyDelete
  3. How about support for Pluscode? https://plus.codes/ Then it can be a true global solution :)

    ReplyDelete
    Replies
    1. Or indeed mapcode, also free and usually more compact. Google seem to have reinvented this with OLC but not added much. https://en.wikipedia.org/wiki/MapCode

      Delete
  4. As to why not TIFF, TIFF is an anti-format. Perhaps it helps to know its history. When scanning ("digitising" was a new technology) the vendors wanted to standardise the files produced by their machines. But they couldn't agree on... anything. Where do scans start (top left of the page? bottom right?) and do they go left-to-right or right-to-left. Is 0 black or white? No problem though, TIFF begins by specifying all the conventions chosen and the rest of the file is just the raw data from the scanner. And this approach continued as features like JPEG compression, tiling, higher precision and so on were added. There are several clearly documented ways to write JPEG data to TIFF, none of which work, and then of course there are the extra ways used in practice by application software.

    As a result TIFF is a nightmare, and it makes no sense to support it on the web where you can expect not to just muddle along with partial support for one particular sub-format.

    ReplyDelete
  5. Chromium on Ubuntu is showing a big blurry blob :(

    ReplyDelete
  6. If you must use this data rather than the openstreetmap, then I would suggest you look at using the version from their tiling server https://os.openstreetmap.org/sv/{zoom}/{x}/{y}.png then you will get 6 monthly updates automatically.

    ReplyDelete
    Replies
    1. As I say, this was mainly an exercise in making use of the open data from OS. I am automating the process for making tiles, etc, so updates will be simpler.

      Delete
  7. As all the country done? My postcode came up with a blank page as have a couple of other BA (Batch) postcodes. I've seen it working for other places.

    ReplyDelete
    Replies
    1. If you give me a postcode, I'll check the data from O/S

      Delete
  8. I've found some L (Liverpool) and London (W1) postcodes that come up with a blank blue page.

    ReplyDelete
    Replies
    1. Have you found any L postcodes that work?

      Delete
    2. What's the most obvious W1 postcode? W1A1AA (BBC Broadcasting House)! That doesn't work.

      Delete
  9. I've worked through the districts shown by https://www.gbmaps.com/4-digit-postcode-maps/ba-bath-postcode-map.html. Then gone to that place in Google maps and zoomed to the place and picked a random pin to find a postcode.
    BA11SJ, BA26JG, BA27AZ,BA33RS, BA45SB, BA52BJ, BA69XE, BA77ES, BA80LH, BA99RT (where my car is being MOT'd today!), BA100AA, BA126FE (5 minutes walk from here) BA113RW, BA133PL, BA149QD, BA151LF, BA160YD, BA201EG, BA214NH, BA229TR

    It is ALL of BA that is missing!

    ReplyDelete
    Replies
    1. FFS, O/S sorted numbers after letters. Fixing.

      Delete
  10. You might find https://postcodeaddressfile.co.uk/downloads/free_products/postcode_district_area_lists.xls useful to check you have all areas.

    ReplyDelete
    Replies
    1. It is just a sort order issue, fixed now.

      Delete
  11. That's better. Liverpool and London W1 looking good now.

    ReplyDelete
  12. OK I have made something to process the csv files and sort them so I get postcodes right next time too. Also added 2831 postcodes from Feb update.

    ReplyDelete
  13. When the LOC record
    57 11 28.615 N 5 53 28.4244 W 119m 0m 25m 10m
    is entered into clueless I get the MAP URL
    https://find.me.uk/2147483648/2147483648/980x684

    but the resulting tile is visually rather uninspiring. What did I do wrong? Any thoughts?

    ReplyDelete
    Replies
    1. Are you putting a LOC record in your DNS? What domain? Ah, does that link to find.me.uk? If so may simply be new system does not know the format I used to use. Will have to take a look.

      Delete
    2. OK, yes, the current find.me.uk does not parse a LOC string. I may update that some time soon...

      Delete
    3. That should work, as well as 6 digit OS grid references.

      Delete
  14. General security note to future visitors, when querying the DNS for [postcode].find.me.uk vs going to find.me.uk/[postcode] be mindful that the former reveals your search to your DNS provider while the latter doesn't.

    ReplyDelete

Comments are moderated purely to filter out obvious spam, but it means they may not show immediately.