Alex's Adventures on the Infobahn - data

Data Tsunami

2010-11-19T11:39:00+00:00

The government has done another public spending data dump today. It's one of the policies that I was really in favour of before the election and it's good to see the coalition holding to it's word with this release. The raw data is available from the Cabinet Office and should soon appear on the government's data portal data.gov.uk. Reports of what is being done with the data is being followed by people like the Guardian Free our Data bloggers. Expect people like the Open Knowledge people to come up with some fancy visualisations of the data at Where Does My Money Go.

I'm sure there will be a number of stories about individual line items that get queried, something ministers want to happen. I'm hoping people will in general take a more holistic view of what's being provided. It won't benefit anyone if we tie up banks of expensive civil servants answering FOI requests to justify and contextualise individual items in the data set. However I'm hoping there will be some serious analysis on the trends of spending as well as leaving people more informed of what government actually does.

Personally I'm looking forward to next year when all contracts over 25,000 will be published. That should really make for some interesting reading.

Adding Google Juice to mutt

2010-03-17T07:35:00+00:00

As I've been mailing out invites I discovered a minor problem with my data.

My main email client is the fantastically functional mutt. It's terminal based but incredibly flexible. When it comes to mass sorting/searching your email it leaves GUI based clients standing. However now I'm a roving around with a Google Phone the majority of my contact data is in the cloud. While I have a small address file used by mutt it only has a few oft-mailed addresses in it.

Luckily thanks to Google's data APIs your information is only a few RESTful requests away. The goobook program provides a handy mutt compatible address book interface to this cloud data.

There is one wrinkle however. The configuration of the script involves putting some rather valuable login details in a plain text file on your home partition. While I like to think my machines are pretty secure and maintained you can always do more. Good security is defence in depth. A quick patch later and I can store those details in an GPG encrypted file that can be decrypted on the fly when required.

The final piece of the puzzle is creating these encrypted config files in the first place. Although you can do this by hand from the command line I find the best method is using EasyPG (now part of Emacs 23). This will automatically cause any files with a .gpg extension to be encrypted. You can control the Emacs mode selection and default encryption key to use by using file variables in the header comments of the file.

It's not all perfect though, when enabling EasyPG I had to do the following:

(if (maybe-load-library "epa-file")
    (progn
      (setenv "GPG_AGENT_INFO" nil) ; gpg-agent confuses epa when getting passphrase
      (epa-file-enable)))

The problem seems to be that when GPG agent runs in terminal mode it confuses Emacs/EasyPG. By suppressing the GPG_AGENT_INFO environment variable EasyPG will fall back to requesting your passphrase in the mode line. While it takes care to flush the value as soon as possible it does open a small window of attack if an attacker can cause emacs to crash and dump core.

Going Mobile

2008-01-08T14:33:00+00:00

I spent bits of yesterday getting final meter readings in and cancelling my Sky account. The rest of it was taken up with a fair amount of cursing at Call of Duty 4 as I struggled my way through act 3. Having the rest of the household point out the danger of hand grenades every time I bit the dust again didn't help that much. I'm really quite enjoying the game but it does get quite hard in places. I'm tempted to say it's one of the best FPS games I've played and I'll stick my neck out and say the experience is probably better than playing on the PC. After all not many PC's have the wide-screen display my PS3 does. Although a lot of PC gamers pour scorn on the controller vs the classic mouse and keyboard I think it's actually a better control method. The two sticks give quite fine control when you are maneuverings and of course all the buttons for grenades and weapon switched are in ergonomically helpful places.

I switched my mobile tariff a few days ago to one that included a decent amount of data. I subsidised this by dropping the number of free minutes as I wasn't quite using all 900 every month. As a result I've been playing with the mobile web.

First up is Opera Mini. It's a little Java app that runs on your phone and runs as a clipping browser. This means it takes normal non-phone optimised web-pages and attempts to render them in a vaguely sensible form. Generally it works very well although I personally preferred the older versions rendering of the BBC new site. The new version added a panning display instead of forcing the text to flow down one column which was a little simpler to read. The only website that has so far failed is the ever present Facebook and that could just be down to finger trouble.

I've also played with some of the Google services. Of all the services Google Calender and Google Maps (separate application) are the most mature. The calender functionality is basic but enough to check your appointments while on the go. Adding a simple events with a natural language (e.g. "Meeting at 1400 tomorrow") works well enough. Given the growing ubiquity of mobile data services I can see my days as a loyal Palm user coming to an end. I've been using the calender a lot over the last year and especially like the ability to have different ones with different collaboration profiles. It's extremely handy for Fliss and I to have one shared calender for all out activities (even though I suspect I rely more on electronic memory augmentation than Fliss does). If the integration with Evolution pans out it will pretty much solve all my calender requirements.

The Google Maps application warns you it is data-intensive but on modern 3G networks it still runs pretty quickly. They have taken care with the design of the UI, especially in the zoom control. You can very quickly get a map of the right scale for navigation in the limited space of a phone display. The only disappointment is the non-GPS "My Location" service isn't supported on my phone. The list of supported devices is a little vague.

There is not much to say about the mobile version of Google Reader except that it works reasonably well. However I suspect the reading of large numbers of RSS feeds is going to hard to optimise for a mobile phone sized screen.

Google Apps is also available in a limited mobile flavour. You can't edit documents but you can certainly view documents and spreadsheets (if you select desktop mode, the default spreadsheet view is one column at a time which isn't all that useful) with Opera Mini. I don't expect to be using it much but it certainly could be useful if you just want to check the last time you called a certain utility company while in the pub.

The last app I played with was MidpSSH which is a GPL'ed Java ssh client implementation. I doubt you will be wanting to do much with the hassle of text-typing to the command line but it may prove useful for an emergency server reboot or service restart. Having said that I haven't managed to log onto one of my boxes yet (I haven't tried very hard yet).

Who can you trust?

2003-08-18T18:42:00+01:00

I was reading another href=http://news.bbc.co.uk/1/hi/technology/3154479.stm>article on MS
WORD on the BBC site today. Sometimes some of my friends grumble when I
ask them to send attachments in standards based formats that I can
easily read (although href=http://www.openoffice.org>OpenOffice.org usually does an
ok job). But really its not just for my convenience...I hope people realize every time they save a word document they are trusting
their data to:

An undocumented file-format (other programs have to guess how to read them)
A binary file (i.e. you can't open it with a text-editor, say to recover data from a corrupted file)
A unnecessarily bloated file (Word save all sorts of junk in a .doc file, have you ever wondered why your cover letter takes 100's of K?)

You don't have to switch to Linux, or even change your word-processor.
Its as simple as clicking Save As and selecting a format like RTF. Not
only will I be happier, but should you ever want to change your
word-processor, or recover data believe me you will be too!

p.s At almost at the same time as me can I wish href=http://www.debian.org>Debian a happy birthday.