Alex's Adventures on the Infobahn - development

Comparing Forge-Based and Email-Based Workflow for Open Source Projects

2023-10-22T12:22:00+01:00

In the open source technology universe, how teams coordinate, collaborate and contribute is determined by the workflow they opt for. At a high level, workflows can fall into one of two camps: Forge-based or Email-based workflows. Forge-based workflows gained popularity with platforms such as GitHub and GitLab, while Email-based workflows have been a stalwart mechanism for open source software development with mailing list platforms like GNU Mailman and SourceHut.

In this post, we will look at the benefits and drawbacks of the various approaches, hopefully lending insight to what will work best for your project.

Forge-Based Workflow

Forge-based workflows have revolutionised open source software development. Various advantages have made it an accessible choice for many, including:

1. User-Friendly

Forge-based platforms such as GitHub or GitLab feature a friendly GUI and offer excellent documentation, making it super easy for beginners to contribute to open source projects without needing an in-depth understanding of git and email tools.

2. Centralised and Organised

These platforms provide central repositories that make project management streamlined. Access controls, issue tracking, continuous integration, and other tools all exist in one place.

3. Collaborative Environment

The pull-request model common to forge platforms encourages collaborative code review, making it an excellent tool for open-source projects where code quality is a priority.

However, there's a flip side to this coin:

Cons: Risk of Vendor Lock-in

Choosing a specific forge platform means accepting their choices of features, tools, standard practices, and policies which subtly enforce vendor lock-in. For most open source projects this also means relying on the forge supporting projects with cost free access to features and CI time.

Email-Based Workflow

Despite being considered 'old school', email-based workflows still have merit in today's software development world. They offer:

1. Decentralized and Flexible

In sharp contrast to forge-based workflows, email workflows are inherently decentralized. This approach offers more flexibility for contributors and maintainers alike, as they are not bound to tools offered by a single platform.

2. Line-by-Line Review and Inline Feedback

Email-based workflows stand out when it comes to the review process. The nature of emails allows for contributors and reviewers to break down commits line-by-line. This allows for more granular attention to detail, leading to clearer communication about specific code changes. Additionally, the threading mechanism of emails allows for inline commentary during reviews, making it easier for contributors to respond and iterate on feedback. This approach can encourage deeper understanding and discussion around code changes, leading to well-reviewed and robust code contributions.

3. Enhances git Understanding

Contributors working with email-based workflows generally have a better understanding of git because it requires more hands-on actions when sharing code.

However, they too have potential drawbacks.

Cons: Less Beginner-Friendly

Email-based workflows requires a steeper learning curve and are less friendly for collaborative code reviewing, making it challenging for new contributors to a project.

Adapting Open Source Projects: Email to Forge Workflow Transitions

Transitioning from an email-based workflow to a forge-based one should be a meticulously planned process that respects the existing community's culture and comfort. It's pivotal to keep this move as an evolution rather than an abrupt change.

Initial steps in such transitions can include moving non-code elements to the forge platform. Functions such as issue tracking, documentation, and discussions could be the first set of activities transferred to the forge platform. This strategy allows the contributors to become proficient with the platform’s tools and interfaces while keeping the code contributions intact on email lists.

Your project might encompass several sub-systems, and it could be worth considering this while transitioning. Instead of transforming the entire project at once, one could start by moving individual sub-systems to the new workflow. This incremental, phased approach can prevent chaos and reduce the chances of any significant disruption in the project's flow.

Following the sub-system shifts, the project should then introduce acceptance of pull requests or merge requests created by maintainers. These PRs should still contain code that underwent email list reviews in the previous workflow, ensuring the robustness of code quality.

The final stage of the transition is welcoming all contributors to submit merge requests through the forge platform. The key is to ensure that contributors understand the reasons behind the shift towards a complete merge request approach. They should be given clear guidelines about how and why to use merge requests, making them comfortable with the transition, and empowering them to take full advantage of the tool's capabilities.

In transition, it's necessary to alleviate concerns, clarify intents, and provide resources to learn and help the contributors adjust to the new workflow. Both email and forge-based workflows hold their place and value in open-source. So, when transiting, the project needs and the contributor adaptability should be the cornerstone of your strategy. No two projects are the same, and neither are their paths in adopting new workflow models.

(Fuller disclosure: as @epilys noticed this post was generated via chatgpt. I may have spent more time fiddling with my pelican settings to make sure the author slug was properly shown than I did iterating with GPT4 on the article. It has had some light copy-editing since to clean up some copy and paste errors between iterations and some of the more artificial phrasing it used)

Almost there

2010-11-19T22:48:00+00:00

I thought I might just mention the current state of the development for Edit with Emacs. A number of useful contributions have come in but I want to be able the "incremental edit" feature in the elisp edit server. I had a go but got stuck (marvel at hack in progress). So if anyone fancies having a go be my guest ;-)

The rest is as follows:

Ryszard Szopa (1):
      Allow the server to send correct Unicode to Chrome.

gfxmonk (4):
      trigger a DOM change() event when textarea content has changed
      clarify that the shortcut key is Enter, not "E"
      only re-request edit when "x-open" header is "true", not merely present
      Made indentation consistent across all source files

Edit with Emacs v1.8

2010-08-19T11:10:00+01:00

After a relatively quiet period a number of patches have flowed my way so I thought it was worth pushing out a new version. Perhaps the most "important" feature is the edit box flashing and fading from yellow after being updated (like It's All Text). It wasn't that hard to do given jQuery and the colour animation plugin do all the heavy lifting.

We've added a new hook to the edit-server for pre-edit customisation. If anyone has some nice examples of using the various hooks it would great if you could add examples at the emacs wiki.

As ever the extension can be found at the Chrome Extensions site. Development versions are hosted at github.

Full Change Log

v1.8

Extension

* Added option to enable/disable visual edit boxes
* Improved feedback as editable elements come in and out of focus
* Updated text box will now fade from yellow after an update

edit-server.el

* Added edit-server-start-hook for additional customisation when edit starts

Thoughts on Java

2010-02-12T11:18:00+00:00

I've been spending the last week hacking around in Java. One of the components of the product I develop is the open source OpenNMS. As I've mentioned before it's been lightly modified by myself to blend in better with our code mainly in the JSP department for it's web interface. Last week I discovered I needed to make a slightly deeper change to the code to export some more data to it's scripting interface.

While I was exploring the code base and scratching my head at some of the exceptions being thrown I thought I'd have yet another go at making my Java development environment a little more integrated than binding the build script to "C-c c".

So far I have a couple of observations about Java code. The first is that it never seems to do very much. As you navigate the code base you tend to find a lot of simple skeleton classes, usually working as simple adaptors between one framework and the next. Thanks to Java's strong support for generics and interfaces you often find yourself looking at a class wondering what else it might do. None of this is helped by Java's standard/deep/directory/naming/strategy/of/doom. My usual tools of a decent programmers editor and grep start to show their limitations. Java is a language that is designed to be serviced by a full IDE to help you make sense of the whole system.

Having previously tried and failed to get JDEE working I thought I would capitulate and try Eclipse. I blithely thought that given the fact it's the standard Java developers tool with a long development history it would Just Work (tm). I was to be sorely disappointed.

I tried several different versions on both my home Gentoo machine as well as various PPA based versions on my work Ubuntu setup. A lot of times it failed to start up due to some Mozilla dependency issues and when I did finally get it started I couldn't get any of the plugins to install. The entire Eclipse stack is designed around the concept of plugins and it seems every Java framework comes with it's own additional plugin for Eclipse. Without the plugins I wasn't able to get a working build or any sort of source level debugging. Eventually I had to concede defeat and bin the IDE approach and return to some tedious grep work and piece together the structure of the software by hand so I could finally crank out what in the end was a fairly simple patch. I doubt I shall return to Eclipse.

I did have a brief tinker with JDEE which at least installed better from it's SVN repo (needing a one line patch). However the documentation is a little sparse on how to import an existing large project into it's view of the world. Annoyingly potentially useful links like the FAQ referenced on the main site lead to dead ends. I never got to the point of seeing if I had a working Java GUD integration. I hope to revisit JDEE in the future when I'm a little less pressed for time at work. It's hard to work up the enthusiasm for plumbing in support for a language you never intend to use for fun in your own time.

Finding Open Source Android applications

2009-10-21T19:00:00+01:00

I love my HTC Hero and growing number of applications that are available for it. However the marketplace really doesn't make clear the difference between free (as in beer) and free (as in freedom). For all sorts of reasons I'd like to be sure the code running on my new favourite device is Open Source. I've been scratching my head about the best way to find a decent applications and had been throwing all sorts of queries at Google and generally getting lost in a sea of links. A lot were irrelevant as proprietary apps get mentioned on news sites with Open Source sections.

Then I had a slight brainwave (you wondered what the burning smell was?). Seeing as the most important thing is having available source code I skipped the main Google index and used their excellent code search. As every Android application contains an XML file called "AndroidManifest.xml" every thing else is a process of refinement.

Migrating to Wordpress?

2009-09-21T17:39:00+01:00

Live Journal have been more than a little crap the last week or so. As many may know I actually host my blog on my own website but most people comment on the syndicated feed on Live Journal. I'm thinking of migrating my blog to the slightly more feature complete Wordpress and enabling comments on my own site (useful for posts that come in via Planet's). The question that leads on from that is if I should maintain the RSS feed for the benefit of Live Journal?

Are the people reading just following their Friends Page or actually following RSS feeds of their friends? Would people still want to comment on LJ or would they be happy commenting directly on the web-site? I'm fairly certain I could get Wordpress working with LJ's OpenID so you wouldn't need *another* login.

The hinterland...

2009-09-12T11:28:00+01:00

..that is waiting for the broadband to turn-up. The first night in the house was nice although lacking in curtains so woke up early this morning. We have a mostly functional kitchen although no fridge yet. Fliss is building furniture upstairs. There are still a fair number of boxes to go, including in what is designated as the library.

Scripts from Scratch

2009-09-04T16:18:00+01:00

I wrote my first from scratch python script* today. It's not the first time I've touched the language having submitted a number of patches to python based programs as well as spell fairly hefty tweaking existing code. But this was one of those tasks I would have usually picked up perl to do. I've never been a big fan of perl as although it's fairly easy to knock stuff up it also encourages all sorts of bad habits in the lazy programmer**. This leads to all sorts of sub-optimal interactions later on when you come back to the code later on. Python code is by virtue much cleaner, as Chris put it: "you can go back and edit the code again later without getting a headache". Next time I need a script from scratch I think I shall reach into the python toolbox again.

* Not quite true, I did a GUI prototype last year. It wasn't really a script though.

** What? All the best programmers are lazy, it's a positive trait

Microsoft vs Google, who will win?

2009-07-08T10:03:00+01:00

There was a slightly hyperbolic BBC article describing Google's Chrome OS announcement. Apart from the obvious significance of the two largest tech behemoths going head to head the BBC did wonder if it would spell the end of the push for Linux on the netbook platform. Of course they didn't actually read the blog post which describes the new Chrome OS architecture being essentially Linux* running a new windowing system designed to support the Chrome browser.

Google's use of Linux as a base shouldn't be surprising. For starters their entire server farm (or is that the "Google Platform") relies on Linux. They used Linux as the basis for their Android Mobile Phone OS. They employ a large number of prominent kernel hackers in the company. Google is very much a Linux company and mighty as they are I think they appreciate that re-writing a kernel is a little like re-inventing the wheel. There are areas that Linux is still a little immature in. It has yet to develop the definitive power management strategy that modern mobile hardware really needs. However Google's involvement can only accelerate work in these directions.

There are some other things worth noting. The push for "Browser as the platform" and "The Cloud" is going to ask some serious questions of the Free Software movement. When you have the freedom to run entirely free (as in libre) software on you hardware it will mean nothing if all your data and applications are proprietary services in the network. There needs to be some serious thought given to the compatibility of Software as a Service with the ideals of computing freedom.

*When I refer to Linux I of course refer to the kernel

Cryptographic Hashes

2009-06-26T09:12:00+01:00

If your new to cryptographic hashes or simply want an intro that isn't overly heavy detailed maths then I can recommend The Code Monkey's Guide by kernel hacker Valerie Aurora.

GStreamer and stdin

2009-06-16T14:25:00+01:00

GStreamer is the media framework used by the Gnome Desktop. For the most part it's working is hidden from users as media players setup up the processing pipelines required. However sometimes you just want to play something from the command line and that's where a lot of head scratching can be involved as you try and work out the correct incantation. Thankfully the developers have tried to make command line invocations simple. The "playbin" option mostly does the right thing:

gst-launch playbin uri=file:///home/alex/broken.ogg

However while I was indulging a little nostalgia and playing with sc68 I found myself wanting to use that other *nix convention of pipes. The sc68 player doesn't actually deal with playback but provides a raw audio stream that can be "piped" to an audio output. Unfortunately searching for "gstreamer pipes" on Google brings up a lot of overly technical discussions about GStreamer's multimedia processing pipeline which while undoubtedly powerful and flexible (and modelled after Unix pipes) doesn't actually help me with my problem. After some playing about with gst-inspect and grep I finally found something that looked promising, the "fdsrc" element. The fd refers to a "file descriptor" and it's relevance to pipes would probably be missed by someone not familiar with *nix terminology. Eventually I arrived at my solution:

./sc68/sc68 music/Jochen\ Hippel\ \(Mad\ Max\)/Atari\ ST/Cuddly\ Demo.sc68 | gst-launch-0.10 fdsrc ! audioparse ! audioconvert ! audioresample ! alsasink

Of course the real aim of exercise was to replace my current ring tone with something a bit more retro. Here another utility comes in, the LAME mp3 encoder. This follows the "standard" convention of using - in place of file names to indicate input (or output) should from standard input (or of course to standard output). The command line was easily guessed:

./sc68/sc68 music/Jochen\ Hippel\ \(Mad\ Max\)/Atari\ ST/Cuddly\ Demo.sc68 | lame -r - cuddly.mp3
Track      : 0/6
Title      : Cuddly Demo
Author     : Jochen Hippel (Mad Max)
Composer   : Jochen Hippel (Mad Max)
Replay     : built-in
Hardware   : Yamaha-2149
Start time : 0:00
Duration   : 06 13:07
Track      : 1/6
Title      : Color Shock
Author     : Jochen Hippel (Mad Max)
Composer   : Jochen Hippel (Mad Max)
Replay     : mcoder
Hardware   : Yamaha-2149
Start time : 0:00
Duration   : 01 03:12
Assuming raw pcm input file
LAME 3.98.2 64bits (http://www.mp3dev.org/)
Using polyphase lowpass filter, transition band: 16538 Hz - 17071 Hz
Encoding  to cuddly.mp3
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III (11x) 128 kbps qval=3

And bingo, one very retro ring tone for my phone :-)

Distributed VCS

2009-06-09T10:17:00+01:00

I've mentioned distributed version control systems a few times. It was interesting note the advice that Savannah is giving about recovering from a recent disk crash. Both Git andMercurial based projects can restore from local repositories as they contain all the revision history within them. The centralised version control systems are relying explicit backups being made. History has proved that getting backups done right is harder than it looks. If your running an open source project there is a lot to be said for having ever developer mirror you code for you. To slightly misquote Linus*: "Only wimps use backup: _real_ men just upload their important stuff and let the rest of the world mirror it".

* The original quote mentions tape and ftp which dates it a bit.

Professional Development

2009-05-07T18:25:00+01:00

I went to a embedded conference today (re: sales pitch). It's been awhile since I've been exposed to salesperson snake oil but I thought it would be useful to see what the state of the art in the embedded world was like. All in all it was a useful event and I did learn a few things.

First of all the first thing that struck me was WindRiver conversion to Linux. Their attitude was a welcome improvement on my last professional involvement with them which was very different. The company still has propensity to binary blobs as I watched their spiel for the Tilcon GUI environment. I wasn't sure exactly what it did better than existing toolkits like QTopia or GTK, especially as the back-end on Linux was X11!

I spoke to the MontaVista guys about their offering and got a few pointers to my holly grail of a decent embedded build system. I'm going to have a play with BitBake and have a go at getting Gentoo Embedded building. Maybe a custom distro for my netbook?

Hardware has moved on a little. Pretty much everyone is using JTAG for ICEs now, most with more advanced tools like integrated Logic Analysers and CPU Trace support. The talk from the Lauterbach guy was good. I was particularly impressed in their coverage tool which can integrate power measurement with code coverage to indicate mJ per line of code. Next time I'm responsible for deciding on development hardware I'll certainly get some of their kit for trail. My only real disappointment is none of the JTAG debugger boards have really worked on decent GDB integration. If GDB is supported it's usually by a hacked up forked version of an old build. A number of the sales drones expressed surprise that I would want to use a clunky command line instead of their pretty GUIs. I will use their GUI tools if I'm doing fancy timing analyses or coverage work, but sometime you just want to connect to the host see where it is and get a back-trace, command line GDB is by far the most useful debugging tool for well over half the problems I ever have to look at.

Weekends of the past and future

2009-04-20T11:02:00+01:00

We finally seem to be have returned fully to the UK time zone. This week sees a return to normal service.

We had Fliss' sister visiting for the weekend as she prepares to head of to sunnier (and more dangerous) climes for her MSc project. This coincided with my Mum's brother and wife visiting from Canada. As a result we celebrated with an extended family meal at local restaurant Sycamore House. Saturday came and went in a blizzard of news papers and some film I succeeded in passing out in front of. The tiredness of jet lag seems to hit me very quickly and suddenly making it very difficult to stay awake.

Sunday involved to Sudbury for breakfast and dropping Steph back at the family home. We took advantage of the lovey sunshine to tackle to the garden. I'd tackled the 5 weeks of unrestricted growth earlier in the week with a general hack attack, however it still wasn't neat. Fliss gave the lawn a second mowing and also cleared up and trimmed all the borders. I was directed to attend to one of the flower beds and give it a general weeding. We have a plethora of seeds for various vegetables which we shall attempt to propagate over the next few days. The garden now looks a lot neater and ready for the summer season of BBQ's and gatherings. Even at this early stage of spring the patio area is still proving to be a lovely sun trap. Despite the days hard labour we made it through Sunday's film even though it was over 2 hours long. I did fall asleep pretty quickly though!

We were planning to visit Manchester next weekend however there is a confluence of sisters and mothers which makes staying down south the more sensible option. We would like come up to Manchester soon and celebrate our engagement amongst our northern friends. It may be worth combining with some other event going on up there so I'm open to suggestions (as well as offers of crash space!). We're thinking sometime in May/June. Ideas?

TED TALKS: Electric Cars

2009-04-18T17:30:00+01:00

I have to say this guy certainly has the vision thing. I share his thesis that green tech is going to be growth opportunity in the next few years. I just wonder how much software it's going to need.

Giving up on Xorg

2009-03-02T12:32:00+00:00

You may recall the adventures of playing with the bleeding edge of Xorg. Well I've given up, I just don't have enough time for tinkering with my main machines display. I managed to get a locally built Xserver after a fashion but I keep hitting various brick walls. I've basically committed myself to trying out the latest driver releases in my working distro xorg setup as and when they are released and tracking and offering feedback on a few bugs.

It's a shame really as accessibility to new developers is a key feature of any successful open source project. While Xorg has come a long way and are making pro-active efforts at improving their code-base and interaction with upstream projects it's still hard to jump in and try. This is of course complicated by the type of project X is. It touches the kernel, the 3D subsystem and the windowing subsystem and is by it's very nature a complex beast. Hopefully things will improve over time and next time I have a desire for compiz bling things will have improved a few more incremental steps. In the meantime it's back to having grouped tabs on my browser and using Gnome's workspace setup.

Apropos my last post

2009-02-11T23:05:00+00:00

The future is here. While I have reservations about wide scale surveillance and recording every e-mail I send it seems very sensible to equip the Persons in Blue with personal video recorders. Especially if it reduces the amount of paperwork the Police have to fill in when the could be out on the streets.

Some thoughts on that compiz bling

2009-02-09T12:03:00+00:00

I spent some time pimping my Samsung NC-10 netbook over the weekend. I had been having problems with Firefox slowing down and becoming sluggish and unresponsive. A little digging around later and I unearthed the culprit as being the closed source Flash plugin. Although well behaved Flash applications like the YouTube and iPlayer viewers don't seem to cause a problem a lot of other flash instances where being created with general browsing. A quick installation of Flashblock improved the situation massively. I now only have Flash starting up and running when I want it to play video.

One of the main deficiencies of the NC-10 is it's rather puny touch-pad and rocker style click buttons. I knew this was an issue when I bought the machine but as I'm mainly keyboard driven didn't think it was a major problem. However it was causing me trouble when selecting windows, especially as I usually keep them maximised. However the NC-10 does have a simple Intel 3D graphics chipset with fully open source drivers so I spent some time tweaking Compiz for my setup.

As can be seen Compiz is responsible for the flashy rotating 3D cubes and general "bling" on the Linux desktop these days. While I had it on my old work machine it was a pretty demo but somewhat marred by the binary nVidia drivers which would periodically freeze my machine. However with careful tweaking it makes my netbook a lot more usable. I can now use key chords (super+[DWAE]) to bring up the desktop, windows on work view, windows of app (i.e. all Firefox windows) and a full Expose type view. The touch pad is now relegated to pointing at a window (no click required) to bring it into focus.

As a result of the way Compiz works every application can always draw to a "display". This means you get, relatively cheap computationally, thumbnails of the actual window contents for these summary displays. So instead of heavily using tabs on my netbook which take up valuable screen real estate I now open fresh windows for every page. Super+A then easily brings me thumbnail views of all my current browser views to switch to a new window.

There remains a debate as to whether enabling Compiz on a Netbook increases or decreases battery life. Without actually measuring the consumption in-line it remains to be seen. It could be that by using the otherwise powered graphics chip to move graphics around you save the main CPU from having to run faster to manually copy stuff around into the frame-buffer. Unfortunately I don't really have a objective way to measure it right now.

A few days of purgatory

2009-01-22T11:40:00+00:00

I spent most of yesterday (and will spend today) operating in the Windows world. Every time I do this it reminds me of how much I hate it. Having to use IE certainly reminds me that Firefox is a much much better browser. I also miss having a comfortable editor or a powerful shell although I do note the DOS prompt has finally caught up with TAB completion for file paths. It still has nothing on a decent *nix command shell though.

The reason I'm putting myself through this is I'm tweaking a psychology experiment for Fliss' mum. The experiment itself is written in Python and commendably uses an Open Source framework called Vision Egg to generate and display the images. Unfortunately the experiment needs to run on some special graphics hardware which doesn't seem to have Python support (odd considering previous generations of hardware do seem to). The system however does supply examples in MATLAB, C++ Builder and a variety of .NET variants. So the choice was either to figure out how to drive the hardware from Python or convert the experiment to MATLAB (the only other tool the department has).

Fortunately Python's built in ctypes library makes interfacing to arbitrary libraries really easy. Once I'd dug out the C headers and loaded the DLL everything else was pretty mechanical. Today I just need to figure out the most expeditious way to thunk VisionEgg to VISAGE.

Python is fast becoming a favourite programming language of mine. I would probably still reach for Perl if the script involved much text grokking having invested so much time in learning it's regexp implementation. However for prototyping or extending a program through plugins Python makes much more sense. It seems remarkably consistent in it's behaviour and has yet to pull something unexpected on me. The interactivity of the shell/interpreter makes exploring approaches a lot easier than the multiple edit/save/run cycles of Perl.

I still miss having a decent editor though :-(

Codec Fixing

2009-01-02T14:33:00+00:00

Rockbox happily made it through the New Years Eve gathering without frustrating anyone with its occasional rough edges. However I am having problems with .m4a files (unlocked iTunes files) causing Rockbox to throw a strop. Unfortunately debugging is made harder by the inability to attach the very much copyrighted tracks to the bug report. I don't know how much love the m4a playback routines have seen. I assume most right thinking developers mainly run Oggs and Mp3's which of course work without any problems.

My hunt for un-encumbered test media continues.

Wikipedia and the IWF

2008-12-10T15:36:00+00:00

It seems the IWF have backed down in their attempt to censor Wikipedia. One of the reasons they cite is the unintended consequence more people having seen the questionable image since they pointed it out. This is sometimes referred to as the Streisand effect after Barbra Streisand's attempt to censor the internet.

There are a number of issues raised by the whole affair including questions about censoring stuff on the belief it may be against the law. However I'd like to concentrate on the technical issues involved.

The Cleanfeed system is actually fairly well thought out. For starters they only filter URLs rather than whole domains. This means an offending page can be filtered out without knocking out a whole site. I'm sure people can imagine the outcry blocking the whole of Wikipedia would have caused.

They way the filtering is handled is by passing your web-request through a transparent proxy. Obviously if everybody had their web surfing routed through this one proxy everyone would suffer from slower surfing. For this reason Cleanfeed utilises a two stage process where only requests to sites in the IWF blacklist are forwarded to the proxy. Unfortunately for UK Wikipedia users this meant that all requests to the site went through the proxy. From Wikipedia's point of view this meant the entire UK population where making requests from a single IP address. As an IP address is the unique identifier of a device connected to the internet Wikipedia use it as a way of blocking sustained vandalism. The side effect being that all anonymous edits of Wikipedia where blocked once some-one had committed some vandalism through the proxy. Another unintended consequence of the IWF filter.

There are a number of lessons that can be drawn by the whole episode. However this case is should be held up as a example to those people that think technical solutions to filtering the internet are workable. Unintended consequences indeed.

Batteries Included

2008-11-14T14:29:00+00:00

I posted my first simple patch for Rockbox to their mailing lists yesterday. It's a pretty simple one but I'm a big believer in trying to make the build system as flexible as possible. Also as I'm thinking of various experiments on my one and only iPod it would be nice not to nuke the stable software.

I've been a little concerned that my iPod has dud batteries. There have been a number of times it hasn't woken up to key presses until given some USB juice. However I'm currently running the battery benchmark tool and it's been happily playing away since I left the house this morning. It could be that spending a lot of the time playing with the setup the last few days is draining the batteries faster as the back-light is on. The other option is the boot firmware is getting itself into a funny state going through various USB connections (connecting to USB switches over to Apple's "Disk Mode" firmware on the flash).

There is a lot to like about Rockbox, not least the ability to play my music collection. However I think the area I'm going to focus on is it's handling of playlists. It's currently a little clunky and seems to have bugs in adding stuff to existing playlists. One nice thing about the setup though is they have a simulator which will run on your desktop. This means I can be a lot faster in tweaking the logic and testing things out. It even plays the music through SDL so you don't have to work in silence ;-)

No Follow

2008-11-11T12:26:00+00:00

I don't really want to mention SPAM blogs in-case I give them Google Juice. However this one is interesting just for the phrasing of the language. For example (from here):

"â€œTransitive congratulates AMD on its incoming promulgation of the newborn Quad-Core AMD Opteron (TM) processor,â€? said Ian Robinson, evilness chair of marketing for Transitive. â€œBy deploying solutions virtualization papers much as the award-winning QuickTransit for Solaris/SPARC-to-Linux/x86-64 or QuickTransit for Solaris/SPARC-to-Solaris/x86-64 on systems armored with these stylish procreation of AMD processors, customers crapper change senior servers with x86 stylish profession patch all avoiding the costs and delays of porting their applications. â€?"

I suspect there is some sort of automatic translate and back going on here with a press releases, the source of most spam blogs posting.

Spot the bug

2008-10-30T17:21:00+00:00

Another example from a long line of bugs that had me scratching my head for a while:

if (sscanf(dirname, "%8x", &scanf_id)==1)
{
// We can  extract the ID right now
su = alloc_vsSu((uint32_t) strtoul(dirname, NULL, 16));
g_assert(su->config->id.idAsUint32==scanf_id);
}
else if (g_strcmp0("default", dirname)==0)
{
// The  "default" has an id of 0
su = alloc_vsSu(0);
}

Can you tell what it was yet?

Athesit Activism

2008-10-21T22:44:00+01:00

It's interesting looking at the graph of donations for the Atheist Bus Ads. For something that kicked off it's fund raising effort today for a now modest looking Â£5,500 it seems to have caught the zeitgeist. Perhaps it's because their is a silent majority* that feels left out of the regular debates about religion that the take up is so high?

I wouldn't describe the campaign as militant atheism just another ad in a sea of adverts competing for peoples attention. The "probably" language is mainly there to skirt around the ASA guidelines** which would prevent the flat out assertion (although curiously other groups can make such assertions). However the use of "probably" does make the advert softer and less prone to sounding like preachy hectoring which would kind of defeat the point.

I'm fairly fluffy and certainly wouldn't want to prevent anyone from practising their sincerely held beliefs. It's hard to predict what this campaign will achieve than pointing out there is a chunk of the population that don't subscribe to any religious viewpoint. Certainly there are other organisations devoted to the more weighty issues of campaigning for removing religion from it's protected and privileged status in public life.

* based on no facts whatsoever. The ONS states we are 70-80% Christian but I don't know how much of that is due to the default effect.

UPDATE: ** From the campaign blog

Yacky Yacc

2008-10-20T16:19:00+01:00

I'd hopped I would get by without having to learn the mysteries of Bison and how to build parsers. Unfortunately parsecvs uses it to grok the CVS files. Someone has added some new binary files into the CVS tree and it's causing parsecvs to go into a death spiral of memory allocation. The backtrace shows yyparse() and yylex() in the call chain so I suppose I'd better learn how this is meant to work before I break it.

Snakes!

2008-10-06T19:24:00+01:00

I've been doing some prototyping recently in Python as an experiment. The idea being I can mock up a simple app with GUI with minimal effort and of course learning new languages is fun. A big driver is that it's not perl. An awful lot of perl is implicit based on context and a lot of the syntactic sugar you end up sprinkling about to make things explicit can make your code look as though it's been vomited on. I didn't fancy trying to cope with that in a moderately complex perl script.

So far I'm pretty happy with how quickly I've picked up Python. The principle of least surprise seems to be holding when using new things. The only problem I have is how to break an arbitrarily sized list into an arbitrary number of arguments for a varargs style function call. I suspect the solution will be beautiful in it's simplicity but I suspect I need to read up more on Python's concept of sequences. Of course this is simple in perl as all function calls basically just pass an array of stuff.

Meters

2008-08-06T13:32:00+01:00

I've been playing with emacsclient in an effort to make things like editing blog entries faster by re-using existing emacs sessions. It's almost doing what I want it to do but there doesn't seem to be a way of checking if there is already a server running withing emacs itself. I'm not touching tty support yet as the many hacks do indeed look quite hacky, especially with screen involved.

We had a water meter installed this morning which in theory should mean lower water bills. We do like the occasional bath but apparently you can have about 15 baths for ï¿½2.40 so we shouldn't be maxing out too soon. Gardening is a worry but really we should get a water butt and collect rain water for that. However I doubt we will bother while renting here.

Spent last night watching another chunk of the very funny Suburban Shootout last night. I'm giving the season 2 DVD some thought. Tonight we do the Dark Knight.

Folk Festival

2008-08-02T11:34:00+01:00

Yesterday was the first full day of the Cambridge Folk Festival. There was a slight delay to getting there as Mufi and TJ had been held up by one of those rare M6 accidents. However once they arrived we took the very environmentally friendly bus to the site.

It's a very gentle festival which is nice because it means the loos are OK, there is a selection of real-ales at the not overly crowded bar and a great selection of food. We spent most of the day in front of the main stage and met some interesting people who where next to our "patch".

Last nights bands included The Waifs (lovely bluesy aussie band), The Peatbog Faries (high energy celtic dance), Bill Bragg (political protest songs of a slightly leftish hue ;-), and the final band was festival stalwarts The Levellers (sing-a-long folk rock from my youth).

We shall see what today brings :-)

screenrc

2008-07-23T18:04:00+01:00

I've been using screen more and more out of habit recently. I used to use a lot of tabs in Gnome Terminal on each project I'm working on. Now I run one terminal per project with a screen session for all the sub-sections. It seemed a good time to revisit my .screenrc which looks something like:

# Ensure we run a fresh login shell shell -${SHELL} # I want more scrollback defscrollback 2048 # Set the caption to show: # current window # other windows # date/time caption always "%{+b Rk}@%H%{kk} %{Yk} %n %t %{-b -s}%{kk} %{wb}%W %=%{kk} %{wB} %c %D %d %M "

The only missing bit is getting the session name in there somehow. I've tried various experiments with the backtick command and echoing $STY but I expect it's not available to the context the script runs in.

Context, Attributes and JSP

2008-07-10T20:02:00+01:00

One thing that is rapidly becoming clear is a lot of different concepts get mixed together when you start playing with JSP. On the face of it it's a simple model, your JSP code get built into a Java class (of type servlet) which is a combination on normal Java code and JSP mark up which gets converted into Java when the page is loaded. At any point in your HTML you can escape back into Java code, do calculations, execute loops, generally compute things. However that approach can get very messy because you end up spreading logic all through your presentational HTML. This is why using an MVC pattern is so encouraged. Put all your logic and complex code at the top of the file and then display page expanding the odd attribute here and there.

However I don't think I've fully grasped whats going on behind the scenes. It doesn't help with Jetty I can't find where the intermediate Java source code is generated. Trying to understand whats going on when I'm not familiar with the cryptic backtraces has been frustrating to say the least.

Now I'm a little confused as to what exactly happens with the ${var} notation in terms of the java code generated. My earlier code has mutated to a list with a custom class. Hence I have had to generate the following:

<c:forEach var="entry" items="${model.jvmMemInfo}" varStatus="status"> <% // This is complex, I'm not sure how you can use ${} syntax on // arbitrary objects to expose the values, hence this bit // of inline java. I'm open to elucidation on how you would do so NameValuePair data = (NameValuePair) pageContext.getAttribute("entry"); %> <tr> <td> <%= data.name %> </td> <td> <%= data.value %> </td> </tr> </c:forEach>

However there are examples on the net showing arbitrary classes exposing their contents with this simple ${data.name} type syntax. I just couldn't get it to work. My NameValuePair class isn't much different from their one. I suspect this would be a lot easier to work out if I could see the generated applet.

Any Java web monkeys got experience with Jetty or can tell me what piece of obviousness I have missed?

JSP tags

2008-07-08T14:56:00+01:00

Arrggh. I've just wasted the best part of half a day scratching my head as to why I was unable to iterate through a Java HashMap in a JSP page:

<c:forEach var="entry" items="${model.jvmInfo}" varStatus="status"> <tr> <td> ${entry.key} </td> <td> ${entry.value} </td> </tr> </c:forEach>

It would of been immediately clear had I looked at the resulting HTML the page was echoing. Note to self, always check you have explicitly included the taglib you are planning to use.

rms on the BBC

2008-07-03T14:37:00+01:00

I see rms has a piece published on the BBC. You can read it here.

End of LUG Radio

2008-07-01T11:08:00+01:00

Well it seems the chaps at LUGRadio are calling it a day. I only discovered the podcast last year and although it's quite long and full of swearing I generally found it very entertaining. The irreverent style blended quite well with the occasional deep technical geekery and random wittering. Time to look for more podcasts for my extended cycles into work.

Linksys Hackery

2008-04-04T13:43:00+01:00

Now the decorating and carpet laying is complete Fliss has her brand new office. Unfortunately due to my lack of desire to run cables under the floor boards and past the bathroom she's missing the needed network connection.

Luckily Will had furnished me with a spare WRT54G so I proceeded to look into building a wireless bridge with it. It's not quite like the wireless bridges I've been building the last 4 days at work though, for starters it doesn't quite supply the same bandwidth :-)

It seems the land of 3rd party firmware for the WRTG has become pretty fragmented with a number of projects, some of which seem to have questionable adherence to the GPL. As I value my freedom I tried to find a project which had the most open development policy and offered the greatest flexibility. In the end I plumped for the FreeWRT project, which itself is a fork of the bigger OpenWRT project. Both seem pretty capable but the FreeWRT project does supply a rather nice web based image builder which clinched it.

The re-flash of the hardware went without a hitch and after a little faffing about with routing default IP addresses I had ssh only access to a minimalist router setup. It didn't take long (and a brief grep for "bridging" in the manual) to get a working WDS setup. Maybe one day I will get around to fixing up my main router with some pimped up firmware.

cvsps and the diet pill revolution

2008-03-27T11:12:00+00:00

One thing I've been doing at work is an experiment to convert the source code repository from CVS to something a little newer. This has been more than a little challenging.

The main problem is the tool cvsps is quite hungry for memory, especially on the rather large repository I'm trying to convert. It very rapidly allocates around 2.5 Gigabytes of heap memory (way more than physical RAM) to parse the revision history of CVS and build up all the change-sets. I set it off yesterday and shut down everything I could on the machine to give it as large a resident set size as I could. However parsing randomly allocated data in the heap is probably as close to a pathological use case for the virtual memory subsystem as you could get. As I logged of the machine last night I could see it doing about 1% real work while the rest of the time it was waiting on IO (I assume swap activity) and idling. Luckily by this morning it had got over that hump (although some of the git-repack calls are pretty memory hungry themselves). If I have to do the conversion again I'm going to invest some time into putting cvsps on a memory diet.

Location

2008-03-21T12:18:00+00:00

I'm not sure what did it but I upgraded the firmware on my phone last week and this week I noticed the "My Location" option now works on Google Maps while running on the phone. I haven't been able to find a Changelog for the firmware version (Dave/Sheila/Jo?) but I'm guessing it was a fix to the info about current base-station exported to the J2ME environment. Either way it's a cool feature, now I just press 0 and the map centers on where I (more or less) am.

Fliss' dad has arrived from the land of Oz this morning. At some point we shall start our heading towards the wilds of Birmingham for Maelstrom.

Exchange the root of All Evil

2008-03-18T17:26:00+00:00

It seems MS Exchange really is as bad as I've heard. Thankfully Evolution can talk to the in-coming/out-going mail bits of Exchange. It also happily slurps up invites and adds them to it's own calender view (which I assume I can publish somewhere that Google Calender could slurp). However when it sends the reply back to Exchange it gets cryptic messages about quotas which make no sense to the IT guy. *sigh* so much for interoperability.

I've been playing with a few UML tools to capture this products Use Cases. I'm not looking for anything very flash (certainly not going down the full code generation root) however it would be nice to have something a little swisher than a hand-drawn document. So far I've played with Umbrello and Gaphor. Both seem reasonable although the Gaphor package is a little broken, it may be easier to run from the bleeding edge. I gave up on Dia as it's UML support was more in the form of a widget palette. Anyone care to recommend any other FLOSS UML tools?

Day One

2008-03-17T22:09:00+00:00

Today has been a busy first day. The journey in was surprisingly easy, so much so I beat my new boss into the office. A brief chat with H.R. was quickly followed diving straight into the technical details of the system and where the product I'll be responsible for is heading. The first few days are always a little overwhelming, especially when initial timescales for delivery are quoted in 3 months. Luckily it was made clear the product development will be an iterative one. I still need to get some planning and analysis in place though as it would be silly to start writing code without some agreement from other stakeholders. I've got some initial coding to do with an existing product as a way to get familiar with the existing code-base although luckily it's divorced from the bundled system build so I was able to wack it into git so I can avoid the horrors of CVS. Hopefully by the end of the week my head will have stopped spinning quite so much and the mountain will seem a little shorter, if only by a few feet :-)

Maintaining the semblance of normaility

2008-02-28T17:42:00+00:00

I've updated the picture on my homepage to make it a little more contemporary. Credit's for the image go to Chris Flemming who took it during the recent weekend in Scotland. It turns out to be possible to get good pictures of me :-)

More Lisp

2008-02-12T20:32:00+00:00

I continue to gain confidence with (Emacs) Lisp. I used to have a hacked up (my-find) function that worked reasonably well but was a bit flaky and capable of missing stuff in other languages. It turns out it's faster to ask git a lot of these questions if your working in a source controlled environment. A quick hack later and f5 magically does that if it sees a .git directory when I start up. I feel I can justify mentioning the language on my CV :-)

Rhythmbox Shuffle

2008-01-30T12:41:00+00:00

It seems I'm not the only one unconvinced by Rhythmbox's shuffle mode. No sooner I mention it on the local IRC than one of my colleges points at some work he did over Christmas. While I've not got time to start running around patching my work box with a suitable new version I am now using the "hidden feature" of clicking Shuffle and Repeat which turns on the true random mode rather than the list based mode which is somehow flawed. According to Dave he's looked at the shuffle code which looks right and should ensure every song is played only once before it is regenerated. However the last few days I've been having repeating songs within 4 or so hours and my library is somewhere in the region of 8000 songs. It's been posited the list is either wrong or being regenerated too often. I'll add it to the list of things to look at should I ever have time.

More commentary than code

2007-10-29T17:23:00+00:00

Yet another small nugget of code has gone into a public source tree. When I get a chance I'll have to fix up the one that didn't make it.

Trashing the OS of "Developers"

2007-09-11T19:41:00+01:00

Sun would really like Solaris to re-capture the hearts of developers since loosing so many to Linux. They are certainly making strides in the right direction with their Open Source strategy. However sometimes the creakiness of the code does show up.

I've spent the best part of the day trying to get the mpstat command to build and still haven't gotten any closer. One of the reasons is you can't quite build things out of the box. Although gcc has been shipping in /usr/sfw for a while you can't just go and start building tools with it. You need a 40 lines of environment variables set up as well as additional packages installed. Even then the compiler wrapper script can throw up weird and wonderful errors that looks like it really didn't believe you wanted to build with gcc (and not with Sun Studio).

It's rare in the Linux world that I've not been able to build a random user space command with ./configure; make. It's true that you usually have to specify the installation of gcc when you install a distribution however it's never more than an apt-get/emerge/yum away. Linux is still my favourite development environment.

Network or Playback?

2007-08-28T23:26:00+01:00

I saw an interesting piece discussing Microsoft's woes with their band-aid fix for playing multi-media while on a loaded network. From the description of the scheduling work around it does seem like a school boy error, you just can't fix performance issues by tweaking priorities.

Although Linux is still meant to be trying to fix interactivity issues it's been a long time since I've noticed any conflict between my media players and other tasks such compiling. About the only performance issue I still come across is "morning swap syndrome" when a lot of overnight activity on your machine can lead to it taking a while to swap your open apps back into main memory.

Of course it's not all totally green on this side of the OS fence. Of the two wireless interfaces on my laptop the old Lucent Gold Card does utilise a rather high amount of processor time when doing large copies. It doesn't interfere with the desktop performance but it is a little disappointing. Looking at /proc/interrupts I can see it would certainly benefit from being able to support a pure poll mode although I suspect the main bottle neck is just the copying of data across the PCMCIA/PCI bridge. It is not yet enough of a bug factor to make me dig into the code though.

LSB, chkconfig and the frustration of multiple distro support

2007-06-06T18:17:00+01:00

I've had a frustrating time with getting some of our init scripts working on both RHEL4 and SLES10. The issues have mostly been to RHEL4's less than stellar LSB implementation. Despite having a compatibility header for chkconfig it still sees the LSB stuff and promptly fails to resolve any of the dependencies. As we are a 3rd party application we can't exactly tell our customers to install a newer version of chkconfig which works (which the Fedora Core 6 one does). It's also a logistic PITA if we start generating separate RPMs for both distros. In the end I took advantage of having the source code for chkconfig and sprinkling fprintf statements in it until I figured out what it was doing. I managed to craft an appropriate header that works properly on SLES and works (by virtue of understanding the bug in chkconfig) on RHEL. I far prefer actual application coding.....

In the News

2007-05-08T11:32:00+01:00

The project I'm currently working on has had it's first Press Release. There is even a competition being held for people that want to Beta test it when it comes out - although I'm guessing there is a friends and family clause somewhere in there.

I for one welcome our new Solaris overlords ;-)

Recreational Coding

2007-02-13T18:59:00+00:00

I had an itch to scratch yesterday for which I engaged in a little recreational programming. I'm toying with making this a full project because it seems to be a niche not currently filled by any FLOSS.

What I need/want is an intelligent configurable auto-sync daemon for my desktop. I'm thinking a little above your current "plug your iPod in and iTunes auto syncs functionality". Features I would be aiming for would be:

Multi-device. A one stop shop where I can configure what happens when I plug in my pen-drive, my music player, my external USB hard-drive.
Recognise new hardware devices and offer to manage syncing to them. e.g. "I've just seen you plug in a 512Mb USB Pen, would you like to set-up any auto sync stuff? Yes, Later, Never"
Backup functions: normal rsync or call a script for extra flexibility
Sync music. The current solutions are sub-optimal as they involve too much manual poking and don't consider small devices well. Features like:
- Where device storage < collection size allow smart selection, remove old tracks/albums to make space for new tracks for example.
- Track twiddling. Examples: Chain tracks on dance albums together into one long file to avoid skipping, transcode Ogg's into Mp3's for broken players.
Intelligent podcast support. Example: Only keep the latest of any program on the device.

Anyone want to suggest any other usage cases?

Luckily I don't think it will be too hard to prototype something up. Gnome already deals with the ickyness of auto-mounting devices. dbus provides a neat mechanism to watch for these hotplug events. I can probably hack stuff like file tagging into Nautilus using it's script interface. I had a quick play with Python last night and I think it has all the bindings I need. I think it's a small enough project to learn a new language while not being performance critical enough to need a hardcore C implementation.