Running Linux in QEMU’s aarch64 system emulation mode

Since I started working on aarch64 support for QEMU the most frequently asked question I got was “when can I run aarch64 system emulation on QEMU?”. Well wait no more as support for a VIRT-IO based aarch64 board was recently merged into the master branch of QEMU. In this post I’ll talk about building QEMU, a rootfs and a kernel that will allow you to start experimenting with the architecture.

Continue reading ‘Running Linux in QEMU’s aarch64 system emulation mode’ »

Anyone interested at an Emacs BoF at LCU14?

I’m fairly atypical as a Linaro employee because I have a desk in a shared office. This means I get to participate in technical banter with the other employees based in the Cambridge office. However it has become quite clear I’m surrounded on all sides by VIMers with only one potential convert who wants to try Emacs out “one day”. As a result I thought it might be nice to have an Emacs Birds of a Feather (BoF) session at LCU14?

BoF sessions are basically an informal gathering where people with a shared interest who come along to swap tips and stories about their area of interest. In the context of LCU it would be an opportunity to network and meet fellow Emacers. So any interest?

Boutique modes

I’ve recently started a new job at Linaro which has been keeping me very busy. My role combines the low-level fun of Dynamic Binary Translation that I so enjoyed at Transitive and the fact Linaro is a fully Open Source company. I work directly on the upstream projects (in this case QEMU) and with the project community. In many ways it’s my ideal job.

Of course I’ve quickly built up a reputation as the Emacs guy in the office surrounded by a sea of VIMers although one of the guys does profess a desire to learn Emacs “one day”.

One of the first things I did was move all my email into Emacs. I’d long been dissatisfied with the state of Thunderbird (and previously Evolution) that I took the opportunity to migrate to mu4e. I spend my days slinging patches and going through mailing lists so I really appreciate being in my editor while I do that.

I have also started streamlining some of my work-flow with a few specialised extensions to Emacs. I think I’m finally comfortable enough with elisp to have a pretty good stab at solving most problems. I’m also appreciating the ability to leverage the mass of Emacs code under the hood to make incremental tweaks rather than solve everything at once. I thought I might give you a tour of the code I’ve written so far.

First up is risu-mode. RISU is the code testing tool we’ve been using to verify our aarch64 ARM work in QEMU. The .risu file is a template that’s used to specify instruction patterns that the tool then uses to generate random code sequences with. risu-mode is really just a bunch of regex expressions wrapped in the mode machinery that highlights the elements on the page. It doesn’t sound like much but when you are working through a bunch of patterns looking for bugs it’s easier on the eye when the different elements are coloured.

Next thing I wrote was my own QEMU mode which is a simple comint-mode based mode for launching QEMU system emulation. It’s still very rough and ready as I’m mostly working on user emulation but I suspect it will be handy once I start on system emulation stuff.

Finally there is lava-mode. LAVA is Linaro’s automated test and validation framework. Although it already provides command line and web interfaces I thought it would be nice to launch and track test jobs from within Emacs itself. The job control files a JSON based so I built on the existing json-mode and a slightly patched version of xml-rpc.el to add job submission. I’ve started a simple job tracking mode that uses the tabulated-list-mode framework and eventually I’ll link it into the tracking library so job completion will be as seamless as my IRC work-flow.

So there you have it, a bunch of code that may well not interest anyone else but shows how Emacs provides a very rich base of functionality on which to build up tools that are useful to you.

Does anyone else want to share an example of their esoteric extensions? What’s the most obscure thing you’ve built on top of our favourite text editor?

Edit with Emacs v1.13 now available

I’ve just pushed the latest version of Edit with Emacs to the Chrome App Store. Hopefully most people are already tracking the latest edit-server.el via MELPA but this does introduce a few minor fixes to the extension itself. A new piece of functionality is the ability to trigger bringing Emacs to the foreground from a key-stroke within Chrome. I added this to support running Emacs on ChromeOS which together with my chromebooks.el package gives me a rather nice development environment without having to dump ChromeOS.

So new for v1.13

Extension

* Change the handling of hidden elements (fix bug #78)
* Add debugging for erroneous hidden text areas (#93)
* Add keyboard shortcut to bring Emacs to foreground
* Pass clipboard contents to foreground request

edit-server.el

* add advice to save-buffers-kill-emacs to avoid prompting on shutdown
* add autoload cookies
* fix bug with format chars in url (#80)
* don’t call kill buffer hooks twice (#92)
* don’t set-buffer-multibyte on process buffer
* support the “foreground” request with optional clipboard contents

Get the latest from the Chrome Webstore.

Travelling with the truffler

We’re are about halfway through our family holiday to the remote ends of the earth. It has been the first time we’ve taken Ursula on a plane so we thought we’d make it a big journey while we are at it.

To her credit she was mostly fine with the 24 hours on a plane required to get to the other side of the world. Most of the tears were during take-off and landing when it was hard to explain pressurisation to a 16 month old child. There were a few other snatches of complaint due to tiredness but otherwise it went well. It helps that she is a very cute child who instantly won over the cabin crew who were keen to help keeping her amused. She even had a freshly prepared meal of Salmon Fried Rice cooked for her by the First Class cabin crew.

Before we left the UK I had left Ursula playing in the kitchen while I sorted something out in the living room. When I came back into the suspiciously quiet kitchen I found the following example of toddler OCD:

OCD much?

An interesting aspect of having psychologist for a mother-in-law is the wonderful insight she gives me on how the mind works. It had only been few weeks earlier that we had been talking about a common behaviour that often precedes a spurt in language development. It seems as children start getting their heads around the concept of things belonging to categories they will start sorting their toys (or anything else) into organised piles. Obviously an understanding of the fact things can exist in categories is a prerequisite for understanding a lot of things about language.

I make the distinction between language and speech because the two are very different skills. Language is primarily a cognitive ability to map communicated ideas to abstract concepts. Speech is the vocalisation of that communication and involves fairly precise control of a dizzying array of muscles in our mouth and vocal chords. The mastery of this physical skill takes a lot longer so often the distinction between words is only recognisable to parents and others who spend a lot of time with the child.

Ursula has been understanding basic instructions for some time now and it’s now possible to send her off to fetch or carry things with a reasonable degree of success. Combined with her deistic pointing there has been genuine two way communication for some time. However perhaps due to fluke or stimulated by the new environment she’s in on holiday we are starting to see an explosion in words. She’s had the basic “Dadadada” and “Momamama” for sometime although it’s hard to distinguish from the baby babbling she’s been doing for a long time. We have long joked about her generic use of “dat/cat” for the cat and then pretty much any other object she was pointing at. Just before we left she had started associating “NaNa” with bananas (a favourite food of hers). We now have distinct sounds for birds, cats, dogs and my favourite “papa” for the Nexus 7 which we call the PadPad so as to avoid confusing it with the Apple brand product 😉

My mum found my description of all this behaviour very amusing as I swing between proud Dad and scientific curiosity. I will put on the record that I’m not treating my daughter as a lab experiment but I do find the whole development of language and mind fascinating. I understand now why watching your kids grow and develop is so often cited by parents as one of the main joys of parenthood.

On the death of Google Reader

You can probably tell the sort of on-line company I keep from the deluge of noise on the social networks regarding Google’s decision to shut down Reader. However we shouldn’t be that surprised. In fact some companies that source content from Reader have anticipated the need to collect content themselves.

I of course will have to make a decision at some point. However I’ll not do it today like a lot of Reader users have. The rush to try out alternatives has over-whelmed some open source based projects who were quietly growing organically. I don’t envy those that have to suddenly gear up their back-end systems because an Internet behemoth gave us 2593 hours notice to sort out a replacement.

I’m mulling over the difference between self-hosting and having someone else do it. I’m not overly worried about going for convenience if I know I can get my data back if I need to. In fact the knowledge that you can theoretically self-host might be enough. To be fair to Google their Data Liberation team made exporting all my Reader data easy.

Before I make a choice I need to decide what my priorities are. Currently I subscribe to 250+ RSS feeds. Obviously I don’t read every single post but I make extensive use of tags to quickly process through stuff I do need to see when I need to see it. Aside from news, blog posts, funny cat pictures I also subscribe to other data feeds like bug trackers, code repositories, and other data sources. I of course want access to all of this data at any point on one of a number of devices. This makes a web hosted solution pretty much a must. There is no point having the data on my desktop when I’m somewhere else. From my point of view I want it to be open source compatible because if the company hosting now decides it no longer wants to I’ll only have to move the data and not break my work-flow.

It would also be very useful if it had a public API so others can interact with the data. I don’t need the solution to be all provided by one company. It’s perfectly fine to have multiple 3rd parties sorting out the Android integration. I might even look to doing something to integrate it with my favourite editor (the name of which even my non-geek readers probably know by now). So far my experiment with moving all of IRC and IM into Emacs seems to be working well and should be a subject of another post.

Are you a Reader user? What are your criteria for it’s eventual replacement? Is RSS just a dying protocol or is the need to aggregate and sift through data becoming more important?

There may well be a much better way of solving this problem around the corner. I certainly am open to persuasion. But don’t take away my current preferred solution until I’m convinced I’m ready to switch 😉

The future is inching its way onto our set-top boxes

I have from time to time lamented the tardiness of media producers in getting content in-front of my eyeballs. It’s not a case of being tight with money, although giving Murdoch £30/month for a glut of programming I’ll never watch is *too much* for me. I’ll happily pay for the stuff I want to watch as long as I can watch it now on whatever device I happen to have to hand. Until it’s made that easy for consumers the pirates will keep winning by default.

There was an interesting quote from the boss of Netflix in a recent interview: “The goal is to become HBO faster than HBO can become us”. Netflix is moving from being just a content distribution company to a producer of content. And if they can create content as good as their new remake of House of Cards hopefully the days of production companies being tied to restrictive bundle contracts are numbered.

The fact that shows can be disconnected from the broadcast schedules also offers other compelling opportunities. The BBC recently aired a number of comedy sketch show pilots which we only caught on iPlayer. Indeed they are now going to start trialling iPlayer net first runs of new shows over the next 12 months. No longer does the success of a show have to hope that it was picked up by enough of the 11500 viewers that happened to catch a random scheduled slot. Each and every viewing can be accounted for and counted even if the show is a slow burner like Firefly which was cruelly cancelled before it built up it’s cult following.

So the balls now in HBOs court. How exactly to I pay you so I can watch the next season of Game of Thrones when it comes out?

2012 a year in review

In past years when I would review the year I’d flip back through my blog. For 2012 however I managed on average one post a month and the majority of those were geek related ones which in some ways are a lot easier to write.

The big change has been becoming a parent of course but at the same time I also took back the reigns of management as I became the head of the software team at work. The combined effect has been a year that flew by very quickly leaving a slight sense of bewilderment about where it has all gone.

I no longer feel quite as out of my depth as I did half way through last year. The first 6 months are pretty gruelling in terms of missing sleep and trying to interpret the needs and desires of something that can basically sleep or cry. However Ursula has grown up so fast you soon forget about what she was like and try and re-focus on what’s she’s doing now.

For starters she’s now a fully mobile and operational battle-truffler able to move under her own direction. Next to that she’s moved from simply reporting state (happy/unhappy) to communication. Granted every object she sees currently seems to be a cat (or possibly “that”) but I suspect her vocal cords just need to catch up.

She’s now pointing to objects she wants to interact with including multiple objects she’d like to combine (e.g. Bonjella and teething ring) like some sort console RPG game. She laughs at silliness, enjoys playing around and being chased and basically has a personality that is uniquely hers.

I used to think that it wouldn’t get interesting until she learnt to talk but of course my preconceptions about what it would be like are constantly being shattered. Sure it still gets frustrating at times especially for Ursula. She understands more and more of what her parents are saying without being able to respond the same way. As she gradually realises she’s an independent being able to manipulate the world around her she’s very quickly having to learn that she can’t control everything. I suspect that would put teenage angst in perspective if only we could remember our really formative years.

Well this post seems to have turned into yet another one about my child growing up. Other things have happened to me that are not Ursula related but really there not as interesting. Instead I shall try and look forward to what’s coming up. The annual February walking holiday is just round the corner and I’m looking forward to catching up with all the other new (and slightly more experienced) parents at that. We’ll be up in Manchester (without Ursula) for P&K’s wedding which we are really looking forward to. And we’ll be off to the land down-under for a much delayed holiday in April. Hopefully the next post will not be too far away 😉

Edit with Emacs v1.12 now available

Apologies for the extended delay in getting out a new release of Edit with Emacs. Real life has been doing a good job keeping me busy that spare hacking time is in short supply. A number of people have submitted pull requests and I’ve also done a bit of clean-up on the code.

The JavaScript in the extension has had the hacky change tracking code replaced with a modern funky 21st century Mutation Observer implementation. This should solve some of the more obvious performance problems on highly dynamic pages.

I’ve also implemented the iterative save method for edit-server.el. Unfortunately it doesn’t quite work as intended if the edited region uses encoding (e.g. longlines). In theory the buffer you have saved should have it’s contents replaced with the new data from the browser but it just doesn’t seem to work. For now I’ve just ensured the kill-ring has the real data in it so it’s not overly painful to restore formatting. Patches and/or explanations are of course welcome!

So grab it now from webstore. The changes are as follows:

Extension

* Upgraded manifest to version 2 for future releases of Chrome
* Added an “Enable Debug” flag to control logging to console
* Use Mutation Observers to better track changes in dynamic DOMs
* Fake a keypress when updating the textarea
* Bug fix for handling editable DIV elements

edit-server.el

* Now available from the MELPA package archive!
* Beta support for iterative C-x C-s saving (it works but is ugly, see kill-ring)
* Default to using UTF-8 coding for process communication
* Raise existing frame consistently on Mac
* Tweak the portion of the URL used to name buffers

Many ways to skin a GNU

It’s time for another examination of Emacs LISP and a selection of ways to solve the same problem. In this case it’s the behaviour of the etags-select package. This handy function will present a selection table when it finds multiple definitions of the same tag. This is handy if your global TAGs file contains reference to multiple binaries that might have a common code heritage. It even provides a handy function etags-select-find-tag-at-point which will check point to see if that is a tag. However if point is on a blank line I’d prefer it to just prompt me for a tag name.

First version:

(defun my-naive-find-tag ()
  "Find at point or fall back"
  (interactive)
  (unless (etags-select-find-tag-at-point)
    (etags-select-find-tag)))

Unfortunately this fails rather badly. I naively assumed etags-select-find-tag-at-point would return ‘nil on failure. Instead it bombs out with an error because etags-select-find expects a parameter and when find-tag-default fails it errors out.

Second version:

(defun my-working-find-tag()
  "Find tag and call etags-select-find-tag-at-point if it will work"
  (interactive)
  (if (find-tag-default)
      (etags-select-find-tag-at-point)
    (etags-select-find-tag)))

This works by checking find-tag-default will work before calling etags-select-find-tag-at-point. Of course there is some duplication here because find-tag-default will get called again once I know it will work. Dissatisfied I asked the stackoverflow community for suggestions. The first solution is to simply trap the error case.

Third version:

(defun my-try-except-find-tag()
  "Find at point or fall to etags-select-find-tag on error"
  (interactive)
  (unless (ignore-errors (or (etags-select-find-tag-at-point) 't))
    (etags-select-find-tag)))

This works by utilising deep lisp black magic to stop the error propagating and returning a ‘nil if it does. The (or (etags-select-find-tag-at-point) ‘t) line is to ensure a successful call returns something so we don’t then fall through. Interestingly the comments around subr.el mentions some of the keywords used may be redefined by common lisp.

Forth version:

(defun my-efficent-find-tag()
  "Find tag at point, caching find-tag-default"
  (interactive)
  (let ((ftd (find-tag-default)))
    (flet ((find-tag-default () ftd))
      (if (find-tag-default)
          (etags-select-find-tag-at-point)
        (etags-select-find-tag)))))

While I expected something like ignore-errors to exist this demonstrates the flexibility of dynamic languages like Emacs Lisp. The key is the use of flet to redefine find-tag-default so when it gets executed again inside etags-select-find-tag-at-point it simply returns the cached value.

So as usual with these posts I try to invite feedback. Which of these forms do you prefer? Would you solve the problem another way? Have you just learnt something new about Emacs Lisp?