Alex's Adventures on the Infobahn - org-mode

Solving the HKG18 puzzle with org-mode

2018-03-26T11:19:00+01:00

One of the traditions I like about Linaro's Connect event is the conference puzzle. Usually set by Dave Piggot they provide a challenge to your jet lagged brain. Full disclosure: I did not complete the puzzle in time. In fact when Dave explained it I realised the answer had been staring me in the face. However I thought a successful walk through would make for a more entertaining read ;-)

First the Puzzle:

Take the clues below and solve them. Once solved, figure out what the hex numbers mean and then you should be able to associate each of the clue solutions with their respective hex numbers.

Clue	Hex Number
Lava Ale Code	1114DBA
Be Google Roe	114F6BE
Natural Gin	114F72A
Pope Charger	121EE50
Dolt And Hunk	12264BC
Monk Hops Net	122D9D9
Is Enriched Tin	123C1EF
Bran Hearing Kin	1245D6E
Enter Slim Beer	127B78E
Herbal Cabbages	1282FDD
Jan Venom Hon Nun	12853C5
A Cherry Skull	1287B3C
Each Noun Lands	1298F0B
Wave Zone Kits	12A024C
Avid Null Sorts	12A5190
Handcars All Trim	12C76DC

Clues

It looks like all the clues are anagrams. I was lazy and just used the first online anagram solver that Google pointed me at. However we can automate this by combining org-mode with Python and the excellent Beautiful Soup library.

from bs4 import BeautifulSoup
import requests
import re

# ask internet to solve the puzzle
url="http://anagram-solver.net/%s" % (anagram.replace(" ", "%20"))
page=requests.get(url)

# fish out the answers
soup=BeautifulSoup(page.text)
answers=soup.find("ul", class_="answers")
for li in answers.find_all("li"):
    result = li.text
    # filter out non computer related or poor results
    if result in ["Elmer Berstein", "Tim-Berners Lee", "Babbage Charles", "Calude Shannon"]:
        continue
    # filter out non proper names
    if re.search("[a-z] [A-Z]", result):
        break

return result

So with :var anagram=clues[2,0] we get

Ada Lovelace

I admit the "if result in []" is a bit of hack.

Hex Numbers

The hex numbers could be anything. But lets first start by converting to something else.

Hex Prompt	Number
1114DBA	17911226
114F6BE	18151102
114F72A	18151210
121EE50	19000912
12264BC	19031228
122D9D9	19061209
123C1EF	19120623
1245D6E	19160430
127B78E	19380110
1282FDD	19410909
12853C5	19420101
1287B3C	19430204
1298F0B	19500811
12A024C	19530316
12A5190	19550608
12C76DC	19691228

The #+TBLFM: is $1='(identity remote(clues,@@#$2))::$2='(string-to-number $1 16)

This is where I went down a blind alley. The fact all they all had the top bit set made me think that Dave was giving a hint to the purpose of the hex number in the way many cryptic crosswords do (I know he is a fan of these). However the more obvious answer is that everyone in the list was born in the last millennium.

Looking up Birth Dates

Now I could go through all the names by hand and look up their birth dates but as we are automating things perhaps we can use computers for what they are good at. Unfortunately there isn't a simple web-api for looking up this stuff. However there is a project called DBpedia which takes Wikipedia's data and attempts to make it semantically useful. We can query this using a semantic query language called SparQL. If only I could call it from Emacs…

select distinct ?birthDate {
  dbr:$name dbo:birthDate|dbp:birthDate ?birthDate
}

So calling with :var name="Ada_Lovelace" we get

"birthDate"
1815-12-10

Of course it shouldn't be a surprise this exists. And in what I hope is a growing trend sparql-mode supports org-mode out of the box. The $name in the snippet is expanded from the passed in variables to the function. This makes it a general purpose lookup function we can use for all our names.

There are a couple of wrinkles. We need to format the name we are looking up with underscores to make a valid URL. Also the output spits out a header and possible multiple birth dates. We can solve this with a little wrapper function. It also introduces some rate limiting so we don't smash DBpedia's public SPARQL endpoint.

;; rate limit
(sleep-for 1)
;; do the query
(let* ((str (s-replace-all '((" " . "_") ("Von" . "von")) name))
       (ret (eval
             (car
              (read-from-string
               (format "(org-sbe get-dob (name $\"%s\"))" str))))))
  (string-to-number (replace-regexp-in-string "-" "" (car (cdr (s-lines ret))))))

Calling with :var name="Ada Lovelace" we get

18151210

Full Solution

So now we know what we are doing we need to solve all the puzzles and lookup the data. Fortunately org-mode's tables are fully functional spreadsheets except they are not limited to simple transformations. Each formula can be a fully realised bit of elisp, calling other source blocks as needed.

Clue	Solution	DOB
Herbal Cabbages	Charles Babbage	17911226
Be Google Roe	George Boole	18151102
Lava Ale Code	Ada Lovelace	18151210
A Cherry Skull	Haskell Curry	19000912
Jan Venom Hon Nun	John Von Neumann	19031228
Pope Charger	Grace Hopper	19061209
Natural Gin	Alan Turing	19120623
Each Noun Lands	Claude Shannon	19160430
Dolt And Hunk	Donald Knuth	19380110
Is Enriched Tin	Dennis Ritchie	19410909
Bran Hearing Kin	Brian Kernighan	19420101
Monk Hops Net	Ken Thompson	19430204
Wave Zone Kits	Steve Wozniak	19500811
Handcars All Trim	Richard Stallman	19530316
Enter Slim Beer	Tim Berners-Lee	19550608
Avid Null Sorts	Linus Torvalds	19691228

The #+TBLFM: is $1='(identity remote(clues,@@#$1))::$2='(org-sbe solve-anagram (anagram $$1))::$3='(org-sbe frob-dob (name $$2))

The hex numbers are helpfully sorted so as long as we sort the clues table by the looked up date of birth using M-x org-table-sort-lines we are good to go.

Workbooks for Benchmarking

2018-02-21T20:34:00+00:00

While working on a major re-factor of QEMU's softfloat code I've been doing a lot of benchmarking. It can be quite tedious work as you need to be careful you've run the correct steps on the correct binaries and keeping notes is important. It is a task that cries out for scripting but that in itself can be a compromise as you end up stitching a pipeline of commands together in something like perl. You may script it all in a language designed for this sort of thing like R but then find your final upload step is a pain to implement.

One solution to this is to use a literate programming workbook like this. Literate programming is a style where you interleave your code with natural prose describing the steps you go through. This is different from simply having well commented code in a source tree. For one thing you do not have to leap around a large code base as everything you need is on the file you are reading, from top to bottom. There are many solutions out there including various python based examples. Of course being a happy Emacs user I use one of its stand-out features org-mode which comes with multi-language org-babel support. This allows me to document my benchmarking while scripting up the steps in a variety of "languages" depending on the my needs at the time. Let's take a look at the first section:

1 Binaries To Test

Here we have several tables of binaries to test. We refer to the

current benchmarking set from the next stage, Run Benchmark.

For a final test we might compare the system QEMU with a reference

build as well as our current build.

Binary title

/usr/bin/qemu-aarch64 system-2.5.log

~/lsrc/qemu/qemu-builddirs/arm-targets.build/aarch64-linux-user/qemu-aarch64 master.log

~/lsrc/qemu/qemu.git/aarch64-linux-user/qemu-aarch64 softfloat-v4.log

Binary	title
/usr/bin/qemu-aarch64	system-2.5.log
~/lsrc/qemu/qemu-builddirs/arm-targets.build/aarch64-linux-user/qemu-aarch64	master.log
~/lsrc/qemu/qemu.git/aarch64-linux-user/qemu-aarch64	softfloat-v4.log

Well that is certainly fairly explanatory. These are named org-mode tables which can be referred to in other code snippets and passed in as variables. So the next job is to run the benchmark itself:

2 Run Benchmark

This runs the benchmark against each binary we have selected above.

import subprocess
import os

runs=[]

for qemu,logname in files:
cmd="taskset -c 0 %s ./vector-benchmark -n %s | tee %s" % (qemu, tests, logname)
    subprocess.call(cmd, shell=True)
    runs.append(logname)

    return runs

So why use python as the test runner? Well truth is whenever I end up munging arrays in shell script I forget the syntax and end up jumping through all sorts of hoops. Easier just to have some simple python. I use python again later to read the data back into an org-table so I can pass it to the next step, graphing:

set title "Vector Benchmark Results (lower is better)"
set style data histograms
set style fill solid 1.0 border lt -1

set xtics rotate by 90 right
set yrange [:]
set xlabel noenhanced
set ylabel "nsecs/Kop" noenhanced
set xtics noenhanced
set ytics noenhanced
set boxwidth 1
set xtics format ""
set xtics scale 0
set grid ytics
set term pngcairo size 1200,500

plot for [i=2:5] data using i:xtic(1) title columnhead

This is a GNU Plot script which takes the data and plots an image from it. org-mode takes care of the details of marshalling the table data into GNU Plot so all this script is really concerned with is setting styles and titles. The language is capable of some fairly advanced stuff but I could always pre-process the data with something else if I needed to.

Finally I need to upload my graph to an image hosting service to share with my colleges. This can be done with a elaborate curl command but I have another trick at my disposal thanks to the excellent restclient-mode. This mode is actually designed for interactive debugging of REST APIs but it is also easily to use from an org-mode source block. So the whole thing looks like a HTTP session:

:client_id = feedbeef

# Upload images to imgur
POST https://api.imgur.com/3/image
Authorization: Client-ID :client_id
Content-type: image/png

< benchmark.png

Finally because the above dumps all the headers when run (which is very handy for debugging) I actually only want the URL in most cases. I can do this simply enough in elisp:

#+name: post-to-imgur
#+begin_src emacs-lisp :var json-string=upload-to-imgur()
  (when (string-match
         (rx "link" (one-or-more (any "\":" whitespace))
             (group (one-or-more (not (any "\"")))))
         json-string)
    (match-string 1 json-string))
#+end_src

The :var line calls the restclient-mode function automatically and passes it the result which it can then extract the final URL from.

And there you have it, my entire benchmarking workflow document in a single file which I can read through tweaking each step as I go. This isn't the first time I've done this sort of thing. As I use org-mode extensively as a logbook to keep track of my upstream work I've slowly grown a series of scripts for common tasks. For example every patch series and pull request I post is done via org. I keep the whole thing in a git repository so each time I finish a sequence I can commit the results into the repository as a permanent record of what steps I ran.

If you want even more inspiration I suggest you look at John Kitchen's scimax work. As a publishing scientist he makes extensive use of org-mode when writing his papers. He is able to include the main prose with the code to plot the graphs and tables in a single source document from which his camera ready documents are generated. Should he ever need to reproduce any work his exact steps are all there in the source document. Yet another example of why org-mode is awesome ;-)

Getting organised

2012-01-13T10:38:00+00:00

What with becoming a parent and getting promoted I suddenly find myself needing to become a lot more organised. Although I've been using org-mode for a bit I need to get a lot more organised with it.

Previously I had two sets of org notes. My personal set where sitting on my server which I could access via the terminal. I generally accessed this at home on the odd occasion when I was doing things like the annual round of insurance quote gathering. The second set was a fairly simple time sheet type affair that I was using at work to keep a vague track of where all my time was spent. The big missing part of this is when I'm on the move.

I've just recently upgraded my phone to the latest Galaxy Nexus which is a fine Google enabled device. I make no apologies for using Google's calendering and shared document services. They work very well and importantly allow me to share things with my wife who doesn't quite share my desire to run everything from a text editor. However for my personal task lists on the move and remembering what's coming up at work it doesn't quite cut it. Besides I like org-mode and I'd heard about MobileOrg so I endeavoured to set it up.

MobileOrg has been around some time for the iPhone but the mechanisms it uses for integrating with org-mode are fairly well documented. As a result there is a couple of Android implementations for it. Matthew Jone's mobileorg-android was the first version I tried.

The original sync method for MobileOrg was to use a service like Dropbox to sync files. Given the history of Dropbox's security I wasn't about to move my files into the proprietary cloud. The alternative is to enable WebDAV on my web-server and therefor enable two way communication via HTTP. It was a little concerning to see self-signed SSL wasn't supported as this does open up a potential attack vector on my machine. I've mitigated it a little by using digest authentication instead of basic-auth but I'd still prefer to be conducting these read-write operations over something more secure.

Initial results were a little underwhelming. After some messing around with the format of org-links I eventually got a basic outline summary up. Unfortunately I can't seem to sync notes created on my phone to the server. This seems to be a Apache problem which I shall have to dig into later.

After perusing the market some more I noticed there is a new project in town. Konstantin's MobileOrgNG was forked some time ago from Matthew's code and on installing I found it looked an awful lot better. I've still be unable to post any locally added notes (due to previously mentioned Apache config issues). However it's presentation is a lot slicker and it shows a lot of potential for being a good MobileOrg client.

I'm now stuck with a classic open source fork dilemma. The code bases look to have diverged enough that these two projects are essential going their own way. Looking at the two impact graphs it looks like they diverged around August 2011 and since then MobileOrgNG looks pretty much like a solo effort albeit with an impressive commit rate of new features.

So the questions for my readers. Which code base should I jump on? Has anyone got experience with the two different code bases and the reason they split? Are there any other Android clients for org-mode I should be looking at?

org-mode and clocking in

2011-09-20T17:06:00+01:00

I've recently started using org-mode's time tracking to keep track of what I spend my time doing at work. This was in response to being asked by one of my managers what I spend my time doing and basically being forced to guess.

Setting up a clock page is fairly simple. It's then just a case of C-c C-x C-i and C-c C-x C-o on the appropriate sub tasks. Dynamic blocks can then be added to your org-document to generate weekly, monthly or annual reports based on the clock lines in the document.

So far the results have been illuminating and certainly shows how optimistic I can be in predicting how much time I spend doing my core job hacking on the code. However one thing that is captured but hard to summarise is interruption cost. I've taken to switching task every-time I'm interrupted in person or by phone call (I'm not counting IM/IRC as it's less disruptive). I can eyeball the raw data and see that some weeks are exceptionally bad for task switching. However what would useful is a break-down of mean and median clock lengths against each task to give some sort of indication of how much straight line hacking I've gotten done. I have a feeling the :formula and :formatter options could be used for this but I've been struggling to find any example. Does anyone do a similar analysis with their org-mode clock data?

UPDATED: fixed keystrokes.