Running Linux in QEMU's aarch64 system emulation mode

2014-05-09T13:14:00+01:00

Since I started working on aarch64 support for QEMU the most frequently asked question I got was "when can I run aarch64 system emulation on QEMU?". Well wait no more as support for a VIRT-IO based aarch64 board was recently merged into the master branch of QEMU. In this post I'll talk about building QEMU, a rootfs and a kernel that will allow you to start experimenting with the architecture.

Quick start

Let's first start with building and running QEMU with some pre-built images.

Build Dependancies

As has been noted in the comments the configure script will automatically enable features as long as the pre-requisite developer libraries are installed on your sytem. With a Debian/Ubuntu system this is easily achieved by running:

sudo apt-get build-dep qemu

Of course if you want to enable a feature (either a bleeding edge or non-standard) that requires additional libraries then you will need to install the appropriate development packages manually. The config.log file is usually a useful first step in working out what headers are being looked for.

Building QEMU

git clone git://git.qemu.org/qemu.git qemu.git
cd qemu.git
./configure --target-list=aarch64-softmmu
make

Assuming the build ran without any problems you should now have an executable ./aarch64-softmmu/qemu-system-aarch64 in your build directory. Grab a pre-built image from here and we'll check it works. The image is a kernel that has been combined with an initial RAM disk (initrd) with a basic root file-system. I go into more details on how to create this later on.

Be aware the command line is quite long so make sure you copy it all ;-)

wget http://people.linaro.org/~alex.bennee/images/aarch64-linux-3.15rc2-buildroot.img
./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img  --append "console=ttyAMA0"

If all went well you should see the familiar Linux boot sequence and eventually get a login prompt. Login as root (no password) and play in the new sandbox.

... usual kernel boot output ...
Welcome to Buildroot
buildroot login: root
# uname -a
Linux buildroot 3.15.0-rc2ajb-00069-g1aae31c #39 SMP Thu Apr 24 11:48:57 BST 2014 aarch64 GNU/Linux

Once you are done type C-a c to enter QEMU's monitor mode and then quit to exit.

QEMU 2.0.50 monitor - type 'help' for more information
(qemu) quit

Accessing your local file-system

This is all very well but the test image only has a fairly limited root file-system attached to it. It will be a lot more useful if you could access your host file-system to test other binaries. Thanks to VIRT FS we can achieve this without too much hassle. Use the following extended QEMU command line:

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0" -fsdev local,id=r,path=/home/alex/lsrc/qemu/rootfs/trusty-core,security_model=none -device virtio-9p-device,fsdev=r,mount_tag=r

This sets up the selected path to be mountable by the guest. In this case I'm using an Ubuntu rootfs which can be downloaded from here. Once the system has booted the following commands on the guest will mount the local file-system:

Welcome to Buildroot
buildroot login: root
# mount -t 9p -o trans=virtio r /mnt
# ls -l /mnt/
total 84
drwxr-xr-x    2 default  default       4096 Apr  2  2014 bin
drwxr-xr-x    2 default  default       4096 Feb 27  2014 boot
drwxr-xr-x    3 default  default       4096 Apr  2  2014 dev
drwxr-xr-x   64 default  default       4096 Apr  3  2014 etc
drwxr-xr-x    2 default  default       4096 Feb 27  2014 home
..

Building your own rootfs

There are many solutions to this (including downloading Linaro engineering builds) but the simplest one I've found for rolling your own from scratch is the Buildroot project. It present the familiar kernel menuconfig interface and deals with all the hassle of setting up cross compilers for you.

git clone git://git.buildroot.net/buildroot buildroot.git
cd buildroot.git
make menuconfig

There are lots of configuration options to choose from but the following are what I use:

* Target Options -> Target Architecture(AArch64)
* Toolchain -> Toolchain type (External toolchain)
* Toolchain -> Toolchain (Linaro AArch64 14.02)
* System configuration -> Run a getty (login prompt) after boot (BR2_TARGET_GENERIC_GETTY)
* System configuration -> getty options -> TTY Port (ttyAMA0) (BR2_TARGET_GENERIC_GETTY_PORT)
* Target Packages -> Show packages that are also provided by busybox (BR2_PACKAGE_BUSYBOX_SHOW_OTHERS)
* Filesystem images -> cpio the root filesystem (for use as an initial RAM filesystem) (BR2_TARGET_ROOTFS_CPIO)

The last one will be important for when we build the kernel next. Once you have configured buildroot to your liking it's time to type make and leave it for a while as you enjoy a nice lunch ;-)

make
.. lots of output ..

Building a kernel

For building the kernel I use my distro's aarch64 cross-compiler. On Debian/Ubuntu systems this is easily added with:

$ sudo apt-get install gcc-aarch64-linux-gnu

And the usual kernel building process, with a few tweaks for cross compiling:

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux.git
cd linux.git
ARCH=arm64 make menuconfig

I've put my full config up here but important options to note are:

CONFIG_CROSS_COMPILE="aarch64-linux-gnu-"                                               # needs to match your cross-compiler prefix
CONFIG_INITRAMFS_SOURCE="/home/alex/lsrc/qemu/buildroot.git/output/images/rootfs.cpio"  # points at your buildroot image
CONFIG_NET_9P=y                                                                         # needed for virtfs mount
CONFIG_NET_9P_VIRTIO=y

Finally you build it all with:

ARCH=arm64 make -j 8

The -j 8 just specifies how many parallel build threads to use. Generally set it to the number of cores you have on your machine.

Final test

All that remains is to test that the newly built kernel works:

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel ../linux.git/arch/arm64/boot/Image  --append "console=ttyAMA0"
... lots more output ...
Welcome to Buildroot
ajbtest login: root
[root@ajbtest ~]# ls -l
total 0
[root@ajbtest ~]# uname -a
Linux ajbtest 3.15.0-rc4ajb-00320-gafcf0a2-dirty #41 SMP Fri May 9 13:05:31 BST 2014 aarch64 GNU/Linux

UPDATED: 27/05/2014
* Added notes about library dependencies
* Cleaned up formatting of shell sections, mention length of command line!
* Fix some spelling errors

Alex's Adventures on the Infobahn - aarch64