Running Linux in QEMU’s aarch64 system emulation mode

Since I started working on aarch64 support for QEMU the most frequently asked question I got was “when can I run aarch64 system emulation on QEMU?”. Well wait no more as support for a VIRT-IO based aarch64 board was recently merged into the master branch of QEMU. In this post I’ll talk about building QEMU, a rootfs and a kernel that will allow you to start experimenting with the architecture.

Quick start

Let’s first start with building and running QEMU with some pre-built images.

Build Dependancies

As has been noted in the comments the configure script will automatically enable features as long as the pre-requisite developer libraries are installed on your sytem. With a Debian/Ubuntu system this is easily achieved by running:

sudo apt-get build-dep qemu

Of course if you want to enable a feature (either a bleeding edge or non-standard) that requires additional libraries then you will need to install the appropriate development packages manually. The config.log file is usually a useful first step in working out what headers are being looked for.

Building QEMU

git clone git://git.qemu.org/qemu.git qemu.git
cd qemu.git
./configure --target-list=aarch64-softmmu
make

Assuming the build ran without any problems you should now have an executable ./aarch64-softmmu/qemu-system-aarch64 in your build directory. Grab a pre-built image from here and we’ll check it works. The image is a kernel that has been combined with an initial RAM disk (initrd) with a basic root file-system. I go into more details on how to create this later on.

Be aware the command line is quite long so make sure you copy it all ;-)

wget http://people.linaro.org/~alex.bennee/images/aarch64-linux-3.15rc2-buildroot.img
./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img  --append "console=ttyAMA0"

If all went well you should see the familiar Linux boot sequence and eventually get a login prompt. Login as root (no password) and play in the new sandbox.

... usual kernel boot output ...
Welcome to Buildroot
buildroot login: root
# uname -a
Linux buildroot 3.15.0-rc2ajb-00069-g1aae31c #39 SMP Thu Apr 24 11:48:57 BST 2014 aarch64 GNU/Linux

Once you are done type C-a c to enter QEMU’s monitor mode and then quit to exit.

QEMU 2.0.50 monitor - type 'help' for more information
(qemu) quit

Accessing your local file-system

This is all very well but the test image only has a fairly limited root file-system attached to it. It will be a lot more useful if you could access your host file-system to test other binaries. Thanks to VIRT FS we can achieve this without too much hassle. Use the following extended QEMU command line:

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0" -fsdev local,id=r,path=/home/alex/lsrc/qemu/rootfs/trusty-core,security_model=none -device virtio-9p-device,fsdev=r,mount_tag=r

This sets up the selected path to be mountable by the guest. In this case I’m using an Ubuntu rootfs which can be downloaded from here. Once the system has booted the following commands on the guest will mount the local file-system:

Welcome to Buildroot
buildroot login: root
# mount -t 9p -o trans=virtio r /mnt
# ls -l /mnt/
total 84
drwxr-xr-x    2 default  default       4096 Apr  2  2014 bin
drwxr-xr-x    2 default  default       4096 Feb 27  2014 boot
drwxr-xr-x    3 default  default       4096 Apr  2  2014 dev
drwxr-xr-x   64 default  default       4096 Apr  3  2014 etc
drwxr-xr-x    2 default  default       4096 Feb 27  2014 home
..

Building your own rootfs

There are many solutions to this (including downloading Linaro engineering builds) but the simplest one I’ve found for rolling your own from scratch is the Buildroot project. It present the familiar kernel menuconfig interface and deals with all the hassle of setting up cross compilers for you.

git clone git://git.buildroot.net/buildroot buildroot.git
cd buildroot.git
make menuconfig

There are lots of configuration options to choose from but the following are what I use:

* Target Options -> Target Architecture(AArch64)
* Toolchain -> Toolchain type (External toolchain)
* Toolchain -> Toolchain (Linaro AArch64 14.02)
* System configuration -> Run a getty (login prompt) after boot (BR2_TARGET_GENERIC_GETTY)
* System configuration -> getty options -> TTY Port (ttyAMA0) (BR2_TARGET_GENERIC_GETTY_PORT)
* Target Packages -> Show packages that are also provided by busybox (BR2_PACKAGE_BUSYBOX_SHOW_OTHERS)
* Filesystem images -> cpio the root filesystem (for use as an initial RAM filesystem) (BR2_TARGET_ROOTFS_CPIO)

The last one will be important for when we build the kernel next. Once you have configured buildroot to your liking it’s time to type make and leave it for a while as you enjoy a nice lunch ;-)

make
.. lots of output ..

Building a kernel

For building the kernel I use my distro’s aarch64 cross-compiler. On Debian/Ubuntu systems this is easily added with:

$ sudo apt-get install gcc-aarch64-linux-gnu

And the usual kernel building process, with a few tweaks for cross compiling:

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux.git
cd linux.git
ARCH=arm64 make menuconfig

I’ve put my full config up here but important options to note are:

CONFIG_CROSS_COMPILE="aarch64-linux-gnu-"                                               # needs to match your cross-compiler prefix
CONFIG_INITRAMFS_SOURCE="/home/alex/lsrc/qemu/buildroot.git/output/images/rootfs.cpio"  # points at your buildroot image
CONFIG_NET_9P=y                                                                         # needed for virtfs mount
CONFIG_NET_9P_VIRTIO=y

Finally you build it all with:

ARCH=arm64 make -j 8

The -j 8 just specifies how many parallel build threads to use. Generally set it to the number of cores you have on your machine.

Final test

All that remains is to test that the newly built kernel works:

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel ../linux.git/arch/arm64/boot/Image  --append "console=ttyAMA0"
... lots more output ...
Welcome to Buildroot
ajbtest login: root
[root@ajbtest ~]# ls -l
total 0
[root@ajbtest ~]# uname -a
Linux ajbtest 3.15.0-rc4ajb-00320-gafcf0a2-dirty #41 SMP Fri May 9 13:05:31 BST 2014 aarch64 GNU/Linux

UPDATED: 27/05/2014
* Added notes about library dependencies
* Cleaned up formatting of shell sections, mention length of command line!
* Fix some spelling errors

20 Comments

  1. Note that VIRTFS and VIRTIO features need to be enabled in the configuration for building QEMU to get the option “-fsdev local,id=r,path=/home/alex/lsrc/qemu/rootfs/trusty-core,security_model=none -device virtio-9p-device,fsdev=r,mount_tag=r” to work.

    • I did a fresh re-build when writing this so I assume it’s enabled automatically when the pre-reqs are around. I shall check next week when I’m back in the office.

      Thanks for the feedback :-)

  2. Thanks for your guide Alex.

    I also did not find the virtio-9p-device firstly.
    In my host, I found missing the “libcap-devel” and “libattr-devel”.
    After I installed the two libraries and re-build, I found that the VIRTFS was enabled automatically:)

    • I’ve updated the post to mention library dependencies. I assume your system is RPM based? Is there a similar build-dep type command that can pull in the extra development libraries for those systems?

  3. Alex, Thanks for the article.

    I am using Ubuntu 12.04 LTS. I was able to successfully run Linux on Qemu following the instructions above and using the pre-built image provided by you.
    However, on building the rootfs on my own as instructed above, Qemu is not showing any output and no buildroot login when invoked with the newly built “../linux.git/arch/arm64/boot/Image” on my system.

    Since the command “sudo apt-get install gcc-aarch64-linux-gnu” could not fetch toolchain on ubuntu 12.04, I have used the toolchain present in “buildroot.git/output/host/opt/ext-toolchain/bin” for compiling the kernel. I configured the kernel as instructed above.

    Any pointers on resolving the problem is very much appreciated.

    Thanks & Best Regards,
    Raghu.

    • Problem Solved. Configuration file provided in the website used instead of changing options in menuconfig. rootfs.cpio path alone is modified in config file.

      However, AFAIK, the major differences between the config files(menuconfig changes ->.config & aarch64-kernel.config) are
      CONFIG_NET_9P=m, CONFIG_NET_9P_VIRTIO=m, CONFIG_9P_FS=m instead of
      CONFIG_NET_9P=y, CONFIG_NET_9P_VIRTIO=y, CONFIG_9P_FS=y.

      Thanks & Best Regards,
      Raghu.

      • Ah yes modules add an extra level of confusion as you need to ensure the initramfs has them and they get loaded as required. They make a lot of sense for generic binary distributions that need to boot on ${RANDOMHW} but for personal kernels and well-known requirements for VMs it saves a lot of pain just to build them directly in.

  4. Hi Alex,

    Thanks for the great post. Was easy to follow!

    I tried to run multiple guest cpus using: -smp 2
    Result:
    qemu-system-aarch64: mach-virt: must enable KVM to use multiple CPUs

    Then tried to enable KVM:
    ./configure –enable-kvm –target-list=aarch64-softmmu
    make
    ./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt,accel=kvm -nographic -smp 2 -m 2048 -kernel ../linux.git/arch/arm64/boot/Image –append “console=ttyAMA0″ -fsdev local,id=r,path=/home/ori/lsrc/qemu/rootfs/trusty-core,security_model=none
    Result:
    KVM not supported for this target
    No accelerator found!

    What is the reason?

    Thanks!

    Ori

    • KVM is only for running on aarch64 hardware where QEMU doesn’t do any translation but just controls the virtual machine and deals with hardware emulation. In that case you’ll want to change your command line include “-enable-kvm -cpu host”.

      As for running SMP on a TCG emulated system I can look into it but there’s not currently much point as internally QEMU will schedule one emulated core after another in a single thread.

  5. Hi Alex,
    Thank you for this write-up, it’s helped me get quite far using aarch64 with qemu. However, this is one area where I’m still stuck. I’m unable to get any kind of networking working in the Linux guest OS. the problem I have is all network devices I try cannot find a PCI bus:
    $ $HOME/git/qemu/aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0" -netdev type=user,id=net0 -device virtio-net-pci,netdev=net0
    qemu-system-aarch64: -device virtio-net-pci,netdev=net0: No 'PCI' bus found for device 'virtio-net-pci'

    Is there something special I need to do to get the networking subsystem to see the qemu internal PCI bus for aarch64?

    Thanks again!

    • The thing to remember about ARM systems is they are typically System on a Chip (SOC) setups and don’t always have the standard bus architectures of the PC space. In our case the virt modeled system doesn’t need a PCI bus as it would just be an additional abstraction. However a PCI device is discoverable which is useful for the kernel to find out what hardware it has at runtime, in the case of ARM SOCs the location of the virt device are passed to the guest kernel via a device tree blob.

      You can describe your network hardware with the following:

        -netdev user,id=unet -device virtio-net-device,netdev=unet
      
      • Alex, thank you! I missed that entry when I ran:

        $ ./aarch64-softmmu/qemu-system-aarch64 -machine type=virt -device \?

        I also find it a bit odd that “virtio-net” is aliased to “virtio-net-pci” and not “virtio-net-device”. As of now my networking is working perfectly for apt-get updates in a virtualized environment. :)

        • Hi Brian,

          Thanks for your write-up. I followed your steps, while my networking is not work. Within QEMU, there’s only a local network at first. I use DHCP to get an IP address: /sbin/udhcpc eth0. While I can neither reach my host nor connect out from QEMU.

          Is there something special I need to do? Do I need to create a tap?

          Thanks & Best Regards,

          Emma

        • Hi Brian,

          I am very new to this, I have couple of issues please help me out to fix the things.
          I have some issue with the network configuration, I am using the same steps mentioned above still the interfaces on virtualized environments are not visible. Just the “lo” loopback is visible.
          And also I built a root file system but it does not contain many commands and entries can you please help me out in this also what I need to set etc.

          Regards,
          Pradeep

  6. Hi Alex,

    Thanks for writing this. It helps me a lot.

    I tried to copy a program from host to guest by mounting the local file-system. And the program is lost after I reboot the guest. I don’t know why. Is there a way to keep the things which is copied to the guest?

    I find out that there’s no compile tools in the linux kernel downloaded from https://git.kernel.org. I tried to install JDK and failed. It seems like missing bash instructions.

    Thanks & Best Regards,
    Emma

    • Yes, you need to have a more permanent file-system. The contents of the initrd RAM disk are not persistent and will get reset to what was bundled in the kernel on each boot. Either run your programs from the 9p mounted file-system or mount a virtio block device for your persistant data.

      As to compilers I’m not sure why you mention the JDK. However Debian/Ubuntu does package a number of cross compilers. Failing that you can download the latest GCC from the Linaro website and add it to your path.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>