Architest -- eBPF testing on different Linux kernel architectures using Buildroot VMs
Welcome to the second chapter of the mini-series on testing the Pulsar eBPF code!
In the previous instalment, we shared the journey that led to our comprehensive test-suite. In this chapter, we tackle the challenging task of running that suite across a diverse range of kernel versions, architectures, and configurations. This is a critical step to ensure the reliability and effectiveness of our eBPF code, and I'm excited to share our insights with you.
The Problem
The primary focus of Pulsar is on the Internet of Things (IoT) domain, which exhibits significant fragmentation in terms of underlying platforms. There are diverse architectures such as Arm and RISC-V, as well as potentially outdated Linux kernels. This is a problem because the kernel does not guarantee backward API compatibility for internal data structures or kprobes, and eBPF programs may use features that are unavailable on older kernels.
Consequently, having a test environment for validating all the supported platforms and versions is of utmost importance to us.
First approach: Vagrant
The initial approach to address the issue was to use Vagrant boxes, which were effective in reproducing user-reported problems. However, we soon discovered that they were inadequate for our needs. We were unable to modify the build-time configuration of the kernel or test uncommon architectures. Moreover, we were limited to pre-released kernel versions, which meant we couldn't test all versions.
This realization led us to understand that we needed complete control over the kernel build process to achieve our desired outcomes.
Buildroot to the rescue
Buildroot is an efficient tool that facilitates the creation of embedded Linux systems via cross-compilation. Once downloaded, the menuconfig interface you can see below will guide you in the system configuration.
When you're done, running make will download the source code of all the selected packages, cross-compile them for the target architecture, build the Linux kernel with the provided configuration, and pack the output in a Root File System image that can be flashed to the target embedded device, or booted in an emulator. Indeed, in our context, the ability to run the generated system using QEMU is fundamental.
While Buildroot was invaluable in making such a complex task manageable, developing a configuration for our use-case took quite some effort and studying. Being quite happy with our results, we've decided to share our work with the community.
Introducing Architest
The outcome of our work is Architest, a combination of pre-built virtual machines and a framework for creating new ones. We hope that it will prove useful to everyone who needs to test software on diverse architectures.
The easiest way to get started is to download and run one of the pre-compiled releases:
wget https://github.com/Exein-io/architest/releases/download/0.2/aarch64_6.0.tar.gz
tar -xzf aarch64_6.0.tar.gz
cd build/aarch64_6.0/images/
./start-qemu.sh
💡 After an Architest virtual machine has been started by the ./start-qemu.sh script, it's accessible via SSH at port 3366:
ssh root@localhost -p 3366
Our range of pre-built machines includes x86_64, aarch64, riscv, and mips, with kernel versions spanning from 5.5 to 6.0. In the event that the specific pre-built machine you require is unavailable, you have the option to build it yourself.
Refer to the README for detailed instructions, and don't hesitate to ask any questions via the issue tracker.
Using Architest with Pulsar
As a case study, let's take a closer look at how we use Architest at Exein for testing Pulsar.
To start the Architest virtual machine, we run the ./start-qemu.sh script which comes with it. Once the virtual machine is up and running, it can be accessed through SSH at port 3366. The virtual machine is now ready for running our comprehensive test-suite.
The test-suite is written in Rust and must be cross-compiled before it can be run on the target architecture.We recommend using musl toolchains for this purpose due to their ease of use. For instance, if you plan to run the tests on an aarch64 virtual machine, you can compile the code using the aarch64-unknown-linux-musl target.
Using the cross tool simplifies this process even further by conducting the cross-compilation inside a Docker image with all the necessary prerequisites pre-installed.
Once the compilation is complete, we strip the binary of its debug symbols to reduce its size. Although this step is optional, it enables QEMU to run with fewer resources allocated.
Next, we need to deploy the binary to the virtual machine using SSH and scp.
To minimize any potential issues, we copy the binary to the/tmp/ folder, as there is no need for it to persist across reboots.
Finally, we can establish a connection to the virtual machine and run the application.
For convenience, we have automated this process with a simple xtask:
cargo xtask cross run --target aarch64-unknown-linux-musl
$ cross build --target aarch64-unknown-linux-musl --workspace --bin test-suite
...
$ llvm-strip target/aarch64-unknown-linux-musl/debug/test-suite
$ scp target/aarch64-unknown-linux-musl/debug/test-suite qemu:/tmp/
test-suite 100% 8086KB 12.8MB/s 00:00
$ ssh qemu /tmp/test-suite
running 34 tests
...
test result: FAILED. 32 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 67.57s
By following the above process, we can efficiently validate all our eBPF code on the specified system configuration.
Our next objective is to enhance this process by enabling parallel execution across multiple virtual machines and integrating it into our CI system.
Conclusion
Buildroot-based virtual machines offer a significant advantage over Vagrant in terms of customization and flexibility, making them an ideal choice for running an eBPF test-suite. Not only can Architest be used by other eBPF-based projects in its current state, but it also has the potential to benefit any software that needs to undergo cross-architecture testing.
I hope this article has been informative and helpful in exploring the benefits of Buildroot-based virtual machines for testing eBPF code. If you have not already done so, I encourage you to check out the first part of this two-part series for a more comprehensive understanding of the topic.
Please consider leaving a star on the Architest and Pulsar Github repositories to show support for these projects and to help increase their visibility to other users. By doing so, you can contribute to the growth of the open-source community and help make testing software more accessible and effective.
Matteo Nardi - Rust System Developer