QEMU Cortex-A7 & Microsoft SEAL: Debugging & Resolutions

1. Introduction

During the implementation of the Docker + QEMU + Buildroot + Microsoft SEAL pipeline, several technical challenges were encountered. Each issue provided useful insights into cross-compilation, embedded Linux systems, and emulation environments. This post details those challenges and their resolutions.

2. Difficulties Faced and Resolutions

2.1 Incomplete Cross-Compilation Toolchain

Issue: During SEAL compilation, CMake reported:

The CMAKE_CXX_COMPILER: arm-linux-gnueabihf-g++ was not found in the PATH.

Root Cause: The Docker image initially included only the ARM C compiler (gcc-arm-linux-gnueabihf) but not the C++ compiler. Microsoft SEAL is a C++ library and requires g++.

Resolution: The Docker image was updated to include: g++-arm-linux-gnueabihf

After rebuilding the Docker image, CMake successfully detected the ARM C++ compiler.

2.2 CMake Toolchain Detection Failure

Issue: CMake failed to detect the ARM C++ compiler correctly, even though it was installed.

Root Cause: The toolchain file referenced the compiler without an absolute path. CMake sometimes fails to resolve toolchain paths if not explicitly defined.

Resolution: The toolchain file was modified to use absolute paths:

set(CMAKE_C_COMPILER   /usr/bin/arm-linux-gnueabihf-gcc)
set(CMAKE_CXX_COMPILER /usr/bin/arm-linux-gnueabihf-g++)

This resolved the detection issue.

2.3 make menuconfig Not Available

Issue: Running make menuconfig produced:

No rule to make target ‘menuconfig’

Root Cause: Only the Buildroot output artifacts (zImage, rootfs.cpio) had been copied from another machine. The full Buildroot source tree was not installed on the server.

Resolution: The full Buildroot repository was cloned on the server:

git clone https://github.com/buildroot/buildroot.git
git checkout 2023.11

After that, menuconfig and full rebuild capability became available.

2.4 Root Filesystem Not Seeing Host Files

Issue: After booting Linux ARM in QEMU, the sealexamples binary was not visible.

Root Cause: The Linux system generated by Buildroot runs entirely from its root filesystem image. It has no visibility into the Docker volume or host directories.

Resolution: A Buildroot root filesystem overlay was created: buildroot/rootfs_overlay/usr/bin/sealexamples

After rebuilding Buildroot, the binary was correctly embedded in the root filesystem.

2.5 QEMU CPU / Machine Mismatch

Issue: QEMU produced the following error:

Cortex-A9MPCore peripheral can only use Cortex-A9 CPU

Root Cause: The vexpress-a9 machine type was used together with -cpu cortex-a7. The vexpress-a9 machine only supports Cortex-A9 CPUs.

Resolution: The QEMU machine was changed to: -M virt

The virt machine supports Cortex-A7 and is the recommended modern generic ARM platform.

2.6 Root Filesystem Format Change (cpio → ext2)

Issue: The initial setup used rootfs.cpio (initramfs), but after installing full Buildroot, the generated filesystem was rootfs.ext2. The previous QEMU boot command no longer worked.

Root Cause: Buildroot configuration changed the root filesystem type to ext2 (disk image).

Resolution: The QEMU boot configuration was updated to use a virtio block device:

-drive file=rootfs.ext2,if=none,format=raw,id=hd0 \
-device virtio-blk-device,drive=hd0 \
-append "root=/dev/vda rw"

This resolved the boot issue.

2.7 Compilation Interrupted

Issue: The Buildroot compilation process was interrupted manually.

Impact: Concern that the entire build process might need to restart.

Resolution: Buildroot supports incremental builds. Simply re-running:

make -j$(nproc)

allowed the build process to resume from the last completed stage. No clean rebuild was required.

2.8 Ensuring C Library Compatibility (glibc vs musl)

Issue: Choice of C library in Buildroot needed to match Microsoft SEAL.

Risk: Using musl or uClibc could lead to C++ runtime incompatibilities.

Resolution: glibc was selected in Buildroot configuration to ensure:

• Full C++ standard library support
• Exception handling compatibility
• Threading compatibility
• Consistency with cross-compilation toolchain