Over the years, we have performed some experiments at Kumina to run the FreeBSD operating system inside KVM. The main problem with FreeBSD was the lack of proper virtio drivers. Whereas a Linux virtual machine is capable of using low-overhead device drivers to communicate with its host operating system, FreeBSD simply used its Intel ATA and Realtek NIC drivers, which gives suboptimal performance. As of recently FreeBSD gained support for virtio, renewing our interest in testing FreeBSD. What makes our setup a bit more complicated, is that we want to be able to boot a FreeBSD recovery image using TFTP exclusively.
When using FreeBSD’s pxeboot inside KVM we observed two issues:
- RTC handling inside the bootloader doesn’t work properly, causing spurious network timeouts and poor console interactivity, and
- The TFTP code doesn’t support prepending a pathname, meaning all files have to be stored inside the root directory of the TFTP server.
Both these issues have been fixed in SVN revisions 228782 and 228798, respectively. Currently this means that we can only use FreeBSD 10.0-CURRENT, but it is expected that these changes, including the virtio drivers, are merged to 9.1 as well.
To create a FreeBSD recovery system that can be booted using TFTP, we tried using MFSBSD. Though MFSBSD works pretty well, we were slightly disappointed with the compression it offered. Even after stripping an excessive number of utilities, it was still hard to create an image that fits within the 45 MB limit imposed by the kernel. This is due to the fact that MFSBSD only allows the system to use geom_uzip or only compress /usr using a high-performance compression algorithm such as xz.
Therefore we have written our own MFSBSD-like system that manages to compress an almost complete FreeBSD system to less than 40 MB. Compared to a regular FreeBSD system it only lacks the following:
- The toolchain (GCC, Binutils, GDB, Clang) and its static libraries and header files,
- BSD games (fortune, rot13, etc),
- 32-bits libraries,
- The /rescue environment, and
- Manual pages for categories 2, 3 and 9.
To achieve a good compression ratio, it stores the entire system in an xz compressed tarball. This tarball is stored together with a handcrafted replacement of /sbin/init on a memory file system. During boot, this statically linked /sbin/init mounts a tmpfs, extracts the tarball using libarchive, chroots into the tmpfs and continues the boot process by executing the actual /sbin/init. This only seems to delay the boot procedure by approximately 2 seconds.
As usual, we have decided to release our modifications to the Open Source community. The following patchset can be applied to the FreeBSD source tree. It introduces a shell script called tfimage.sh that automatically creates a recovery system. The resulting system is placed in the output/ directory. Have fun!