FreeBSD 15.0 on a Headless Linux Host with Cloud-Init and nftables

I needed a FreeBSD box to test jail orchestration, and my only server is a headless Linux machine I access over SSH. That meant working without local console access and relying on the FreeBSD cloud image, libvirt, and a serial console from the start.

This is a companion to the FreeBSD jail orchestration series - you need a working FreeBSD before you can build anything on it. If your host runs nftables with policy drop (and especially if Docker is also installed), you’ll want to read the firewall section before you start wondering why your VM boots into a network black hole.

Picking the Right Image

FreeBSD 15.0-RELEASE ships four qcow2 VM images. They look similar, but they are not interchangeable:

The non-CLOUDINIT images ship without a root password, SSH keys, a DHCP client, or a serial console. On a headless host, that leaves you with a running VM and no practical way to reach it.

If you need ZFS (and you do, if you’re planning to work with jails or bhyve), pick BASIC-CLOUDINIT-zfs. You cannot convert UFS to ZFS in-place after the fact.

Download from the FreeBSD VM images directory:

wget -O ~/Downloads/FreeBSD-15.0-RELEASE-amd64-BASIC-CLOUDINIT-zfs.qcow2.xz \
  "https://download.freebsd.org/releases/VM-IMAGES/15.0-RELEASE/amd64/Latest/FreeBSD-15.0-RELEASE-amd64-BASIC-CLOUDINIT-zfs.qcow2.xz"

626 MB compressed, 2.7 GB decompressed.

What You Need on the Linux Host

I’m running Manjaro (Arch-based), kernel 6.12. The setup is the standard libvirt/QEMU stack plus a few extras:

sudo pacman -S qemu-full libvirt virt-install dnsmasq edk2-ovmf cdrtools
sudo systemctl enable --now libvirtd

The easy package to miss is edk2-ovmf: FreeBSD cloud images are UEFI-only, so you need OVMF firmware. cdrtools provides mkisofs for building the cloud-init seed ISO. The rest is standard libvirt setup.

You also need an SSH key (ssh-keygen -t ed25519 if you don’t have one).

Cloud-Init with nuageinit

If you’ve used cloud-init on Ubuntu or RHEL, the FreeBSD version is narrower. FreeBSD has nuageinit, a native C implementation that reads a CD-ROM labeled cidata (the NoCloud datasource). It handles hostname, users, SSH keys, write_files, runcmd, and packages, but it does not cover the full Python cloud-init feature set.

Create two files in /tmp/cidata/:

meta-data:

instance-id: freebsd-oci
local-hostname: freebsd-oci

user-data:

#cloud-config
hostname: freebsd-oci
fqdn: freebsd-oci.local

ssh_pwauth: true

users:
  - name: freebsd
    shell: /bin/sh
    groups: wheel
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: false
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... your-key-here

packages:
  - sudo

network:
  ethernets:
    vtnet0:
      dhcp4: true

write_files:
  - path: /boot/loader.conf.d/serial.conf
    content: |
      boot_multicons="YES"
      boot_serial="YES"
      comconsole_speed="115200"
      console="comconsole,vidconsole"
  - path: /etc/rc.conf.d/sshd
    content: |
      sshd_enable="YES"

runcmd:
  - echo '-S115200 -Dh' > /boot.config
  - service sshd enable
  - service sshd start

A few notes on the user-data:

• Serial console goes in /boot/loader.conf.d/serial.conf, not the main loader.conf. Cleaner, and you won’t accidentally overwrite existing settings.
• Update: the network: block handles DHCP natively via nuageinit. I originally had sysrc ifconfig_vtnet0="DHCP" in runcmd because I didn’t think nuageinit supported the network: directive. I was wrong - nuageinit’s code (/usr/libexec/nuageinit, line 403) checks for dhcp4: true and writes the config. The man page even has an example. I should have read it more carefully. Thanks to the r/freebsd thread for pushing me to actually look.
• Put your SSH key on both the freebsd user and root. Belt and suspenders.

Build the ISO:

mkisofs -output /var/lib/libvirt/images/freebsd-cidata.iso \
  -volid cidata -joliet -rock \
  /tmp/cidata/user-data /tmp/cidata/meta-data

Prepare the Disk and Create the VM

# Decompress (keep the original)
xz -dk ~/Downloads/FreeBSD-15.0-RELEASE-amd64-BASIC-CLOUDINIT-zfs.qcow2.xz

# Copy to libvirt's image directory and resize
sudo cp ~/Downloads/FreeBSD-15.0-RELEASE-amd64-BASIC-CLOUDINIT-zfs.qcow2 \
  /var/lib/libvirt/images/freebsd-oci.qcow2
sudo qemu-img resize /var/lib/libvirt/images/freebsd-oci.qcow2 30G

The qcow2 is thin-provisioned: 30 GB virtual, 2.5 GB actual on disk.

Create the VM:

sudo virt-install \
  --name freebsd-oci \
  --memory 4096 \
  --vcpus 2 \
  --os-variant freebsd15.0 \
  --import \
  --disk path=/var/lib/libvirt/images/freebsd-oci.qcow2,format=qcow2 \
  --disk path=/var/lib/libvirt/images/freebsd-cidata.iso,device=cdrom \
  --network network=default \
  --graphics vnc,listen=127.0.0.1,port=5900 \
  --serial pty \
  --boot uefi \
  --noautoconsole

The flags that matter:

• --os-variant freebsd15.0: on up-to-date Arch, osinfo-db already includes FreeBSD 15.0. Many guides suggest freebsd14.0 as a fallback: check with osinfo-query os | grep freebsd before defaulting to that.
• --boot uefi: FreeBSD cloud images require UEFI. libvirt automatically uses OVMF from /usr/share/edk2/x64/.
• --graphics vnc,listen=127.0.0.1: VNC bound to localhost only. You can tunnel it with ssh -L 5900:127.0.0.1:5900 if you need visual access. SPICE does NOT work with FreeBSD on QEMU.
• --noautoconsole: critical for headless operation. Without this, virt-install tries to open an interactive console and hangs.

Firewall Issues

The VM booted, but there was no DHCP lease. virsh net-dhcp-leases default returned nothing for more than 3 minutes.

I could confirm that the VM was running (virsh qemu-monitor-command freebsd-oci "info status" --hmp), the network interface was attached, and dnsmasq was listening on virbr0. The missing part was DHCP traffic reaching the bridge.

Problem 1: nftables Blocking DHCP on the Bridge

My host has a strict nftables firewall:

chain input {
    type filter hook input priority filter; policy drop;
    ct state established,related accept
    iif lo accept
    ip saddr 192.168.1.0/24 accept
    # ... blocklists, GeoIP, etc.
}

The VM sends a DHCPDISCOVER from 0.0.0.0 on virbr0. That traffic does not match the loopback rule, and 0.0.0.0 is not in 192.168.1.0/24, so nftables drops it before dnsmasq sees the request.

The non-obvious part is that libvirt creates its own nftables table (ip libvirt_network) with the right bridge rules, but it cannot modify your custom inet filter table. Both tables are evaluated. If either one drops the packet, it is gone.

The fix is to accept bridge traffic on virbr0.

# Runtime (immediate)
sudo nft add rule inet filter input position 10 iif "virbr0" accept

# Persistent: add to /etc/nftables.conf, after "iif lo accept"
iif "virbr0" accept

After adding the rule and rebooting the VM: DHCP lease within 10 seconds.

Problem 2: No Internet from the VM

SSH worked and the VM had an IP, but ping 8.8.8.8 showed 100% packet loss. DNS resolution still worked because dnsmasq handled that locally, while routed traffic could not leave the host.

Two firewalls were blocking forward traffic:

nftables forward chain:

chain forward {
    type filter hook forward priority filter; policy drop;
    # zero rules
}

iptables-legacy (Docker):

Chain FORWARD (policy DROP)
    DOCKER-USER -> DOCKER-FORWARD -> DROP

Docker installs iptables-legacy rules alongside nftables. Each has a FORWARD chain, each defaults to DROP, and the packet has to survive both paths to be forwarded. It is an awkward setup because the two firewalls use different rule syntaxes and do not coordinate with each other.

Fix for nftables:

sudo nft add rule inet filter forward iif "virbr0" accept
sudo nft add rule inet filter forward oif "virbr0" ct state established,related accept
sudo nft add rule inet filter forward oif "virbr0" ip daddr 192.168.122.0/24 accept

Fix for iptables-legacy (Docker):

sudo iptables-legacy -I DOCKER-USER -i virbr0 -j ACCEPT
sudo iptables-legacy -I DOCKER-USER -o virbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

After both sets of rules were in place, the VM had full internet connectivity.

Making the Rules Persistent

The nftables rules go in /etc/nftables.conf. Add iif "virbr0" accept after the loopback rule in the input chain, and add the three forward rules inside the forward chain.

The iptables-legacy rules for Docker are trickier to persist: you need either a systemd service or to remove Docker entirely. If this host does not need Docker, removing it makes the firewall much simpler.

Post-Boot: sudo and the packages Directive

Update: my original version of this post claimed nuageinit doesn’t install packages. I was wrong. Reddit user EinalButtocks pointed out that the packages: directive works fine. I had a typo in my user-data and jumped to the wrong conclusion. The user-data example above now includes packages: [sudo], which installs sudo on first boot automatically.

If you’re working from an older version of this guide without the packages: directive, you can bootstrap manually:

ssh freebsd@192.168.122.149
su -l root -c 'pkg install -y sudo'
su -l root -c 'echo "%wheel ALL=(ALL:ALL) NOPASSWD: ALL" > /usr/local/etc/sudoers.d/wheel'

Either way, the sudo: directive in the user config only creates sudoers entries: FreeBSD base doesn’t ship sudo, so you need to install the binary separately via packages: or pkg.

Final State

FreeBSD 15.0-RELEASE amd64 (GENERIC)
Hostname:  freebsd-oci.local
IP:        192.168.122.149 (DHCP via libvirt NAT)
ZFS pool:  zroot, 28.5 GB, ONLINE, healthy
User:      freebsd (wheel, sudo NOPASSWD)
SSH:       key-based auth
Internet:  full connectivity
Serial:    configured (works after first reboot)

SSH in, and you’re on FreeBSD:

$ ssh freebsd@192.168.122.149
$ uname -a
FreeBSD freebsd-oci.local 15.0-RELEASE FreeBSD 15.0-RELEASE releng/15.0-n280995-7aedc8de6446 GENERIC amd64
$ zpool status -x
all pools are healthy

Watch Out

Seven things that will bite you if you don’t see them coming:

1. Wrong image, no way in. The non-CLOUDINIT images (without BASIC-CLOUDINIT in the name) have no root password, no SSH keys, no DHCP, no serial console. On a headless host, you’re locked out. Use the CLOUDINIT variant.

2. virsh needs sudo for network commands. On Arch/Manjaro, virsh net-list --all returns empty without sudo. The networks exist but aren’t visible to your user. Either use sudo consistently or set up polkit rules for the libvirt group.

3. UFS to ZFS is a one-way street. You cannot convert an existing UFS root to ZFS in-place. Choose the right image from the start.

4. Serial console needs a reboot. Cloud-init writes the serial config on first boot, but FreeBSD’s bootloader reads /boot/loader.conf at boot time: the settings only apply on the NEXT boot. Your first boot has no serial output. Access via SSH or VNC tunnel.

5. nftables policy drop blocks libvirt DHCP. libvirt creates its own nftables table, but your custom inet filter table is evaluated independently. If your input chain drops traffic from virbr0, dnsmasq never sees the VM’s DHCP requests. Add iif "virbr0" accept to your input chain.

6. Docker and libvirt: double firewall. Docker uses iptables-legacy with FORWARD policy DROP. nftables also has a forward chain. Your VM traffic must survive both. This is the most confusing part: a packet traverses nftables, then iptables-legacy, and if either drops it, it’s gone. Explicitly allow virbr0 in both.

7. FreeBSD base doesn’t include sudo. The cloud-init sudo: directive only creates sudoers entries, it doesn’t install the binary. Add packages: [sudo] to your user-data (nuageinit supports it), or install manually via su -l root -c 'pkg install -y sudo'. Thanks to EinalButtocks on Reddit for the correction on packages: support.

What’s Next

This VM is the lab for the FreeBSD jail orchestration series. The base FreeBSD 15.0 install is in place, with ZFS and working network access. The next step is testing jail management tools on top of it.

Sources and references:

• FreeBSD 15.0-RELEASE VM images
• nuageinit(7) man page
• NoCloud datasource - cloud-init docs
• Arch Wiki: libvirt
• nftables wiki
• Docker and iptables

Antenore Gatta

A proud and busy Hacker, Father and Kyndrol

Post comment