Bastille on FreeBSD 15.0: VNET Jails, Bastillefiles, and PF

I used Bastille here for two VNET jails on the same bridge. Creating the jails was easy enough, and the ZFS side felt mature. The rough part started once they had to talk to each other, because from that point on the host-side networking was manual.

The comparison post left me with a single jail running nginx. Here I used two VNET jails on a bridge, nginx reverse-proxying to a backend, and Bastillefiles to set them up.

Setup and first gotcha

Same VM from the headless guide with Bastille added to cloud-init. After boot, bastille setup enables ZFS, creates a loopback interface, and drops a PF skeleton in /etc/pf.conf. Then you bootstrap a release:

$ sudo bastille bootstrap 15.0-RELEASE update
Fetching distfile: base.txz
/usr/local/bastille/cache/15.0-RELEASE/base.txz  157 MB  15 MBps  10s

Bootstrap successful.

157 MB later, the base system was ready.

Now: I wanted VNET jails on their own subnet, which means a bridge. Bastille doesn’t create one. The setup command only creates a loopback for shared-IP jails. For VNET, you do it yourself:

$ sudo ifconfig bridge create
bridge0

$ sudo ifconfig bridge0 inet 10.0.0.254/24
$ sudo sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 0 -> 1

The bridge has to be created by hand because bastille setup does not do it for you.

PF

The PF skeleton that bastille setup generated:

ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if
pass in proto tcp from any to any port ssh flags S/SA keep state

With this config, jails on the bridge can’t talk to each other, can’t reach the internet, can’t resolve DNS. The block in all rule kills everything on bridge0. I went back and forth on this for a while before I understood: PF inspects every packet between jails on the bridge and drops them all. Also, the NAT rule uses <jails>, which is the shared-IP table - empty for VNET jails.

What I ended up with:

ext_if="vtnet0"
bridge_if="bridge0"
jail_net="10.0.0.0/24"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
set skip on $bridge_if

table <jails> persist
nat on $ext_if from $jail_net to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if
pass in proto tcp from any to any port ssh flags S/SA keep state

The working config differed in two places: set skip on $bridge_if so PF leaves bridge traffic alone, and NAT on the actual subnet instead of the empty table.

Two jails, one bridge

$ sudo bastille create -B web 15.0-RELEASE 10.0.0.1/24 bridge0
$ sudo bastille create -B backend 15.0-RELEASE 10.0.0.2/24 bridge0

-B is bridge-based VNET. I initially used -V, which is for physical interface passthrough and not what I wanted. I did not catch that distinction from the docs on the first read.

Both jails start on create, but they can’t reach anything. Another gotcha: Bastille sets defaultrouter to 192.168.122.1, which is the libvirt gateway on the host side. The jails are on 10.0.0.0/24. They need the bridge as gateway. You fix this per jail:

$ sudo bastille cmd web route add default 10.0.0.254
$ sudo bastille cmd backend route add default 10.0.0.254

After that:

$ sudo bastille cmd web ping -c 1 10.0.0.2
64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=0.039 ms

$ sudo bastille cmd web ping -c 1 8.8.8.8
64 bytes from 8.8.8.8: icmp_seq=0 ttl=115 time=12.475 ms

Jail-to-jail latency was 0.039 ms, so the epair plus bridge path was not where the setup was going wrong.

VNET jail names also cannot contain - or _. I tried template-test first and got [ERROR]: VNET jail names may not contain (-|_) characters. The name ends up in the epair interface name, so I had to stick to alphanumeric names.

Nginx reverse proxy across jails

Nothing fancy: backend serves a page on 8080, web proxies to it on port 80.

Backend config:

worker_processes 1;
events { worker_connections 64; }
http {
    server {
        listen 8080;
        server_name localhost;
        location / {
            root /usr/local/www/api;
            index index.html;
        }
        location /health {
            return 200 "ok\n";
        }
    }
}

Web reverse proxy:

worker_processes 1;
events { worker_connections 64; }
http {
    upstream backend {
        server 10.0.0.2:8080;
    }
    server {
        listen 80;
        server_name localhost;
        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

$ sudo bastille pkg web install -y nginx
$ sudo bastille pkg backend install -y nginx
$ sudo bastille sysrc backend nginx_enable=YES
$ sudo bastille service backend nginx start
$ sudo bastille sysrc web nginx_enable=YES
$ sudo bastille service web nginx start

Test the chain:

$ curl -s http://10.0.0.1/
Hello from backend jail (10.0.0.2)

The request hit web:80, got proxied to backend:8080, and came back with the backend page.

Bastillefiles

Bastille has a simple template system. A Bastillefile supports directives like PKG, CP, SYSRC, and SERVICE, and the supporting files live in the same directory.

I wrote one at /usr/local/bastille/templates/local/nginx-proxy/:

# Bastillefile
PKG nginx
CP nginx.conf /usr/local/etc/nginx/nginx.conf
SYSRC nginx_enable=YES
SERVICE nginx start

CP copies a file from the template directory into the jail. Fresh jail, apply it:

$ sudo bastille create -B tmpltest 15.0-RELEASE 10.0.0.3/24 bridge0
$ sudo bastille cmd tmpltest route add default 10.0.0.254
$ sudo bastille template tmpltest local/nginx-proxy

[tmpltest]:
Applying template: local/nginx-proxy...
[tmpltest] [2/2] Installing nginx-1.28.0_10,3...
nginx_enable:  -> YES
Starting nginx.

Template applied: local/nginx-proxy

$ curl -s http://10.0.0.3/
Deployed via Bastillefile

I applied the template to a fresh jail and nginx came up with the expected config.

ZFS: snapshots, clones, and a footgun

Thin jails (default) share the base system via nullfs: 76 MB with nginx installed vs 374 MB for a thick jail with nothing. With 10 jails, that’s 3 GB of duplicated base system you don’t need.

Snapshots and rollback are built in:

$ sudo bastille zfs web snap beforechange
$ sudo bastille cmd web sh -c 'echo MODIFIED > /usr/local/www/api-test.txt'
$ sudo bastille zfs web rollback beforechange
$ sudo bastille cmd web ls /usr/local/www/
nginx
nginx-dist

After the rollback, the file is gone. You can also clone a running jail (bastille clone -l web webclone 10.0.0.5) and start it immediately with the new IP and the existing config.

The footgun: bastille export without a compression flag (--xz, --gz, --zst) dumps 121 MB of raw ZFS stream to stdout. Always pass a compression flag.

Port forwarding: one more PF surprise

I started with:

$ sudo bastille rdr web tcp 8080 80
[ERROR]: VNET jails do not support rdr.

bastille rdr only works with shared-IP jails. For VNET, you write PF rdr rules by hand:

rdr on $ext_if proto tcp from any to ($ext_if) port 8080 -> 10.0.0.1 port 80
pass in proto tcp from any to 10.0.0.1 port 80 flags S/SA keep state

And PF rule ordering bit me again: rdr rules must come before filter rules, or you get Rules must be in order: options, ethernet, normalization, queueing, translation, filtering.

Where this leaves me

The missing pieces were all around VNET networking. I still had to create the bridge, fix the default routes inside the jails, and adjust PF for bridge traffic. Port forwarding also meant writing rdr rules by hand because bastille rdr does not apply.

Once that was in place, the two-jail setup behaved normally.

Next up: Pot + Nomad + Consul. That setup shifts from local jail networking to scheduling and service-to-service plumbing.

Sources and references:

• Bastille documentation
• FreeBSD Handbook: Jails
• FreeBSD Handbook: PF
• Comparison post: Bastille vs Pot
• VM setup guide

Antenore Gatta

A proud and busy Hacker, Father and Kyndrol

Post comment