Bastille on FreeBSD 15.0: VNET Jails, Bastillefiles, and Zero-Fuss Networking

The comparison post left me with a single jail running nginx. That’s a demo, not a test. So I set up something closer to real: two VNET jails on a bridge, one running nginx as a reverse proxy, the other serving a backend on port 8080. Deployed with Bastillefiles. The kind of thing you’d actually want to run.

The commands themselves are fine. I spent most of my time fighting PF and bridge routing.

The VM

Same VM from the headless guide. I just added Bastille to cloud-init:

packages:
  - sudo
  - bastille
  - nginx
  - curl

That gives me Bastille pre-installed on first boot. Everything else (4 GB RAM, 2 vCPUs, 30 GB ZFS on QEMU/KVM) is identical.

Bastille setup

First thing after boot: bastille setup. It enables ZFS, creates a loopback interface, and drops a PF skeleton in /etc/pf.conf.

$ sudo bastille setup
bastille_enable:  -> YES
bastille_zfs_enable: NO -> YES
bastille_zfs_zpool:  -> zroot

Using ZFS filesystem.

Configuring bastille0 loopback interface
cloned_interfaces:  -> lo1
ifconfig_lo1_name:  -> bastille0

Bringing up new interface: [bastille0]
Created clone interfaces: lo1.

Loopback interface successfully configured: [bastille0]

Determined default network interface: (vtnet0)
/etc/pf.conf does not exist: creating...
pf_enable: NO -> YES
pf ruleset created, please review /etc/pf.conf and enable it using 'service pf start'.

Then you bootstrap a release:

$ sudo bastille bootstrap 15.0-RELEASE update
Fetching distfile: base.txz
/usr/local/bastille/cache/15.0-RELEASE/base.txz  157 MB  15 MBps  10s

Bootstrap successful.

157 MB and you have a base system ready for jails. So far, easy.

The networking problem

This is where the comparison post stopped being useful to me.

Bastille’s setup creates a loopback interface and a PF config. That works for shared-IP jails. But I wanted VNET jails on their own subnet, which means a bridge, and Bastille doesn’t create one. You do it yourself:

$ sudo ifconfig bridge create
bridge0

$ sudo ifconfig bridge0 inet 10.0.0.254/24
$ sudo sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 0 -> 1

10.0.0.254 is the host side of the bridge, acting as gateway. IP forwarding because otherwise nothing goes anywhere.

Then PF. The skeleton that bastille setup generated looks like this:

ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if
pass in proto tcp from any to any port ssh flags S/SA keep state

This took me a while to figure out. With this config, jails on the bridge can’t talk to each other, can’t reach the internet, can’t resolve DNS. The block in all rule kills everything on bridge0. I rewrote it:

ext_if="vtnet0"
bridge_if="bridge0"
jail_net="10.0.0.0/24"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
set skip on $bridge_if

table <jails> persist
nat on $ext_if from $jail_net to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if
pass in proto tcp from any to any port ssh flags S/SA keep state

set skip on $bridge_if is what fixed it. Without that, PF inspects every packet between jails on the bridge and drops them all. I also changed the NAT rule from <jails> (the shared-IP table, empty for VNET jails) to the actual subnet.

Two jails, one bridge

$ sudo bastille create -B web 15.0-RELEASE 10.0.0.1/24 bridge0
Attempting to create jail: web
Valid IP: 10.0.0.1/24
Valid interface: bridge0
Creating a thinjail...

$ sudo bastille create -B backend 15.0-RELEASE 10.0.0.2/24 bridge0
Attempting to create jail: backend
Valid IP: 10.0.0.2/24
Valid interface: bridge0
Creating a thinjail...

-B is bridge-based VNET. I initially used -V, which is for physical interface passthrough - not what I wanted.

Both jails start on create, but they can’t reach anything. Bastille sets defaultrouter to 192.168.122.1 (the libvirt gateway on the host side). The jails are on 10.0.0.0/24, they need the bridge as gateway:

$ sudo bastille cmd web route add default 10.0.0.254
add net default: gateway 10.0.0.254

$ sudo bastille cmd backend route add default 10.0.0.254
add net default: gateway 10.0.0.254

After that, both jails can reach each other and the internet:

$ sudo bastille cmd web ping -c 1 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=0.039 ms

$ sudo bastille cmd web ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=115 time=12.475 ms

0.039ms jail-to-jail. The epair+bridge overhead is basically zero.

Oh, and VNET jail names can’t have - or _. I tried template-test first:

[ERROR]: VNET jail names may not contain (-|_) characters.

The jail name ends up in the epair interface name (e0a_web, e0b_web), and hyphens break it. Alphanumeric only.

The multi-service setup

Nothing fancy: backend serves a page on port 8080, web runs nginx as reverse proxy on port 80 forwarding to backend. Install nginx in both:

$ sudo bastille pkg web install -y nginx
$ sudo bastille pkg backend install -y nginx

Backend config (/usr/local/etc/nginx/nginx.conf inside the jail):

worker_processes 1;
events { worker_connections 64; }
http {
    server {
        listen 8080;
        server_name localhost;
        location / {
            root /usr/local/www/api;
            index index.html;
        }
        location /health {
            return 200 "ok\n";
        }
    }
}

Web reverse proxy config:

worker_processes 1;
events { worker_connections 64; }
http {
    upstream backend {
        server 10.0.0.2:8080;
    }
    server {
        listen 80;
        server_name localhost;
        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

Start both:

$ sudo bastille sysrc backend nginx_enable=YES
$ sudo bastille service backend nginx start
Starting nginx.

$ sudo bastille sysrc web nginx_enable=YES
$ sudo bastille service web nginx start
Starting nginx.

Test the chain:

$ curl -s http://10.0.0.2:8080/
Hello from backend jail (10.0.0.2)

$ curl -s http://10.0.0.2:8080/health
ok

$ curl -s http://10.0.0.1/
Hello from backend jail (10.0.0.2)

That last curl is the one I cared about: the request hit web:80, got proxied to backend:8080, and came back. The PF config is the real work here, the rest is just Bastille commands.

Bastillefiles

Bastille has its own template system - think Dockerfiles, but simpler. You put directives (PKG, CP, SYSRC, SERVICE) in a Bastillefile and apply it to a jail.

I wrote one at /usr/local/bastille/templates/local/nginx-proxy/:

# Bastillefile
PKG nginx
CP nginx.conf /usr/local/etc/nginx/nginx.conf
SYSRC nginx_enable=YES
SERVICE nginx start

CP copies a file from the template directory into the jail. I put an nginx.conf next to the Bastillefile:

worker_processes 1;
events { worker_connections 64; }
http {
    server {
        listen 80;
        server_name localhost;
        location / {
            return 200 "Deployed via Bastillefile\n";
        }
    }
}

Fresh jail, apply the template:

$ sudo bastille create -B tmpltest 15.0-RELEASE 10.0.0.3/24 bridge0
$ sudo bastille cmd tmpltest route add default 10.0.0.254
$ sudo bastille template tmpltest local/nginx-proxy

[tmpltest]:
Applying template: local/nginx-proxy...

[tmpltest]:
[tmpltest] Installing pkg-2.5.1...
[tmpltest] Extracting pkg-2.5.1: .......... done
...
[tmpltest] [2/2] Installing nginx-1.28.0_10,3...

[tmpltest]:
/usr/local/bastille/templates/local/nginx-proxy/nginx.conf -> /usr/local/bastille/jails/tmpltest/root/usr/local/etc/nginx/nginx.conf

[tmpltest]:
nginx_enable:  -> YES

[tmpltest]:
Starting nginx.

Template applied: local/nginx-proxy

$ curl -s http://10.0.0.3/
Deployed via Bastillefile

Package installed, config in place, service running. One command. I can see myself using this for any jail that needs more than a couple of packages.

Thin vs thick jails

Bastille creates thin jails by default: the base system is shared via nullfs from the release, the jail only has its own changes. You can also create thick jails with -T, which get a full copy of everything.

$ sudo bastille create -T -B thicktest 15.0-RELEASE 10.0.0.4/24 bridge0
Creating a thickjail. This may take a while...

The difference in disk usage is pretty clear:

$ sudo zfs list zroot/bastille/jails/web
NAME                       USED  AVAIL  REFER  MOUNTPOINT
zroot/bastille/jails/web  76.1M  23.6G   108K  /usr/local/bastille/jails/web

$ sudo zfs list zroot/bastille/jails/thicktest
NAME                             USED  AVAIL  REFER  MOUNTPOINT
zroot/bastille/jails/thicktest   374M  23.6G   108K  /usr/local/bastille/jails/thicktest

$ sudo zfs list zroot/bastille/releases/15.0-RELEASE
NAME                                   USED  AVAIL  REFER  MOUNTPOINT
zroot/bastille/releases/15.0-RELEASE   374M  23.6G   374M  /usr/local/bastille/releases/15.0-RELEASE

76 MB for a thin jail with nginx vs 374 MB for a thick jail with nothing installed. With 10 jails, that’s over 3 GB of duplicated base system you don’t need.

Thick jails make sense if you need to modify base system files or want something fully portable. For everything else, thin.

ZFS snapshots and rollback

This part I liked. Bastille wraps ZFS snapshots directly:

$ sudo bastille zfs web snap beforechange
Snapshot created: beforechange

$ sudo bastille cmd web sh -c 'echo MODIFIED > /usr/local/www/api-test.txt'
$ sudo bastille cmd web ls /usr/local/www/
api-test.txt
nginx
nginx-dist

$ sudo bastille zfs web rollback beforechange
Snapshot restored: beforechange

$ sudo bastille cmd web ls /usr/local/www/
nginx
nginx-dist

File gone. Rollback covers both jail config and the root filesystem. ZFS makes this trivial, and once you’re used to it you really don’t want to go back to anything else.

Live clone and export

You can clone a running jail. Bastille snapshots via ZFS and duplicates it:

$ sudo bastille clone -l web webclone 10.0.0.5
Attempting to clone 'web' to 'webclone'...
Valid IP: 10.0.0.5/24
ifconfig_vnet0: inet 10.0.0.1/24 -> inet 10.0.0.5/24

Cloned 'web' to 'webclone' successfully.

Starts immediately with the new IP. The whole nginx config came along:

$ sudo bastille cmd webclone route add default 10.0.0.254
$ curl -s http://10.0.0.5/
Hello from backend jail (10.0.0.2)

bastille export creates a compressed ZFS stream for backups or migration:

$ sudo bastille export -l --xz web
Sending ZFS data stream...
(stdin): 43.0 MiB / 121.8 MiB = 0.353, 0:02

Exported '/usr/local/bastille/backups/web_2026-03-20-160851.xz' successfully.

43 MB compressed. -l means you don’t have to stop the jail first. Without --xz (or --gz, --zst), you get 121 MB of raw ZFS stream dumped to stdout. I found out the hard way.

Port forwarding from the host

I expected this to be simple:

$ sudo bastille rdr web tcp 8080 80
[ERROR]: VNET jails do not support rdr.

Right. bastille rdr is for the shared-IP loopback model. VNET jails have their own network stack, so you write PF rdr rules directly:

# In /etc/pf.conf, in the translation section (before filter rules):
rdr on $ext_if proto tcp from any to ($ext_if) port 8080 -> 10.0.0.1 port 80

# And a pass rule:
pass in proto tcp from any to 10.0.0.1 port 80 flags S/SA keep state

$ sudo pfctl -f /etc/pf.conf
$ curl -s http://192.168.122.105:8080/
Hello from backend jail (10.0.0.2)

Works, but PF rule ordering bit me again: rdr rules must come before filter rules, or you get Rules must be in order: options, ethernet, normalization, queueing, translation, filtering.

Watch Out

1. Bridge setup is manual. Bastille creates loopback interfaces for shared-IP jails but doesn’t create or configure bridges for VNET. You need ifconfig bridge create, assign it an IP, and enable IP forwarding yourself.

2. PF blocks everything by default. The generated /etc/pf.conf has block in all and no rules for bridge traffic. Add set skip on $bridge_if or you’ll spend time debugging why jails can’t ping each other.

3. Default route points to the wrong gateway. Bastille sets defaultrouter in the jail’s rc.conf to the host’s external gateway (192.168.122.1 in my case). VNET jails on a bridge need the bridge IP as gateway (10.0.0.254). You have to fix this per jail.

4. VNET jail names can’t have - or _. The jail name becomes part of the epair interface name. Keep it alphanumeric.

5. bastille rdr doesn’t work with VNET. Use PF rdr rules directly. And remember PF rule ordering: translation before filtering.

6. bastille export without a compression flag dumps raw ZFS to stdout. Always use --xz, --gz, or --zst.

What’s next

Bastille for single-node works. The Bastillefiles, the ZFS integration, the jail management itself - all solid. The networking is where you pay the price, and I think Bastille could do more there: create the bridge, set the right default route, maybe even generate working PF rules for VNET. But once you’ve set it up once, you can repeat it.

Next I want to look at Pot + Nomad + Consul - scheduling, service discovery, health checks. A different set of problems.

Sources and references:

• Bastille documentation
• FreeBSD Handbook: Jails
• FreeBSD Handbook: PF
• Comparison post: Bastille vs Pot
• VM setup guide

Antenore Gatta

A proud and busy Hacker, Father and Kyndrol

Post comment