Bastille on FreeBSD 15.0: VNET Jails, Bastillefiles, and Zero-Fuss Networking

The comparison post showed Bastille works: VNET jails in two commands, networking without touching pf. But that was a single jail running nginx. This post is what happens when I try to build something real with it: two jails that need to talk to each other, Bastillefiles that deploy a service in one command, and every networking detail that the quick test didn’t cover.

Fair warning: the networking section is where most of the pain lives. Bastille’s setup command gets you 80% of the way. The last 20% is bridge configuration, PF rules, and routing that you have to do yourself. I’ll show exactly what that looks like.

The VM

Same setup from the headless VM guide. The only change is the cloud-init packages: list:

packages:
  - sudo
  - bastille
  - nginx
  - curl

That gives me Bastille pre-installed on first boot. Everything else (4 GB RAM, 2 vCPUs, 30 GB ZFS on QEMU/KVM) is identical.

Bastille setup

bastille setup does the initial configuration: enables ZFS on zroot, creates the bastille0 loopback interface, generates a PF skeleton in /etc/pf.conf.

$ sudo bastille setup
bastille_enable:  -> YES
bastille_zfs_enable: NO -> YES
bastille_zfs_zpool:  -> zroot

Using ZFS filesystem.

Configuring bastille0 loopback interface
cloned_interfaces:  -> lo1
ifconfig_lo1_name:  -> bastille0

Bringing up new interface: [bastille0]
Created clone interfaces: lo1.

Loopback interface successfully configured: [bastille0]

Determined default network interface: (vtnet0)
/etc/pf.conf does not exist: creating...
pf_enable: NO -> YES
pf ruleset created, please review /etc/pf.conf and enable it using 'service pf start'.

Then bootstrap the release:

$ sudo bastille bootstrap 15.0-RELEASE update
Fetching distfile: base.txz
/usr/local/bastille/cache/15.0-RELEASE/base.txz  157 MB  15 MBps  10s

Bootstrap successful.

157 MB download, and you have a base system to create jails from.

The networking problem

This is the part the comparison post glossed over.

Bastille’s setup creates a loopback interface (bastille0) and a PF config that works for shared-IP jails. But if you want VNET jails on their own subnet - and you do, because that’s where the real isolation lives - you need a bridge. Bastille doesn’t create one for you.

Here’s what I had to do:

$ sudo ifconfig bridge create
bridge0

$ sudo ifconfig bridge0 inet 10.0.0.254/24
$ sudo sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 0 -> 1

The bridge is the subnet for the jails. 10.0.0.254 is the host acting as gateway. IP forwarding lets traffic flow between the bridge network and the outside world.

Now PF. The skeleton from bastille setup looks like this:

ext_if="vtnet0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if
pass in proto tcp from any to any port ssh flags S/SA keep state

This blocks everything on bridge0. Jails can’t talk to each other, can’t reach the internet, can’t resolve DNS. I had to rewrite it:

ext_if="vtnet0"
bridge_if="bridge0"
jail_net="10.0.0.0/24"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
set skip on $bridge_if

table <jails> persist
nat on $ext_if from $jail_net to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if
pass in proto tcp from any to any port ssh flags S/SA keep state

The key line is set skip on $bridge_if. Without it, PF inspects all traffic between jails on the bridge and the block in all rule drops it. With it, jail-to-jail traffic flows freely.

The NAT rule changed too: from <jails> (the shared-IP table, which is empty for VNET jails) to the actual jail subnet.

Two jails, one bridge

$ sudo bastille create -B web 15.0-RELEASE 10.0.0.1/24 bridge0
Attempting to create jail: web
Valid IP: 10.0.0.1/24
Valid interface: bridge0
Creating a thinjail...

$ sudo bastille create -B backend 15.0-RELEASE 10.0.0.2/24 bridge0
Attempting to create jail: backend
Valid IP: 10.0.0.2/24
Valid interface: bridge0
Creating a thinjail...

-B means bridge-based VNET. Not -V, which is for physical interface passthrough. I got this wrong the first time.

Both jails start automatically on create. But there’s a routing problem: Bastille sets defaultrouter to 192.168.122.1 (the libvirt gateway on the host’s external network). The jails can’t reach that. They need the bridge gateway:

$ sudo bastille cmd web route add default 10.0.0.254
add net default: gateway 10.0.0.254

$ sudo bastille cmd backend route add default 10.0.0.254
add net default: gateway 10.0.0.254

Now they can reach each other and the internet:

$ sudo bastille cmd web ping -c 1 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=0.039 ms

$ sudo bastille cmd web ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=115 time=12.475 ms

0.039ms jail-to-jail. That’s the epair+bridge overhead: basically nothing.

One more gotcha: VNET jail names cannot contain - or _. I tried template-test and got:

[ERROR]: VNET jail names may not contain (-|_) characters.

This is an epair interface naming limitation. The jail name becomes part of the interface name (e0a_web, e0b_web), and hyphens cause problems. Use short alphanumeric names.

The multi-service setup

Backend jail serves a simple page on port 8080. Web jail runs nginx as a reverse proxy on port 80, forwarding to the backend.

Install nginx in both jails:

$ sudo bastille pkg web install -y nginx
$ sudo bastille pkg backend install -y nginx

Backend config (/usr/local/etc/nginx/nginx.conf inside the jail):

worker_processes 1;
events { worker_connections 64; }
http {
    server {
        listen 8080;
        server_name localhost;
        location / {
            root /usr/local/www/api;
            index index.html;
        }
        location /health {
            return 200 "ok\n";
        }
    }
}

Web reverse proxy config:

worker_processes 1;
events { worker_connections 64; }
http {
    upstream backend {
        server 10.0.0.2:8080;
    }
    server {
        listen 80;
        server_name localhost;
        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

Start both:

$ sudo bastille sysrc backend nginx_enable=YES
$ sudo bastille service backend nginx start
Starting nginx.

$ sudo bastille sysrc web nginx_enable=YES
$ sudo bastille service web nginx start
Starting nginx.

Test the chain:

$ curl -s http://10.0.0.2:8080/
Hello from backend jail (10.0.0.2)

$ curl -s http://10.0.0.2:8080/health
ok

$ curl -s http://10.0.0.1/
Hello from backend jail (10.0.0.2)

The last line is the one that matters: the request went to web:80, got proxied to backend:8080, and the response came back. Two jails, one bridge, zero fuss once you get past the PF config.

Bastillefiles

Bastillefiles are Bastille’s version of Dockerfiles. You write a template with directives (PKG, CP, SYSRC, SERVICE, etc.) and apply it to a jail. The jail gets configured in one shot.

I created a template at /usr/local/bastille/templates/local/nginx-proxy/:

# Bastillefile
PKG nginx
CP nginx.conf /usr/local/etc/nginx/nginx.conf
SYSRC nginx_enable=YES
SERVICE nginx start

The CP directive copies a file from the template directory into the jail. So I put an nginx.conf next to the Bastillefile:

worker_processes 1;
events { worker_connections 64; }
http {
    server {
        listen 80;
        server_name localhost;
        location / {
            return 200 "Deployed via Bastillefile\n";
        }
    }
}

Applied it to a fresh jail:

$ sudo bastille create -B tmpltest 15.0-RELEASE 10.0.0.3/24 bridge0
$ sudo bastille cmd tmpltest route add default 10.0.0.254
$ sudo bastille template tmpltest local/nginx-proxy

[tmpltest]:
Applying template: local/nginx-proxy...

[tmpltest]:
[tmpltest] Installing pkg-2.5.1...
[tmpltest] Extracting pkg-2.5.1: .......... done
...
[tmpltest] [2/2] Installing nginx-1.28.0_10,3...

[tmpltest]:
/usr/local/bastille/templates/local/nginx-proxy/nginx.conf -> /usr/local/bastille/jails/tmpltest/root/usr/local/etc/nginx/nginx.conf

[tmpltest]:
nginx_enable:  -> YES

[tmpltest]:
Starting nginx.

Template applied: local/nginx-proxy

$ curl -s http://10.0.0.3/
Deployed via Bastillefile

One command: package installed, config copied, service enabled and started. This is what makes Bastille practical for repeatable deployments. Write the template once, apply it to as many jails as you need.

Thin vs thick jails

By default Bastille creates thin jails: the base system is shared via nullfs mounts from the release, and the jail only stores its own modifications. Thick jails get a full copy.

$ sudo bastille create -T -B thicktest 15.0-RELEASE 10.0.0.4/24 bridge0
Creating a thickjail. This may take a while...

The size difference:

$ sudo zfs list zroot/bastille/jails/web
NAME                       USED  AVAIL  REFER  MOUNTPOINT
zroot/bastille/jails/web  76.1M  23.6G   108K  /usr/local/bastille/jails/web

$ sudo zfs list zroot/bastille/jails/thicktest
NAME                             USED  AVAIL  REFER  MOUNTPOINT
zroot/bastille/jails/thicktest   374M  23.6G   108K  /usr/local/bastille/jails/thicktest

$ sudo zfs list zroot/bastille/releases/15.0-RELEASE
NAME                                   USED  AVAIL  REFER  MOUNTPOINT
zroot/bastille/releases/15.0-RELEASE   374M  23.6G   374M  /usr/local/bastille/releases/15.0-RELEASE

The thin jail with nginx installed: 76 MB. The thick jail with nothing extra: 374 MB (the full release). If you’re running 10 jails, thin saves you over 3 GB of duplicated base system.

Thick jails make sense when you need to modify base system files or want a completely self-contained jail you can move to another machine. For most use cases, thin is the right default.

ZFS snapshots and rollback

Bastille wraps ZFS snapshots neatly:

$ sudo bastille zfs web snap beforechange
Snapshot created: beforechange

$ sudo bastille cmd web sh -c 'echo MODIFIED > /usr/local/www/api-test.txt'
$ sudo bastille cmd web ls /usr/local/www/
api-test.txt
nginx
nginx-dist

$ sudo bastille zfs web rollback beforechange
Snapshot restored: beforechange

$ sudo bastille cmd web ls /usr/local/www/
nginx
nginx-dist

The file is gone. Rollback covers both the jail config and the root filesystem. This is one of those things that ZFS makes trivially easy and that you’d miss on any other filesystem.

Live clone and export

Bastille can clone a running jail via ZFS snapshot:

$ sudo bastille clone -l web webclone 10.0.0.5
Attempting to clone 'web' to 'webclone'...
Valid IP: 10.0.0.5/24
ifconfig_vnet0: inet 10.0.0.1/24 -> inet 10.0.0.5/24

Cloned 'web' to 'webclone' successfully.

The clone starts immediately with the updated IP. The nginx reverse proxy config came along:

$ sudo bastille cmd webclone route add default 10.0.0.254
$ curl -s http://10.0.0.5/
Hello from backend jail (10.0.0.2)

For backups or migration, bastille export creates a compressed ZFS stream:

$ sudo bastille export -l --xz web
Sending ZFS data stream...
(stdin): 43.0 MiB / 121.8 MiB = 0.353, 0:02

Exported '/usr/local/bastille/backups/web_2026-03-20-160851.xz' successfully.

43 MB compressed. The -l flag means you don’t have to stop the jail. Without --xz (or --gz, --zst), the raw ZFS stream goes to stdout, which is probably not what you want: 121 MB of binary data in your terminal.

Port forwarding from the host

I expected bastille rdr to handle this:

$ sudo bastille rdr web tcp 8080 80
[ERROR]: VNET jails do not support rdr.

Makes sense when you think about it: bastille rdr works with the shared-IP loopback model. VNET jails have their own network stack, so you need PF rdr rules directly:

# In /etc/pf.conf, in the translation section (before filter rules):
rdr on $ext_if proto tcp from any to ($ext_if) port 8080 -> 10.0.0.1 port 80

# And a pass rule:
pass in proto tcp from any to 10.0.0.1 port 80 flags S/SA keep state

$ sudo pfctl -f /etc/pf.conf
$ curl -s http://192.168.122.105:8080/
Hello from backend jail (10.0.0.2)

Works. But the PF ordering matters: rdr rules must come before filter rules, or you get Rules must be in order: options, ethernet, normalization, queueing, translation, filtering. I got that error on the first try.

Watch Out

1. Bridge setup is manual. Bastille creates loopback interfaces for shared-IP jails but doesn’t create or configure bridges for VNET. You need ifconfig bridge create, assign it an IP, and enable IP forwarding yourself.

2. PF blocks everything by default. The generated /etc/pf.conf has block in all and no rules for bridge traffic. Add set skip on $bridge_if or you’ll spend time debugging why jails can’t ping each other.

3. Default route points to the wrong gateway. Bastille sets defaultrouter in the jail’s rc.conf to the host’s external gateway (192.168.122.1 in my case). VNET jails on a bridge need the bridge IP as gateway (10.0.0.254). You have to fix this per jail.

4. VNET jail names can’t have - or _. The jail name becomes part of the epair interface name. Keep it alphanumeric.

5. bastille rdr doesn’t work with VNET. Use PF rdr rules directly. And remember PF rule ordering: translation before filtering.

6. bastille export without a compression flag dumps raw ZFS to stdout. Always use --xz, --gz, or --zst.

What’s next

This was Bastille for single-node: two services finding each other on a bridge, deployed via Bastillefiles, backed by ZFS snapshots. It works well. The networking setup is the main friction point, and once you’ve done it once you can repeat it.

The Pot + Nomad + Consul deep dive is the other side: what happens when you want scheduling, service discovery, and health checks. That’s a different kind of complexity.

Sources and references:

• Bastille documentation
• FreeBSD Handbook: Jails
• FreeBSD Handbook: PF
• Comparison post: Bastille vs Pot
• VM setup guide

Antenore Gatta

A proud and busy Hacker, Father and Kyndrol

Post comment