Recently I built up a new storage server running FreeBSD. Initially I was going to go with FreeNAS like my old storage server, however the FreeNAS project is in a bit of flux at the moment and I thought this would be a good way to learn about the inner workings of FreeBSD. Part of this is segregating the applications running on the server in to “jails”. They are a form of OS-level virtualization, where each jail has its own files, processes and user accounts.
I was tempted to run a jail management tool such as ezjail, iocage or qjail, however configuring manually through jail.conf and the jail command seems to be quite easy once you wrap your head around it. This guide walks through building a base jail template that can easily be added as a layer to new jails. This approach is great as you only need to update one layer when OS-level updates get released.
First, enable jails in your OS.
Create a dataset for the jails and thinjails
zfs create -o mountpoint=/usr/local/jails zroot/jails
zfs create zroot/jails/thinjails
Then, create another dataset for the 11.0-RELEASE files
zfs create -p zroot/jails/releases/11.0-RELEASE
Download and extract all required binaries from a FreeBSD mirror. I chose Optus Australia as that is my closest/fastest mirror.
fetch ftp://ftp4.au.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/base.txz -o /tmp/base.txz
fetch ftp://ftp4.au.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/lib32.txz -o /tmp/lib32.txz
fetch ftp://ftp4.au.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/ports.txz -o /tmp/ports.txz
tar -xvf /tmp/base.txz -C /usr/local/jails/releases/11.0-RELEASE
tar -xvf /tmp/lib32.txz -C /usr/local/jails/releases/11.0-RELEASE
tar -xvf /tmp/ports.txz -C /usr/local/jails/releases/11.0-RELEASE
Update to latest patch level (p10 at the release of this blog post)
env UNAME_r=11.0-RELEASE freebsd-update -b /usr/local/jails/releases/11.0-RELEASE fetch install
Verify nothing was damaged in transit (this is more paranoia than anything…)
env UNAME_r=11.0-RELEASE freebsd-update -b /usr/local/jails/releases/11.0-RELEASE IDS
Add local timezone and DNS servers to files
cp /etc/resolv.conf /usr/local/jails/releases/11.0-RELEASE/etc/resolv.conf
cp /etc/localtime /usr/local/jails/releases/11.0-RELEASE/etc/localtime
Snapshot the release. Since mine is patch level 10, I chose the name
zfs snapshot zroot/jails/releases/11.0-RELEASE@p10
11.0-RELEASE snapshot to a new base jail. This will be the first layer in future jails.
zfs create zroot/jails/templates
zfs clone zroot/jails/releases/11.0-RELEASE@p10 zroot/jails/templates/base-11.0-RELEASE
Create a skeleton dataset that will be used for jail-specific files
zfs create -p zroot/jails/templates/skeleton-11.0-RELEASE
Create folders in skeleton dataset
mkdir -p /usr/local/jails/templates/skeleton-11.0-RELEASE/usr/ports/distfiles
mkdir -p /usr/local/jails/templates/skeleton-11.0-RELEASE/home
mkdir -p /usr/local/jails/templates/skeleton-11.0-RELEASE/portsbuild
Move folders from the base jail layer to the skeleton jail layer. These are jail-specific directories.
mv /usr/local/jails/templates/base-11.0-RELEASE/etc /usr/local/jails/templates/skeleton-11.0-RELEASE/etc
mv /usr/local/jails/templates/base-11.0-RELEASE/usr/local /usr/local/jails/templates/skeleton-11.0-RELEASE/usr/local
mv /usr/local/jails/templates/base-11.0-RELEASE/tmp /usr/local/jails/templates/skeleton-11.0-RELEASE/tmp
mv /usr/local/jails/templates/base-11.0-RELEASE/var /usr/local/jails/templates/skeleton-11.0-RELEASE/var
mv /usr/local/jails/templates/base-11.0-RELEASE/root /usr/local/jails/templates/skeleton-11.0-RELEASE/root
For some reason
/var/empty still remained when I moved the folders, so I had to remove it manually.
chflags 0 /usr/local/jails/templates/base-11.0-RELEASE/var/empty
rm -r /usr/local/jails/templates/base-11.0-RELEASE/var
Symlink the directories to the skeleton layer. Make sure you are in
/usr/local/jails/templates/base-11.0-RELEASE/ folder before running the commands, as the paths are all relative.
ln -s skeleton/etc etc
ln -s skeleton/home home
ln -s skeleton/root root
ln -s ../skeleton/usr/local usr/local
ln -s ../../skeleton/usr/ports/distfiles usr/ports/distfiles
ln -s skeleton/tmp tmp
ln -s skeleton/var var
Update the ports make configuration to build in the non-default directory
echo "WRKDIRPREFIX?= /skeleton/portbuild" >> /usr/local/jails/templates/skeleton-11.0-RELEASE/etc/make.conf
Snapshot the skeleton so we can clone it in each jail
zfs snapshot zroot/jails/templates/skeleton-11.0-RELEASE@skeleton
Create /etc/jail.conf with the below text. I have added comments to explain each section.
# The interface that the jails will interface with on the host
interface = "igb0";
# The name of each jail. $name is a placeholder. In the example below, $name would be jail1
host.hostname = "$name.domain.local";
# The path to the jail files
path = "/usr/local/jails/$name";
# The IP address of the jail. In the example below, it would be 10.1.1.40
ip4.addr = 10.1.1.$ip;
# The fstab file for the jail. This resolves to jail1.fstab in the example.
mount.fstab = "/usr/local/jails/$name.fstab";
# Common functions for all jails
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
# jail1 specific configuration
$ip = 40;
Done! We now have the components to create many thin jails. Now we will create our first thin jail off this template.
Creating a thin jail
First, clone the skeleton files for the new jail, and then set the hostname
zfs clone zroot/jails/templates/skeleton-11.0-RELEASE@skeleton zroot/jails/thinjails/jail1
echo hostname=\"jail1\" > /usr/local/jails/thinjails/jail1/etc/rc.conf
Create folder where the layers for the jail will be mounted
mkdir -p /usr/local/jails/jail1
Create fstab at
/usr/local/jails/jail1.fstab . The first layer is the base, mounted as read-only. The next is the skeleton we cloned, mounted as read-write.
/usr/local/jails/templates/base-11.0-RELEASE /usr/local/jails/jail1/ nullfs ro 0 0
/usr/local/jails/thinjails/jail1 /usr/local/jails/jail1/skeleton nullfs rw 0 0
Create and start jail “jail1”
Add jail to auto-start on boot in
For each new jail, just follow the process:
Add config to
Clone skeleton to
Write hostname to
/etc/rc.conf in new jail files
/usr/local/jails/<jailname> for new jail
/usr/local/jails/<jailname>.fstab and populate with layer information
Create and start with
jail -c jailname
Whenever we want to update all jails at once, shut down the jails and run:
env UNAME_r=11.0-RELEASE freebsd-update -b /usr/local/jails/templates/base-11.0-RELEASE fetch install
portsnap -p /usr/local/jails/templates/base-11.0-RELEASE/usr/ports auto
This will update the base to the latest patch level, and update to the latest ports tree.