Recently I built up a new storage server running FreeBSD. Initially I was going to go with FreeNAS like my old storage server, however the FreeNAS project is in a bit of flux at the moment and I thought this would be a good way to learn about the inner workings of FreeBSD. Part of this is segregating the applications running on the server in to “jails”. They are a form of OS-level virtualization, where each jail has its own files, processes and user accounts.
I was tempted to run a jail management tool such as ezjail, iocage or qjail, however configuring manually through jail.conf and the jail command seems to be quite easy once you wrap your head around it. This guide walks through building a base jail template that can easily be added as a layer to new jails. This approach is great as you only need to update one layer when OS-level updates get released.
First, enable jails in your OS.
Create a dataset for the jails and thinjails
1
2
zfs create -o mountpoint=/usr/local/jails zroot/jails
zfs create zroot/jails/thinjails
Then, create another dataset for the 11.0-RELEASE files
1
zfs create -p zroot/jails/releases/11.0-RELEASE
Download and extract all required binaries from a FreeBSD mirror. I chose Optus Australia as that is my closest/fastest mirror.
1
2
3
4
5
6
fetch ftp://ftp4.au.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/base.txz -o /tmp/base.txz
fetch ftp://ftp4.au.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/lib32.txz -o /tmp/lib32.txz
fetch ftp://ftp4.au.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/ports.txz -o /tmp/ports.txz
tar -xvf /tmp/base.txz -C /usr/local/jails/releases/11.0-RELEASE
tar -xvf /tmp/lib32.txz -C /usr/local/jails/releases/11.0-RELEASE
tar -xvf /tmp/ports.txz -C /usr/local/jails/releases/11.0-RELEASE
Update to latest patch level (p10 at the release of this blog post)
1
env UNAME_r=11.0-RELEASE freebsd-update -b /usr/local/jails/releases/11.0-RELEASE fetch install
Verify nothing was damaged in transit (this is more paranoia than anything…)
1
env UNAME_r=11.0-RELEASE freebsd-update -b /usr/local/jails/releases/11.0-RELEASE IDS
Add local timezone and DNS servers to files
1
2
cp /etc/resolv.conf /usr/local/jails/releases/11.0-RELEASE/etc/resolv.conf
cp /etc/localtime /usr/local/jails/releases/11.0-RELEASE/etc/localtime
Snapshot the release. Since mine is patch level 10, I chose the name p10
1
zfs snapshot zroot/jails/releases/11.0-RELEASE@p10
Clone the 11.0-RELEASE snapshot to a new base jail. This will be the first layer in future jails.
1
2
zfs create zroot/jails/templates
zfs clone zroot/jails/releases/11.0-RELEASE@p10 zroot/jails/templates/base-11.0-RELEASE
Create a skeleton dataset that will be used for jail-specific files
1
zfs create -p zroot/jails/templates/skeleton-11.0-RELEASE
Create folders in skeleton dataset
1
2
3
mkdir -p /usr/local/jails/templates/skeleton-11.0-RELEASE/usr/ports/distfiles
mkdir -p /usr/local/jails/templates/skeleton-11.0-RELEASE/home
mkdir -p /usr/local/jails/templates/skeleton-11.0-RELEASE/portsbuild
Move folders from the base jail layer to the skeleton jail layer. These are jail-specific directories.
1
2
3
4
5
mv /usr/local/jails/templates/base-11.0-RELEASE/etc /usr/local/jails/templates/skeleton-11.0-RELEASE/etc
mv /usr/local/jails/templates/base-11.0-RELEASE/usr/local /usr/local/jails/templates/skeleton-11.0-RELEASE/usr/local
mv /usr/local/jails/templates/base-11.0-RELEASE/tmp /usr/local/jails/templates/skeleton-11.0-RELEASE/tmp
mv /usr/local/jails/templates/base-11.0-RELEASE/var /usr/local/jails/templates/skeleton-11.0-RELEASE/var
mv /usr/local/jails/templates/base-11.0-RELEASE/root /usr/local/jails/templates/skeleton-11.0-RELEASE/root
For some reason /var/empty still remained when I moved the folders, so I had to remove it manually.
1
2
chflags 0 /usr/local/jails/templates/base-11.0-RELEASE/var/empty
rm -r /usr/local/jails/templates/base-11.0-RELEASE/var
Symlink the directories to the skeleton layer. Make sure you are in /usr/local/jails/templates/base-11.0-RELEASE/ folder before running the commands, as the paths are all relative.
1
2
3
4
5
6
7
8
9
cd /usr/local/jails/templates/base-11.0-RELEASE
mkdir skeleton
ln -s skeleton/etc etc
ln -s skeleton/home home
ln -s skeleton/root root
ln -s ../skeleton/usr/local usr/local
ln -s ../../skeleton/usr/ports/distfiles usr/ports/distfiles
ln -s skeleton/tmp tmp
ln -s skeleton/var var
Update the ports make configuration to build in the non-default directory
1
echo "WRKDIRPREFIX?= /skeleton/portbuild" >> /usr/local/jails/templates/skeleton-11.0-RELEASE/etc/make.conf
Snapshot the skeleton so we can clone it in each jail
1
zfs snapshot zroot/jails/templates/skeleton-11.0-RELEASE@skeleton
Create /etc/jail.conf with the below text. I have added comments to explain each section.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# The interface that the jails will interface with on the host
interface = "igb0";
# The name of each jail. $name is a placeholder. In the example below, $name would be jail1
host.hostname = "$name.domain.local";
# The path to the jail files
path = "/usr/local/jails/$name";
# The IP address of the jail. In the example below, it would be 10.1.1.40
ip4.addr = 10.1.1.$ip;
# The fstab file for the jail. This resolves to jail1.fstab in the example.
mount.fstab = "/usr/local/jails/$name.fstab";
# Common functions for all jails
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
# jail1 specific configuration
jail1 {
$ip = 40;
}
Done! We now have the components to create many thin jails. Now we will create our first thin jail off this template.
Creating a thin jail First, clone the skeleton files for the new jail, and then set the hostname
1
2
zfs clone zroot/jails/templates/skeleton-11.0-RELEASE@skeleton zroot/jails/thinjails/jail1
echo hostname=\"jail1\" > /usr/local/jails/thinjails/jail1/etc/rc.conf
Create folder where the layers for the jail will be mounted
1
mkdir -p /usr/local/jails/jail1
Create fstab at /usr/local/jails/jail1.fstab . The first layer is the base, mounted as read-only. The next is the skeleton we cloned, mounted as read-write.
1
2
/usr/local/jails/templates/base-11.0-RELEASE /usr/local/jails/jail1/ nullfs ro 0 0
/usr/local/jails/thinjails/jail1 /usr/local/jails/jail1/skeleton nullfs rw 0 0
Create and start jail “jail1”
Add jail to auto-start on boot in /etc/rc.conf
For each new jail, just follow the process:
Add config to /etc/jail.conf
Clone skeleton to /usr/local/jails/thinjails/<jailname>
Write hostname to /etc/rc.conf in new jail files
Create folder /usr/local/jails/<jailname> for new jail
Create fstab /usr/local/jails/<jailname>.fstab and populate with layer information
Create and start with jail -c jailname
Whenever we want to update all jails at once, shut down the jails and run:
1
2
env UNAME_r=11.0-RELEASE freebsd-update -b /usr/local/jails/templates/base-11.0-RELEASE fetch install
portsnap -p /usr/local/jails/templates/base-11.0-RELEASE/usr/ports auto
This will update the base to the latest patch level, and update to the latest ports tree.