FreeBSD virtual environment management and repository

2020-10 upd: we reached the first fundraising goal and rented a server in Hetzner for development! Thank you for donating !

Attention! I apologize for the automatic translation of this text. You can improve it by sending me a more correct version of the text or fix html pages via GITHUB repository.

CBSD jail on GlusterFS HOWTO

This howto demonstrates CBSD jail setup on cluster filesystem GluserFS

Please note that replication causes additional overhead for system resources. You will need about +1Gb RAM for glusterfs processes. Also, it is desirable to have fast hard drives (SSD and 1Gbit/s give you aprox. 250 Gbit/s on gluster(?)) In addition, avoid such installations for containers with a high write load


1) Install sysutils/cbsd

2) Init CBSD workdir: env workdir=/usr/jails /usr/local/cbsd/sudoexec/initenv

Glusterfs setup:

1) kldload fuse

2) sysrc -qf /boot/loader.conf fuse_load="YES"

3) zfs create -o mountpoint=/glusterfs zroot/glusterfs

4) pkg install -y net/glusterfs

5) sysrc glusterd_enable="YES"

6) service glusterd start

7) vi /etc/hosts: - add IP/name for each node, e.g: glu1 glu2

8) mkdir /glusterfs/brick1 /glusterfs/brick2 /glusterfs/brick3

glu1% gluster peer probe glu2

glu2% gluster peer status

glu1% (optional): gluster peer probe glu1

glu1% gluster pool list

glu1% gluster volume create jails-data replica 2 glu1:/glusterfs/brick1 glu2:/glusterfs/brick1

glu1% gluster volume create jails-system replica 2 glu1:/glusterfs/brick2 glu2:/glusterfs/brick2

glu1% gluster volume create jails-fstab replica 2 glu1:/glusterfs/brick3 glu2:/glusterfs/brick3

glu1% gluster volume start jails-data

glu1% gluster volume start jails-system

glu1% gluster volume start jails-fstab

glu1% gluster volume bitrot jails-data enable

glu1% gluster volume heal jails-data enable

glu1% gluster volume bitrot jails-system enable

glu1% gluster volume heal jails-system enable

glu1% gluster volume bitrot jails-fstab enable

glu1% gluster volume heal jails-fstab enable

Check state:

gluster volume info jails-data

gluster volume info jails-fstab

gluster volume info jails-system

Automount for flusterfs volumes:

Create script in /usr/local/etc/rc.d/gluster_mount:

# PROVIDE: gluster_mount
# REQUIRE: LOGIN glusterd
# KEYWORD: shutdown

. /etc/rc.subr


        [ ! -d ${cbsd_workdir}/jails-data/.trashcan ] && /usr/local/sbin/mount_glusterfs localhost:/jails-data ${cbsd_workdir}/jails-data
        [ ! -d ${cbsd_workdir}/jails-system/.trashcan ] && /usr/local/sbin/mount_glusterfs localhost:/jails-system ${cbsd_workdir}/jails-system
        [ ! -d ${cbsd_workdir}/jails-fstab/.trashcan ] && /usr/local/sbin/mount_glusterfs localhost:/jails-system ${cbsd_workdir}/jails-fstab

        [ -d ${cbsd_workdir}/jails-data/.trashcan ] && /sbin/umount ${cbsd_workdir}/jails-data
        [ -d ${cbsd_workdir}/jails-system/.trashcan ] && /sbin/umount ${cbsd_workdir}/jails-system
        [ -d ${cbsd_workdir}/jails-fstab/.trashcan ] && /sbin/umount ${cbsd_workdir}/jails-fstab

load_rc_config $name
run_rc_command "$1" 

chmod +x /usr/local/etc/rc.d/gluster_mount

service gluster_mount start

Re-run CBSD initenv to invert zfsfeat (automatically):

cbsd initenv


Create jail and start it as usual