FreeBSD virtual environment management and repository

2020-10 upd: we reached the first fundraising goal and rented a server in Hetzner for development! Thank you for donating !

Operation with jail

Attention! Current pages describe CBSD version 13.0.x. If you are using an older version, please update first.

Attention! I apologize for the automatic translation of this text. You can improve it by sending me a more correct version of the text or fix html pages via GITHUB repository.

CREATING JAILS

commands: jcreate, jconstruct-tui

		% cbsd jconstruct-tui
		
		% cbsd jcreate jconf=/path/to/conf.jconf
		
Description:

The jail is created according to configuration file generated by cbsd jcreate jconf=path_to_cfg. For create configuration, use a command cbsd jconstruct (question-answer dialog mode), or dialog(1)-based interface: cbsd jconstruct-tui, or via WEB interface. When pkg.conf and repository configured properly, you can preset to new jail some packages mark them through pkglist menu. For safety reason recommended to use the official FreeBSD pkg repo or build your own package repository.

Please note: when repo unavailable, item pkglist in cbsd jconstruct-tui did not show anything.

Please note: multi-repo and and the ability to choose a specific repository from the list at the moment is missing, and will be used repository prescribed in pkg.conf of master host.

Attention! Because the list and choose of packages formed by pkg.conf of master machine, keep in mind that the content of the selected packages on the list will only work if the architecture and version of the jail equal with master node

Use the features of profiles:

If you do not use automation (Puppet, Ansible, own implementation for generation of jconf) to create environments and you have to create a container with different parameters, use the ability to create your own profile to override the default values. You can override any parameter - the proposed template for jail name, version, binding to a specific interface instead of 'auto', etc..

for creating config via dialog:

			% cbsd jconstruct-tui
		

The same menu is available to create a jail when configuring CBSD via bsdconfig

Let it not scare you a small ;-) the number of settings in jconstruct-tui - here are displayed only the most important options, suitable for most people. If you need a more flexible configuration - please use cbsd jconfig after jail creation

If jconstruct-tui work correctly, on the exit the question for jail create will be asked. In a case positively answer, jcreate it will be executed on a new configuration automatically. Otherwise, the script will save configuration file (in $workdir/ftmp), on which it is possible to create a jail by hand.

Important:

When a new jail is created or obtaining from the repository, make it a rule ALWAYS change the user's password root in jail, even if you do not plan to run it ssh/ftp/rsh and similar services. If the jail is created with applytpl=0, by default /etc/{passwd,master.passwd,group} in the jail as the original "clean" files FreeBSD, so password of root user is empty. If jail created with applytpl=1 (it also refers to images from repository) $workdir/share/jail-skel files will be used as templates where root password is 'cbsd' in default CBSD installation. You can change default root password when new jail is created via edit of hash in skel master.passwd via:

% vipw -d ${workdir}/share/jail-skel/etc

commands, or specify alternative path to jail-skel dir in .jconf (jcreate tools) config

You can override the jailskeldir="$workdir/share/jail-skel", parameters which is stored in $workdir/etc/defaults/jail-freebsd-default.conf to specify an alternate template directory, which will overwrite the files in the original base files when creating jail. To do this, create a file jail-freebsd-XXXXX.conf in $workdir/etc and enter the value.:

See for details: Profiles for jail creation

DIALOG MENU OPTIONS

  • profile - profile in which the parameters are set by default. Described above
  • pkglist - Select package list for new jail
  • jname - Short (one word) name of the jail
  • host_hostname - FQDN for the jail environment
  • ip4_addr - specify IP addresses (separated by commas if more than one) or 0 for vnet/bhyve environments
  • ver - FreeBSD version for the base (10.1, 11.0 for RELEASE or 10, 11 for STABLE)
  • baserw - mount copy of base via nullfs in read only (baserw=no) or populate own copy (baserw=yes) with write access
  • mount_ports - mount /usr/ports dir from base system to jail (read only). To the environment can build ports (and not interfere with each other vorkdir) - reassigned WRKDIRPREFIX parameters to alternate location. Or, make sure that applytpl params is set to 1 - then CBSD does this automatically via WRKDIRPREFIX=/tmp in /etc/make.conf of the jail
  • astart - Automatic start jail on the system boot
  • interface - Specify uplink for jail (on what interface create/remove IP) or prohibit.
  • applytpl - Automatically adjust the same settings of the jail (create /etc/hosts, change WRKDIRPREFIX, etc..)
  • floatresolv - Automatically adjust /etc/resolv.conf from jnameserver IP (settings from initenv/initenv-tui)
  • arch - Specify architecture of jail environment
  • vnet - Enable or disable VIMAGE feature

Other methods of creating jail:

jcreate command, part 2

		% cbsd jconstruct
		
		% cbsd jcreate jconf=/path/to/conf.jconf
		
description:

If DIALOG-based script jconstruct-tui for some reason did not come for create configuration of the new jails, you can use the script, dialogue like "question and answer" jconstruct

			% cbsd jconstruct
		

Which specifies the same question as the tui-version. Also, the configuration can avoid the formation without additional interactive utilities, for example building systems that automatically create the necessary jails.

Example of a standard configuration for jcreate might look like:

		jname="jail1";
		path="/usr/jails/jails/jail1";
		host_hostname="jail1.my.domain";
		ip4_addr="10.0.0.24/24";
		mount_devfs="1";
		allow_mount="1";
		allow_devfs="0";
		allow_nullfs="0";
		mount_fstab="/usr/jails/jails-fstab/fstab.jail1";
		arch="amd64";
		mkhostsfile="1";
		devfs_ruleset="4";
		ver="10.0";
		basename="";
		slavenode="0";
		baserw="0";
		basename="";
		mount_src="0";
		mount_obj="";
		mount_kernel="0";
		mount_ports="1";
		astart="1";
		data="/usr/jails/jails-data/jail1-data";
		vnet="0";
		applytpl="1";
		mdsize="0";
		rcconf="/usr/jails/jails-rcconf/rc.conf_jail1";
		floatresolv="1";
		exec_start="/bin/sh /etc/rc";
		exec_stop="/bin/sh /etc/rc.shutdown";
		
		exec_poststart="0";
		exec_poststop="0";
		exec_prestart="0";
		exec_prestop="0";
		
		exec_master_poststart="0";
		exec_master_poststop="0";
		exec_master_prestart="0";
		exec_master_prestop="0";
		interface="auto";
		jailskeldir="${sharedir}/jail-skel"
	

Note parameters arch and ver. Values ​​of them may be "i386", "amd64" for arch and "9.2", "10.0", "10.1" "11" and so on, depending on the version of the base, which would you prefer for jail. Values native in these parameters force CBSD always take the version and architecture that is running the node, which makes the floating version. So, if you use a template with arch="native", ver="native" and switch from FreeBSD 10.2 to 11.0, it will use the 11.0 version of the base. If you want to fix a specific version - specify version instead native.

If you want to create jail with installed some packages from pkg repository, in this configuration must have an pkglist pointing to a file with a list of packages, for example:

		pkglist="/tmp/newjail.txt";
	

The /tmp/newjail.txt file might look like:

	mc
	lynx
	nginx-devel
	lsof
	

cbsd jcreate remove file pointed by pkglist variables after jail create

Important:

When a new jail is created or obtaining from the repository, make it a rule ALWAYS change the user's password root in jail, even if you do not plan to run it ssh/ftp/rsh and similar services. If the jail is created with applytpl=0, by default /etc/{passwd,master.passwd,group} in the jail as the original "clean" files FreeBSD, so password of root user is empty. If jail created with applytpl=1 (it also refers to images from repository) $workdir/share/jail-skel files will be used as templates where root password is 'cbsd' in default CBSD installation. You can change default root password when new jail is created via edit of hash in skel master.passwd via:

% vipw -d ${workdir}/share/jail-skel/etc

commands, or specify alternative path to jail-skel dir in .jconf (jcreate tools) config

By default, the directory specified by jailskeldir will be used as source files that will be added (or they will be overwritten with standard files) into a jail automatically when applytpl=1. Accordingly, you can create any configuration templates and content environments that will be copied when creating a new jail.

Profiles for jail creation

Command: jcreate

			% cbsd jcreate
		

Profile for jail settings

Description

For example, you do not work with a huge number of nodes and environments, when such tools as Puppet, Ansible, SaltStack, Chef or your own implementation for generating .jconf file for jcreate can be useful. However, you often have to create environments and at the same time do not accept a number of default options. In this case we recommend that you use profiles to override the default values, or create your own profile

Take a look on content of $workdir/etc/defaults/jail-freebsd-default.conf file. It represents the settings that are used in jconstruct-tui by default.

Let's say you want to create a container always in baserw=1 mode (instead of baserw=0 by default), on interface lo0 (instead of auto, which selects the interface depending on the subnet jail) and you prefer a name cell instead of jail wih domain example.com and at the same time, container must be launched instantly after creation ( runasap="1" ) ( do not confuse with astart="1" for lauch on boot)

To do this, create in the directory (or copy from $workdir/etc/defaults) $workdir/etc/ file with same name, in which we can reassign the settings:

		% echo 'baserw="1"' > ~cbsd/etc/jail-freebsd-default.conf
		% echo 'interface="lo0"' >> ~cbsd/etc/jail-freebsd-default.conf
		% echo 'default_jailname="cell"' >> ~cbsd/etc/jail-freebsd-default.conf
		% echo 'default_domain="example.com"' >> ~cbsd/etc/jail-freebsd-default.conf
		% echo 'runasap="1"' >> ~cbsd/etc/jail-freebsd-default.conf
		

So we got file $workdir/etc/jail-freebsd-default.conf with follow content:

		baserw="1"
		interface="lo0"
		default_jailname="cell"
		default_domain="example.com"
		runasap="1"
		

It's all! Now we can run cbsd jconstruct-tui, where you need to do even fewer settings to create an environment!


You may want to create several profiles of your own. To do this, also create the files in the directory ~cbsd/etc/ by using in the name of file prefix: jail-freebsd-YOUR_PROFILE.conf

In this case, as the _invariant_ parameter within each profile, there must be a name for this profile in the variable jail_profile="default" and jail_active set to 1 (active profile). For example, let's create two profiles: baserw and lo0:


  • 		% echo 'jail_profile="baserw"' > ~cbsd/etc/jail-freebsd-baserw.conf
    		% echo 'jail_active="1"' >> ~cbsd/etc/jail-freebsd-baserw.conf
    		% echo 'baserw="1"' >> ~cbsd/etc/jail-freebsd-default.conf
    		

  • 		% echo 'jail_profile="lo"' > ~cbsd/etc/jail-freebsd-lo0.conf
    		% echo 'jail_active="1"' >> ~cbsd/etc/jail-freebsd-lo0.conf
    		% echo 'interface="lo0"' >> ~cbsd/etc/jail-freebsd-lo0.conf
    		

    Now you will see the profiles in the choice of possible options for the 'profile' menu at startup of cbsd jconstruct-tui

    Perhaps you want the profile lo0 to be the default, since you use it most often. You can override in $workdir/etc/jail-freebsd-default.conf profile by default via params default_profile:

    		% echo 'default_profile="lo"' > ~cbsd/etc/jail-freebsd-default.conf
    		

    The contents of the file ~cbsd/etc/jail-freebsd-default.conf will be:

    		default_profile="lo"
    		

    In this case, when you execute cbsd jconstruct-tui, you do not even need to choose a profile. There are very few actions to run a new environment.

    Profile for jail content

    Description

    Suppose you need to create a series of alike jails which all require a certain set of software and configuration,for example an environment with running an nginx http server with a custom index.html. One way to achieve this is to create a jail (eg jail1), perform all necessary adjustments and do a jexport. Afterwards whenever you need a new instance, run the command:

       % cbsd jimport jname=jail1 newjname=jail2
    		

    This creates an image on the basis of jail1 called jail2. Another way to get similar results is making use of jclone. However, it is not always convenient (especially in terms of keeping software up to date in the original jail). Another way is to to apply profiles and alternative skel-directories for applications we need to change on the fly when creating fresh jails.

    Example:

    1) Write jconf:

    If you want to create a non-interactive scripts, create a template from which to create new a jail. To do this, run cbsd jconstruct-tui and when asked if you want to create the jail immediately select not to. In this case, the command to be withdrawn for jcreate and the path to jconf - this is the configuration for which you are creating the jail. Alternatively, it is possible to write manually, for example:

    % mkdir /root/share
    % cat > /root/share/nginx.jconf << EOF
    # DO NOT EDIT THIS FILE. PLEASE USE INSTEAD:
    # cbsd jconfig jname=jail1
    relative_path="1";
    jname="jail1";
    path="/usr/jails/jails/jail1";
    host_hostname="jail1.my.domain";
    ip4_addr="DHCP";
    mount_devfs="1";
    allow_mount="1";
    allow_devfs="1";
    allow_nullfs="1";
    mount_fstab="/usr/jails/jails-fstab/fstab.jail1";
    arch="native";
    mkhostsfile="1";
    devfs_ruleset="4";
    ver="native";
    basename="";
    baserw="0";
    mount_src="0";
    mount_obj="0";
    mount_kernel="0";
    mount_ports="1";
    astart="1";
    data="/usr/jails/jails-data/jail1-data";
    vnet="0";
    applytpl="1";
    mdsize="0";
    rcconf="/usr/jails/jails-rcconf/rc.conf_jail1";
    floatresolv="1";
    zfs_snapsrc="";
    
    exec_poststart="0";
    exec_poststop="";
    exec_prestart="0";
    exec_prestop="0";
    
    exec_master_poststart="0";
    exec_master_poststop="0";
    exec_master_prestart="0";
    exec_master_prestop="0";
    pkg_bootstrap="1";
    pkglist="/root/share/pkglist.txt";
    
    with_img_helpers="";
    runasap="0";
    interface="auto";
    jailskeldir="/root/share/nginx-jail"
    jail_profile="default";
    # root password
    user_pw_root='rootpw'
    exec_start="/bin/sh /etc/rc"
    exec_stop="/bin/sh /etc/rc.shutdown"
    emulator="jail"
    EOF
    		

    Where the most important for us:

    • jail1 - jail name
    • user_pw_root - parameter sets the appropriate root password (you can generate password file instead in skel-dir)
    • ip4_addr="DHCP" - causes CBSD take the first free IP range of nodepool
    • jailskeldir="/root/share/nginx-jail" - alternative path to skel-dir, applied after the jail creation
    • pkglist="/root/share/pkglist.txt" - path to a file that defines a set of software in the cell when it is created
    • arch="native" - use/inherit 'hoster' architecture. Or specify: i386, amd64
    • ver="native" -use/inherit 'hoster' version. Or specify: 10.3, 11.1, 12

    2) Write pkglist.txt

    Just listed origin or packagename of software that want to get in the jail:

    % cat > /root/share/pkglist.txt << EOF
    www/nginx
    shells/bash
    EOF
    		

    3) skel-directories

    Custimize directories of additional files that are copied to the jail. Namely - prescribe nginx in rc.conf inside the jail and push into /usr/local/www/nginx/index.html some text to output:

    % cp -a /usr/local/cbsd/share/jail-skel /root/share/nginx-jail
    % mkdir -p /root/share/nginx-jail/usr/local/www/nginx
    % cat > /root/share/nginx-jail/usr/local/www/nginx/index.html << EOF
    <html>
       <body>
          <pre>
          It's been a hard day's night
          And I've been working like a dog
          It's been a hard day's night
          I should be sleeping like a log
          </pre>
       </body>
    </html>
    EOF
    % sysrc -f /root/share/nginx-jail/etc/rc.conf nginx_enable="YES"
    		

    4) the creation and launch

    Create a environment, run and check:

    % cbsd jcreate jconf=/root/share/nginx.jconf
    % cbsd jstart jail1
    % curl http://X.Y.N.M
          It's been a hard day's night
          And I've been working like a dog
          It's been a hard day's night
          I should be sleeping like a log
    		

    Profiles for jconstruct-tui

    If you want to manually create these jails, choosing when it should be necessary skel-directories or the default settings, create the directory $workdir/etc file named jail-freebsd-XXXX.conf with minimum content:

    jail_profile="XXX"
    		

    where XXX - name of the profile. Profile vnet in CBSD is created as an example, you see it in $workdir/etc/defaults/jail-freebsd-vnet.conf

    In turn, it overwrites the default values from the profile defaults: jail-freebsd-default.conf

    If you rarely use the default profile and want to CBSD default offered at your profile, change the value default_profile putting it in the name of your profile:

    echo 'default_profile="XXX"' > ~cbsd/etc/jail-freebsd-default.conf
    		
  • Please note that the files in the $workdir/etc/defaults can not be edited - as well as in the /etc/defaults. If you want to override the default values, copy the directory $workdir/etc/defaults file with the same name in the directory $workdir/etc and change

    Jail config

    jconfig, jset commands

    			% cbsd jconfig
    			% cbsd jset
    		

    Descriptions:

    Cconfiguration parameters jail

    Each CBSD jail stores the settings in SQLite3 database. In addition, $workdir/jails-fstab/ may have fstab files (see below). To change the settings of jail can serve cbsd jconfig command, which is runs the TUI menu to change basic settings:

    Jails IP address

    IP addresses that are bound to the jail sets in ip4_addr parameter. As an IP may serve as IPv4, and the IPv6 address. When starting and stopping jail, working with IP may take place in two modes:

    • automatic on-the-fly creation of IP addresses for the jail at the time of launch + automatic removal from the interface IP when stopping
    • the use of previously initialized IP addresses.

    When for jails assigned to more than one address, they should be listed separated by commas without spaces. IP can include network prefix specified through IP/prefix. By default, aliases created with the prefix /32, that may not be appropriate if the jail uses a separate subnet from the network server — in this case, the correct /prefix is needed.

    The parameter that controls this behavior pointed by interface parameter. When interface=0, jstart and jstop will not be called ifconfig alias and ifconfig -alias, respectively. If its value is auto or name of network nics, this command will be executed:

    				% ifconfig interface ips alias
    			

    and when jail stop:

    				% ifconfig interface ips -alias
    		

    Be careful with this option, if you have only one IP for server that is used and this IP is assigned to the same jail, when stopping jail, ip address of the server will be removed automatically that will make the server unavailable. In this case, you need to use:

    			interface=0
    		

    For example, to run the configuration tool for the jail1, run:

    			% cbsd jconfig jname=jail1
    		

    Mounting File Systems in jail

    Each jail can have your fstab file, which lists the file systems that are mounted jail is startup. System records (they are managed by CBSD and edit this file inappropriately) are located in the file $workdir/jails-fstab/fstab.name and the same syntax to format the file system /etc/fstab with the exception that as a mount point is the path relative to the root jail, not the master system.

    For user entries, you can use the file in the same directory with the extension .local. For example, if you want to make between jail1 and jail2 a shared directory (via nullfs), which physically located in the master node (e.g.: /usr/home/sharefs), the files $workdir/jails-fstab/fstab.jail1.local and $workdir/jails-fstab/fstab.jail2.local should have:

    			/usr/home/sharefs /usr/home/sharefs nullfs rw 0 0
    		

    If you want to mount tmpfs to /tmp dir in jail1 (those actually in /usr/jails/jails/jail1/tmp), then the entry in the $workdir/jails-fstab/fstab.jail1.local should have:

    			tmpfs /tmp tmpfs rw 0 0
    		

    both entries perform the same action as commands:

    			mount -t nullfs -o rw /usr/home/sharefs /jailroot/usr/home/sharefs
    			mount -t tmpfs -o rw tmpfs /jailroot/tmp
    		

    If you want to mount into jail2 a directory from jail1, jail1 path should point to the directory containing the data jail1 (and their mount points ${workdir}/jails/jail1). For example entry $workdir/jails-fstab/fstab.jail2.local:

    			/usr/jails/jails-data/jail1-data/usr/local/www /usr/local/www nullfs ro 0 0
    		

    Make the shared directory /usr/local/www between jail1 and jail2, but it will jail2 in read-only mode

    There are flaws when mounting via mount -t /type/ does not work or mounting must be done in a non-standard way. In this case, use the 'external' reserved word, where the 'source' field will point to your external script, the output of which should be a mounted directory.

    An example of such an entry in fstab.local:

    		/root/ntfs-3g.sh        /usr/home/web/downloads external rw 0 0
    		

    and an example working script /root/ntfs-3g.sh for this entry: ntfs-3g.sh, which mounts an external USB drive directly into the container.

    Presentation of ZFS file systems in jail

    If you want to attach separate ZFS filesystems in jail (ie, want to be able to perform in jail zfs mount), ZFS dataset must list in $workdir/jails-fstab/fstab.$jname.local file with specifying the keyword zfs in FStype field. For example, if you want to present the file system ZFS: zroot/jail1_webfs for jail jail1 $workdir/jails-fstab/fstab.jail1.local must have:

    			zroot/jail1_webfs /usr/home/web zfs rw 0 0
    		

    Note: mount point (/usr/home/web in this example) is not important

    Note: jail must have allow_zfs paramaters set to 1, what can be done via cbsd jconfig jname=$jname

    In fact, it makes CBSD execute commands:

    			% zfs set jailed=on $FS
    			% zfs jail $jname $FS
    		

    when jail started and

    			% zfs set jailed=off $FS
    			% zfs unjail $jname $FS
    		

    when stoped.

    Change settings via the jset

    Another possibility is to change certain parameters of the jail — use the command cbsd jset. Full list of possible arguments can be accessed through --help:

    			% cbsd jget --help
    		

    For example, to change ip of jail1:

    			% cbsd jset jname=jail1 ip4_addr="10.0.0.20/24,192.168.0.20/24" 
    		

    cbsd jconfig jname=jail2

    cbsd jset

    Custom scripts for starting and stopping action on jail

    You can write your own scripts to be executed within the jail and in the master host on startup and shutdown of the environment. For this, the system directory of jail ( $workdir/jails-system/$jname/ ) have the following directories:

    • master_poststart.d - scripts for executing in the master host after jail starts (Be careful, because the scripts are is not jailed)
    • master_poststop.d - scripts for executiong in the master host after jail stops (Be careful, because the scripts are is not jailed)
    • master_prestart.d - scripts for executing in master host before jail starts (Be careful, because the scripts are is not jailed)
    • master_prestop.d - scripts for execution in master host after jail stops (Be careful, because the scripts are is not jailed)
    • start.d - scripts for execution within jail when jail is starts. Analog of parameter exec.start from original jail.conf
    • stop.d - scripts for execution within jail when jail is starts. Analog of parameter exec.stop from original jail.conf
    • remove.d - (since CBSD 11.0.10) scripts for execution on remove command

    Write scripts to the master_\* directories can be useful if at the start-stop jail you need to perform some action is not associated with content of environment - for example, create a ZFS snapshot, put rules in IPFW and etc.

    In scripts, you can use CBSD variables, such as $jname, $path, $data, $ip4_addr, for example, by placing a script (with execute permission) in /usr/jails/jails-system/jail1/master_poststart.d/notify.sh:

    		#!/bin/sh
    		echo "Jail $jname started with $ip4_addr IP and placed on $path path" | mail -s "$jname started" root@example.net
    		

    You will receive notification upon startup cells by email: root@example.net

    The functionality of execution custom scripts and the availability of variables in environments can play a big role in the integration of CBSD and external applications, such as Consul

    As an example of use, see the article Example of using CBSD/bhyve and ISC-DHCPD (Serve IP address in bhyve through pre/post hooks)

    As an example of use, see the article Example of using CBSD/jail and Consul (Register/unregister jail's via pre/post hooks, in Consul)

    Starting and stoping jail

    jstart, jrestart, jorder commands

    			% cbsd jstart jname=jail1 jail2
    			% cbsd jstart jail1 jail2 ... jailX
    
    			% cbsd jstop jname=jail1 jail2
    			% cbsd jstop jail1 jail2 ... jailX
    
    			% cbsd jrestart
    			% cbsd jorder
    		
    Description:

    Running jails occurs at startup CBSD/server automatically if parameter astart (auto-start) corresponding jail set to 1. You can change this setting through cbsd jconfig or cbsd jset. When you stop the server or service cbsdd, all running jails will be stopped automatically.

    Starting the jail manually by the command:

    			% cbsd jstart jname=jail1
    		

    or

    			% cbsd jstart jail1
    		

    or

    			% cbsd jstart jail1 jail2 jail3 ..
    		

    (to launch multiple jails by one command)

    When ( You can change this behavior in cbsd initenv-tui ) parallel=0 and you try to launch/stoping a few jails, start/stop will be held consecutively. It is not always good, as any error triggered inside jails in the rc-scripts, which leads to a pause, can block start/stop next jails. In the case when parallel have a non-zero value, each next jails will be launched/stopped in N seconds after the previous one, where N — parallel value. After this timeout, no matter whether the previous jail o start fully and next jail will be started.

    To stoping jail jstop must be used, with the same syntax and behavior:

    			% cbsd jstop jname=jail1
    		

    or

    			% cbsd jstop jail1
    		

    or

    			% cbsd jstop jail1 jail2 jail3 ..
    		

    (to stopping multiple jails by one command)

    If the argument of the jstart/jstop/jrestart command is missing, will displays the corresponding list of all running or stopped jail for interactive selection

    When the jail is started, it creates a lock file, a sign that the jail working via file ${jailsysdir}/${jname}/locked which records the name of the node. This feature is used when this jail presented several nodes, these jail are in the DFS (NFS, glusterfs, etc.) and any node is able to run it. This lock ensures that the presence of the same jail at the second node, it will not be launched.

    With a large number of jails (particularly databases, with services such as MySQL, redis, cassandra, etc.), it should be borne in mind that the low value parallel (for example, less than 5 seconds) can generate a very intensive storage I/O load that can increase the amount of start-up time of all jails, than if they were run sequentially or higher timeout.

    Additionally, when a shutdown command is running on the server with lots of jails/services that should be taken into consideration low timeout that defaults to perform rc.shutdown sequence. In this regard, the process init(8) can interrupt the rc-scripts on this timeout, resulting in abnormal shutdown jails. In this case, the database can lead to nonconservation or damage data. To avoid this, /etc/rc.conf master system should be adjusted parameter rcshutdown_timeout to a more reasonable value (default: 90 seconds)

    In the absence of rcshutdown_timeout in the system /etc/rc.conf, cbsd initenv will put this option in its sole discretion automatically.

    Also, keep in mind that when using the zfs features ($workdir/nc.inventory, the parameter zfsfeat=1), and the file system ZFS, at jstop, file system of jails will be unmounted and those catalog $workdir/jails-data/jail1-data will be empty. If in such a case when jail data is requires without running it, by the command zfs list You can see the name appropriate file system and run zfs mount fs.

    Writing scripts executed on starting and stopping jail

    Please read details on the Jail config

    Priority launch of jails

    In CBSD you can specify the sequence in which the jails will be started. Please read details on the jail starting order

    Priority launch environments

    Commands: jorder, jorder-tui, border, border-tui

    % cbsd jls display=jname,b_order
    		

    Description:

    Sometimes a jail/vm will require an already running service from another machine/jail before it is launched. Examples might be a database SQL or a running LDAP.

    In this case, you can edit Boot Order (b_order in jail/vm settins)

    By default, all environments are created with b_order set to 10 (which is defined in the profile and can be changed)

    If you need jail2 to be launched before jail1, its value for b_order must be set to a lower one than that of the second jail.

    You can display the current configuration with the command jls:

    % cbsd jls display=jname,b_order
    JNAME    B_ORDER
    firefly  3
    jail1    10
    kde4     1
    kdeold   9
    spicy2   2
    test     2
    	

    This sample configuration will launch kde4 first while jail1 will come up last

    You can also display the sequence using the command jorder

    % cbsd jorder
    	

    In what sequence jail printer - in this sequence they will run

    To edit a launch sequence use jset or the TUI-editor jorder-tui

    To edit bhyve priority, use the commands border or border-tui

    Jail removal:

    jremove command

    			% cbsd jremove
    		

    Description:

    Removal of jail mentions all files anyway connected with a jail:

    • a) fstab for current jail
    • b) data directory or ZFS filesystem with jail data
    • c) statistics and description for jail
    • d) snapshots

    in a case when jremove executed for jail in status On, it will be automatically stopped.

    Example:
    			% cbsd jremove jail1
    		

    Jail renaming:

    jrename command

    			% cbsd jrename
    		

    Description:

    Renames a jail and modifies all relevant according to the new name. The command can only be executed on stopped jails.

    jrename has the two following mandatory parameters:

    • old — old name of the jail
    • new — new name of the jail

    Optional parameters:

    • host_hostname — FQDN, new full hostname of jail
    • ip4_addr — new IP address of the jail (if multiple IPs, separated by a comma with no spaces)

    Example (renaming jail jail1 into jail50 with a FQDN and ip addresses change):

    			% cbsd jrename old=jail1 new=jail50 host_hostname=jail50.my.domain ip4_addr=192.168.0.5/24
    		

    Jail upgrade

    Warning

    Description:

    Upgrade procedure jail always carries certain risks in the form of disruption of service, therefore, make it a rule to always create backups of the jail. If you are running on ZFS file system, you can use the command cbsd jsnapshot for a frozen state of the jail before the start of work. For example:

    			% cbsd jsnapshot mode=create jname=jail1 snapname=before_update
    		

    and the end of work, if all ok, remove snap through:

    			% cbsd jsnapshot mode=destroy jname=jail1 snapname=before_update
    		

    If the file system is UFS, you can create an image of the jail through:

    			% cbsd jexport jname=jail1
    		

    And in the case of success, remove the image jail1.img from $workdir/export directory. An upgrade jail CBSD, we assume only update files base FreeBSD. Upgrade procedure 3rd-party software in jails similar to upgrade in the non-jail system. You can update the database as between different versions of the system (eg from version 9.2 to FreeBSD 9.3 or FreeBSD 9.3 -> FreeBSD 10.1), and update the files within the same version. It should be remembered that the jails have two modes — baserw=1, when the base part of each jail — have their own copy located in the directory $workdir/jails-data/$jail, and baserw=0, when one and the same base dir ${workdir}/basejail/base_\*_\*_ver mounted to all jails.

    jupgrade command

    			% cbsd jupgrade
    			% cbsd jconfig
    		

    Upgrading the basejail

    			% cbsd jset
    		

    This time, the idea is to set the basejail version of the target jail to that of the desired FreeBSD release. We achieve this by successively: shutting down the jail, setting the appropriate version tag, starting the jail. In case we specified a version for which we don't already have the base, it will be downloaded at jail startup.

                            % cbsd jstop XXX
                            % cbsd jset jname=XXX ver=11.2
                            % cbsd jstart XXX
                    

    Upgrading via freebsd-update

    freebsd-update can be used without a reference to CBSD. Note that the utility works with the official FreeBSD resource and configure it to the repository CBSD is impossible, those only one source.

    Update files for base jail through freebsd-update can via specified key -b path to directory which used by CBSD when mounting jail baserw=0. For example, if your cbsd $workdir - /usr/jails, then for amd64 and version 10.0 is the path to base: /usr/jails/basejail/base_amd64_amd64_10.0. This directory should be updated:

    % freebsd-update -b /usr/jails/basejail/base_amd64_amd64_10.0 fetch
    % freebsd-update -b /usr/jails/basejail/base_amd64_amd64_10.0 install
    		

    If you do not have baserw=1 jail, then its the end. If baserw=1 jails exists, to update them there are two possibilities:

    • As in the case above, for each jail cause freebsd-update and specify the path to data directory of the jail ( $workdir/jails-data/$jname )
    • Or, after basejail upgrade as described above, execute:
      cbsd jupgrade jname=XXXX
      			
      for each baserw=1 jail - this command reshapes base ( $workdir/jails-data/$jname ) of the jail from the $workdir/basejail/* files

    Upgrading baserw=0 jails between different versions of the bases via CBSD

    Option with the upgrade jail in mode baserw=0 to a new version base is the easiest is to change the version in the configurator

    			% cbsd jconfig jname=XXX
    		

    Since in this case only changes the source directory that link on start jail. Example of updating jail from 9.2 to 10.0 version.

    Baseline: there x86-64 system with a base of FreeBSD 9.2 and jail jail1 on this base, upgrade to 10.0.

    When jail is stopped, we go in the configurator via

    			% cbsd jconfig jname=jail1
    		

    and change ver parameter to 10.0, then save it via "Commit". Or, as in the screenshot — do change through cbsd jset

    On the next run, the jails will mount base 10.0 (on screenshot the base in the system not found and CBSD has invited her to download). Of course, with such a upgrade to a new major version, after this operation you need to rebuild the software in the jail or update it through pkg — operation is highly desirable because library system changed. As a minimum, to identify this fact we can use the libchk utils.

    Upgrading baserw=1 jails between different versions of the base via CBSD

    Upgrade of baserw=1 takes a different scenario, since on this operation CBSD will overwrite the old base system files to the new in jail data directory $workdir/jails-data/$jname.

    Baseline: we need uprgade jail jail1 baserw=1 mode (in-law, its root path PATH points in jls pointed to jails-data, rather than jail directory) from 9.2 to 10.0 version.

    Verify through the file utility, which reads ELF table that file /bin/sh of jails belongs version 9x. And in the same way, check the version of the file after the upgrade:

    			% file -s /usr/jails/jails-data/jail1-data/bin/sh
    			/usr/jails/jails-data/jail1-data/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), for FreeBSD 9.2, stripped
    		

    When stopped jails perform change version through cbsd jconfig or cbsd jset, then perform the upgrade procedure:

    			% cbsd jupgrade jname=jail1
    		

    This operation causes the system to overwrite all the files in the base jail from your base original directory $workdir/basejails/base_\*_\*_ver

    Upgrading baserw=0,1 jail in one version, switch to stable=1

    There are cases when you need to upgrade the files in one version, for example, base 10.0 to 10.0-p1. For jails who have mounted base through nullfs (baserw=0), simply re-download the $workdir/basejail/base_\*_\*_ver directory to more recent version. For this you can use the command:

    			% cbsd repo action=get sources=base mode=upgrade
    		

    - flags mode=upgrade permits to CBSD overwrite this directory with new files, if you already have a version of the base for this version. Or, you can build a more recent version of the base, using Building and upgrading bases Also, you can go with the base version with a RELEASE to STABLE (in this case, the name of the base directory will not X.Y, just X. Ie, instead base_\*_\*_9.2 will be used base_\*_\*_9 directory. For this you need in configurator of jail (cbsd jconfig) change the parameter stable=0 to stable=1 (either through cbsd initenv-tui mode is set STABLE branches globally), and do not forget to add stable=1 flags in repo command (if not set globally)

    			% cbsd repo action=get sources=base mode=upgrade stable=1
    		

    Similar rules for jails with baserw=1, it is only necessary to remember after update in CBSD to start the update process files by cbsd jupgrade

    In addition, provided that you have a base system in the corresponding version (or you jail migrated to another server where there is a base more recent), when you start the jail with baserw=1, CBSD can automatically check for more recent files for this version and show information message "You have a more recent version of the base in ...":

    Update of configuration files in jail, etcupdate/mergemaster

    Updating binary files is half the trouble) You must also synchronize the configuration files. See how CBSD may be useful here: etcupdate helper

    Working with NAT

    natcfg, naton, natoff commands

    			% cbsd natcfg
    			% cbsd naton
    			% cbsd natoff
    		

    Description:

    Jails do not always require external IP, or, for security reasons, a number of services need to deploy on private IPs, so they were not available from the Internet. Thus, the jails may be needed for Internet access.

    In this case the NAT translating the private IP address of the jails to external IP of the server. CBSD functional has a configuration template NAT rules for translating of private networks RFC1918. To do this, this command as the first step is required:

    			% cbsd natcfg
    		

    for selecting the appropriate framework for which the configuration will be loaded NAT: pf, ipfw and ipfilter. The names of the relevant frameworks

    Attention! When you configure this, system file /boot/loader.conf nodes will be modified to load the appropriate modules.

    Selection framework and IP for NAT alias executed when you first start cbsd initenv, can later be reconfigured through cbsd initenv-tui To natip changed in force, you must run cbsd natcfg and cbsd naton again. Currently, the cbsd configuration NAT limited to the creation of rules for translating private networks. If you need to get something more from simple NAT rule, you can edit the rules file created manually in the directory $workdir/etc/ in files:

    • pfnat.conf, when PF is used
    • ipfw.conf, when IPFW is used, or
    • ipfilter.conf, wnen using IPNAT from IPFilter

    Note:

    If nodeip (IP of nodes), he is within RFC1918 networks for the subnet broadcast NAT rule will not be created. To disable nat control by CBSD, use the follow command:

    			% cbsd natoff
    		

    Attention! Be careful, if you activate NAT through ipfw. This rule loads the module ipfw.ko, the settings of which are prohibited by default all packets. If you spend this operation remotely, you can through /boot/loader.conf enable ipfw rule, by default allowing all packets:

    ipfw_load="YES"
    net.inet.ip.fw.default_to_accept=1
    		

    Example. Start the configurator and activate the NAT through pf, with aliasing-IP as 192.168.1.2, previously initialized in /etc/rc.conf:

    % cbsd natcfg
    Configure NAT for RFC1918 Network?
    [yes(1) or no(0)]
    yes
    Set IP address as the aliasing NAT address, e.g: 192.168.1.2
    
    Which NAT framework do you want to use: [pf]
    (type FW name, eg pf,ipfw,ipfilter or "exit" for break)
    
    % cbsd naton
    No ALTQ support in kernel
    ALTQ related functions disabled
    No ALTQ support in kernel
    ALTQ related functions disabled
    pfctl: pf already enabled
    		

    Jails list

    jls command

    			% cbsd jls
    		
    Description:

    Show the lists of environment on a local node or for all nodes. You can adjust the displayed fields through the argument display. If display is not specified, the value takes from $workdir/etc/defaults/jls.conf file, which you can change at its discretion via $workdir/etc/jls.conf. To hide the header, use the argument header=0.

    All possible options for the sample described in the $workdir/share/jail-arg file

    In addition to the basic CBSD values, you can add your own fields and values for output with the jls command. More on this: environment custom facts

      JIDJail ID
      JNAMEenvironment name
      IP4_ADDRlist of assigned IP addresses (IPv4,IPv6)
      HOST_HOSTNAMEenvironment FQDN
      PATHroot filesystem for jail
      STATUSOn (running), Off (stoped), Unregister (missing in the SQL database)

    For vnet-based environments with a virtual stack, the REAL_IP4 field for displaying the actual IP addresses inside the environment may also be useful to you.

    Note: Jail in the unregister status may be insert to SQL database via command: cbsd jregister. If remote nodes are added to the local server, you can display all env in the farm through:

    			cbsd jls alljails=1
    		

    or

    			cbsd jls alljails=1 shownode=1
    		

    for output with node name where jail are hosted. In the output from cbsd jls alljails, it is possible to see only active jails (in status On)

    Another example:

    			% cbsd jls display=jname,ip4_addr,vnet,real_ip4,astart
    			% cbsd jls header=0 display=jname,mycustom1,mycustom2,status
    		

    Command execution in jail

    command: jexec

    			% cbsd jexec
    		

    Description:

    You can run the command in the container from the master environment via jexec command

    The required parameter is the name of the jail via jname=. Everything that comes after - is directly a command and arguments that will be launched in the jail

    You can run a command in one container or simultaneously in several. To do this, use the jname= argument mask of the jails name in which the command will be executed.

    For example, a mask of the form jname='test*jail' will execute the command in containers with the same name as test1jail, test2jail and so on. If you want to run the command at once in all containers of this node, use jname='*'

    Be careful when launching long commands or actions that can lead to interactive dialogues. You can get the output of the last entries of the active log files through the sending of the SIGINFO command by pressing the Ctrl + 't' keys - this functionality will allow you to see and understand at what stage the command is executed in a container.

    You will see the execution result on stdout, while the auxiliary messages are on stderr, respectively, if CBSD messages prevent you, use redirection stderr to /dev/null

    Example:

    			% cbsd jexec jname='jail*' pkg update -f
    			% cbsd jexec jname='*' pkg update -y
    			% cbsd jexec jname='*' pkg clean -ya
    		


    Multiple command execution:

    Jail login

    jlogin command

    			% cbsd jlogin
    		

    Description:

    Performs a login to the jail as root user. If the jail isn't present on the local node, but rather on one of the remote nodes, jlogin will attempt to login over ssh.

    If no jail is specified, a list of all known online jails in the farm is displayed (provided remote hosts were added).

    If you remotely connect to a jail and node which have tmux installed, tmux will be launched at login and the session is given the name taken from the server's nodename (taken from jlogin).

    Additional sessions will automatically join the tmux session through a tmux-attach. When the last connection is closed the tmux session ends (you may detach via Ctrl + b , d to keep it running).

    Should you prefer NOT to use tmux on jlogin, copy ${workdir}/defaults/jlogin.conf to ${workdir}/etc/jlogin.conf and set tmux_login to 0.

    In order to deactivate "try to login?" when logging in to remote nodes, set always_rlogin to 1 in your ${workdir}/etc/jlogin.conf.

    Example:

    			% cbsd jlogin kde4
    		

    Starting with the version of CBSD 11.1.2, you can customize the command, redefining the action on you more suitable

    This is achieved through the configuration file blogin.conf and the parameter login_cmd.

    The file can be placed for the individual environment in the directory $workdir/jails-system/$jname/etc, and globally, overwriting the value from $workdir/etc/defaults/blogin.conf. To do this, create a file with your configuration in the directory $workdir/etc/

    With a custom call, you can use CBSD variables - for this or that environment

    For example, if you want instead of the standard behavior, when the blogin lauched ssh client, the file $workdir/etc/blogin.conf can look like this:

    		login_cmd="/usr/bin/ssh your_user@${ipv4_first}"
    		

    Work with jail parameters

    jget, jset commands

    			% cbsd jget
    		
    			% cbsd jset
    		

    Description:

    Example:

    Jail cloning

    jclone command

    			% cbsd jclone
    		
    			% cbsd jrclone
    		

    Description:

    Creates a clone of a given jail. The command requires the name of the original jail (passed with old) as well as a name for the clone, specified with new and a FQDN (hostname) with host_hostname. Optionally a new IP Address can be given with the ip4_addr parameter (multiple IPs need to be separated by commas).

    Attention: when you are using a ZFS-based system, CBSD will use ZFS cloning by default!

    ZFS cloning features is ultra fast operation (thanks to Copy-on-write), but imposes some restrictions - you will be dependent on the parent snapshot.

    You can control this behaviour via clone_method= argument or, to set it globally, use rclone.conf and bclone.conf to overwrite settings from 'auto' to 'rsync':

    		% echo 'clone_method="rsync"' > ~cbsd/etc/rclone.conf
    		% echo 'clone_method="rsync"' > ~cbsd/etc/bclone.conf
    		

    Custom data cloning jail method

    You can create your own data cloning method using script(s) in the directory $workdir/jail-system/$jname/clone-local.d.

    For example, creating a script to the $workdir/jail-system/$jname/clone-local.d/ directory with the name myclone.sh and the contents of the form:

    			#!/bin/sh
    			echo "My custom clone for: $jname -> $newjname
    			echo "${data} -> ${newdata}"
    			/bin/cp -Ra ${data} ${newdata}
    		

    You will force CBSD to execute your script as a data cloning method.

    Attention: The script must always end with 0 exit code when success

    Please note that standard variables are available in the script CBSD variables. Additionally, these variables are exported: $newjname,$newdata

    Why it is needed: if you use NAS or NFS as a storage and have access there, you can clone the data using NAS or directly on the NFS server, which will significantly increase the cloning speed and reduce unnecessary traffic. See: Issue #373

    Example:

    cloning jail jail2 to jail3 with changes ip address ip4_addr and name of hosts host_hostname:

    			% cbsd jclone old=jail2 new=jail3 host_hostname=jail3.my.domain ip4_addr=10.0.0.22/24
    		

    Jails snapshot (zfs-only)

    jsnapshot command

    			% cbsd jsnapshot
    		
    Description:

    A variety of operations on jails become available through jsnapshot when the node is kept on a ZFS filesystem and zfsfeat is set to 1 in $workdir/nc.inventory. The mode parameter is used to specify which of the following actions are taken:

    • list — show a list of existing snapshots for the selected jail
    • create — create a snapshot
    • destroy — delete a specific snapshot of the jail
    • destroyall — remove all of the jail's snapshots
    • clone — clone an existing snapshot into new jail
    • rollback — rollback the selected jail to current snapshot state

    Additional arguments:

    • jname — specifies the jail upon which the action will be performed
    • snapname — gives the snapshot a name

    It should be kept in mind that snapshots follow a tree structure. This means, if you created a series of snapshots : 1,2,3,4 and roll back to snapshot 2, then the snapshots 3 and 4 will be lost, since from the point of snapshot 2 they did not exist yet. Also use a unique name for a snapshot at creation. You can specify snapname=gettimeofday. In this case, the system automatically sets the current timestamp as the name for the new snapshot. When listing snapshots, you can use modifiers to customize output fields with arguments to display=

    Example:

    create snapshot named gromozeka for jail1 jail:

    			% cbsd jsnapshot mode=create jname=jail1 snapname=gromozeka
    		

    create snapshot named zelepuka for jail1 jail:

    			% cbsd jsnapshot mode=create jname=jail1 snapname=zelepuka
    		

    Run jail1 and stop after some modification:

    			% cbsd jstart jail1
    			..
    			% cbsd jexec jname=jail1 cp /bin/date /root
    			% cbsd jexec jname=jail1 file -s /root/date
    			/root/date: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), for FreeBSD 9.0 (900506), stripped
    			% cbsd jstop jail1
    			..
    		

    Rollback jail1 to snapshot zelepuka state:

    			% cbsd jsnapshot mode=rollback snapname=zelepuka jname=jail1
    			% cbsd jstart jail1
    			...
    			% cbsd jexec jname=jail1 file -s /root/date
    			/root/date: ERROR: cannot open `/root/date' (No such file or directory)
    		

    Jail export

    jexport command

    			% cbsd jexport
    		

    Description:

    Attention: command execution allows on jail in status On. However it is necessary to remember (especially for jail with databases) when you import such jails, with a high probability it is possible to got problems with inconsistency filesystem in jails, old .pid files that can break work of the imported jails

    Export jail into file (*.img). In jname arguments you can set jail for export. img-file stored in $workdir/export directory. Original jail after exports is not modified

    You can control compress level via compress arguments

    CBSD use xz(1), tools for compress images and you can learn in man page about compress diffrence between compress level.

    By default CBSD use compress=6. You can disable compression with compress=0

    Example (export mysqljail jail to $workdir/export/mysqljail.img):

    			% cbsd jexport jname=mysqljail
    		

    Exceptions for jexport

    There are situations where you do not want to include one or another information from the container in the exported image. For example, if you are exporting a working container having a mounted port of ports in /usr/ports.

    In this case, you can help the jexport_exclude parameter, which can be specified globally in the jexport.conf configuration file (just copy default jexport.conf from /usr/local/cbsd/etc/defaults/ to ~cbsd/etc/ and adjust the value for jexport_exclude

    If you want to specify alternative exclude for a specific Jail, copy this file to the ~cbsd/jails-system/$jname/etc / directory. In this case, when you make an jexport call, these exceptions will be applied for $jname container only.

    Finally, you can do exclude without a configuration file at all, just listing all exceptions (space separated) as the jexport_exclude= parameter when calling the jexport script, for example:

    			cbsd jexport jname=jail1 jexport_exclude="/var/run/* /usr/ports"
    		

    In this example, ports tree directory will not copy to image: since this data can always be mounted from the host. And also, eliminates the presence of dead .pid files that can affect the correctness of the launch of the container's services after importing.

    Exceptions are indicated relative to the root directory of the container. So, if you want to migrate to another host jail1 container configuration using jexport/jimport and do not want to copy any of its data, simply specify / as an exception:

    			cbsd jexport jname=jail1 jexport_exclude="/"
    		

    Jail import

    jimport command

    			% cbsd jimport
    		
    Description:

    Import jail from (*.img) file.

    In jname argument you can set full path to file or just filename (without .img extension) if file placed in $workdir/import directory. img-file after import is not modified. In addition, you can import an image of a jail in an alternate name (for example, you want to deploy the image of the jail1 from jail1.img, however, your system already has a jail of the same name). In this case, an argument newjname can be used.

    Example:

    import jail from /tmp/mysqljail.img file:

    			% cbsd jimport jname=/tmp/mysqljail.img
    		

    Example:

    import the image jail2.img placed in $workdir/import with a new name jail5:

    			% cbsd jimport jname=jail2 newjname=jail5 
    		

    Backup and file replication for jail:

    jbackup, j2slave commands

    			% cbsd jbackup
    		

    Description:

    Example:

    Jail description

    jdescr command

    			% cbsd jdescr
    		
    Description

    Each jail can have an optional description/summary. With this it is possible to generate dynamic documentation on admin pages. Using cbsd jdescr by itself outputs a description of all local jails.

    Command

    			% cbsd jdescr mode=update jname=jname
    		

    runs nvi in edit mode and opens the description of the jail specified by jname. If you prefer using a different editor such (such as vim or mcedit it can easily be done by passing your choice with the editor variable. Descriptions for each jail are stored in a text file residing at $workdir/jails-system/$jname/descr, where jname is the name of the jail. When a jail is moved or copied with jcoldmigrate, jclone, jimport/export operations, the description will remain the same.

    Example (running mcedit to modify the kde4 jail using russian UTF8 charset):

    			% setenv LANG ru_RU.UTF-8
    			% cbsd jdescr jname=kde4 editor=mcedit mode=update
    		

    It is advisable to maintain meaningful descriptions as your collection of jails grows. In addition it can be quite convenient to aggregate this information in order to create dashboards and build jail maps. One example of how this could look is the following simple script: jmap2html.sh.html which generates an overview in the form of a HTML page dashboard sample

    jail cold migration

    jcoldmigrate command

    			% cbsd jcoldmigrate
    		
    Description:

    cbsd jcoldmigrate does cold (with stoping) migration jail from one node to another. Argmument node point for destination node. Preliminary, RSA/DSA key for remote node must be added via cbsd node mode=add operation. Also, on a remote node service rsyncd should work (cbsdrsync).

    By default, the jail status on a new node is inherited — if the jail worked, it also will be automatically started on a new node. If the jail didn't work — remains in Off status. To operate, what status should be on a remote jail not is dependent on a condition on the original, it is possible through parameter start.

    Jail on the source, after successful jcoldmigrate will be stoped and switch to Slave status (unregister). In rc.conf file of jail on the destination node there will be a record where this jail came from.

    Notes: For jcoldmigrate, the next action is performed (in process they pass automatically and aren't visible)

    • copy configuration files to remote node, status of jail set to Slave on remote node (cbsd j2prepare)
    • exec rsync, which does a full copy of the directory with data to remote node (cbsd j2slave)
    • stop jails (if running) on the source node (cbsd j2slave)
    • one more rsync job, for synchronization of those files which be modified
    • switch jail status on source to Slave (cbsd jswmode)
    • switch jail status to master on the remote node (cbsd rexe + jswmode)
    • If jail was running — start jail on remode node (cbsd rexe + cbsd jstart)

    Example

    make cold migration for jail amp123 to netsnap node):

    			% cbsd jcoldmigrate node=netsnap jname=amp123
    	

    on the destionation node anything isn't present now:

    from node cbuilder64 migrate jail amp123 to netsnap:

    jail amp123 on netsnap was started automatically:

    Jail limits control

    jrctl, jrctl-tui commands

    			% cbsd jrctl
    			% cbsd jrctl-tui
    		
    Description:

    CBSD supports many of FreeBSD's mechanisms for enforcing limits on a jail's resource usage, such as: zfs quota, rctl(8) и cpuset

    CPU and Memory limits

    The most popular limitation parameters in the installation are the limits on the consumption of CPU and RAM.

    You can limit the CPU consumption via RACCT (cputime= and pcpu= parameters), or use a fixed number of cores allocation (via cpuset) via the cpu= parameter.

    In our research and testing, cpu= is the most practical and demanded solution.

    The cpu= parameter specifies how many cores the container can recycle. You can immediately create a container limited, for example, to one core:

    		% cbsd jcreate jname=myjail cpu=1 runasap=1
    		

    Besides limiting kernels, you may want to limit memory consumption via vmemoryuse=:

    		% cbsd jcreate jname=myjail cpu=1 vmemoryuse=1g runasap=1
    		

    Or, if you prefer to create environments through the CBSDfile, it might look like this:

    jail_test2()
    {
            ip4_addr="DHCP"
            host_hostname="test2.my.domain"
            pkg_bootstrap="0"
    
            # LIMIT/QUOTAs:
            fsquota="10g"           # set ZFS dataset quota
            cpu="1"                 # only one vCPU allowed!
            vmemoryuse="512m"       # only 512MB RAM allowed!
    }
    		

    File quotas

    Floating file quotas are only possible for jails residing on ZFS-file system. The quota is regulated through the fsquota= parameter. For systems stored on UFS a similar restrictions can be enforced using an md(4)-based vnode file/image and making use of mdsize for the jail).

    Renice prioritization

    CBSD uses renice(8) to prioritize each jail's access to the CPU. This makes it possible to select different priorities on a per jail basis and give the most impoartant jails the highest share of CPU time. For example, you may want to have your distcc jail set to a low priority, give your web server medium and the jail hosting the databse the highest priority. The actual priorization is taken care of by nice which gets the value for each jail from jail rctl. The values set here correspond to the behavior of nice(1) — the lowest integer resulting in the highest priority.

    Renice example:

    1) Let's create an AMP jail and have it run a php script that performs some work (such as bench.zip taken from http://www.php-benchmark-script.com/). We then clone the jail, calling the first highprio1 and the second lowprio1. Using cbsd jrctl-tui we give the first the highest possible priority -20, and set the second jail to the lowest priority of 20. In addition we limit the jail to one core through cpuset with cbsd jconfig (single-core systems are hard to come by these days and smart schedulers do not allow for a clean experiment without taking this step ;-).

    			% cbsd jls display=jid,jname,ip4_addr,cpuset
    			16   highprio1   10.0.0.121/24  4
    			17   lowprio1    10.0.0.122/24  4
    		

    — jid 16 is the higher prioritised jail while jid 17 — is set to a lower priority. Both run on the fourth core.

    make poll top state congestion php-fpm with JID output:

    		% export iter=1
    		% while [ 1 ]; do
    			printf "Iter: $iter" ;
    			iter=$((iter+1))
    			top -jab | grep php
    			sleep 1
    		done
    		

    We launch the script on both IPs at the same time:

    	% fetch -T 300 -o /dev/stdout http://10.0.0.121 & fetch -T 300 -o /dev/stdout http://10.0.0.122 & 
    	[1] 65985
    	[2] 65986
    	
    	--------------------------------------
    	|        PHP BENCHMARK SCRIPT        |
    	--------------------------------------
    	Start : 2014-01-06 16:28:59
    	Server : @10.0.0.121
    	PHP version : 5.4.23
    	Platform : FreeBSD
    	--------------------------------------
    	test_math                 : 12.870 sec.
    	test_stringmanipulation   : 15.896 sec.
    	test_loops                : 8.968 sec.
    	test_ifelse               : 7.864 sec.
    	--------------------------------------
    	Total time:               : 45.598 sec.
    	
    	--------------------------------------
    	|        PHP BENCHMARK SCRIPT        |
    	--------------------------------------
    	Start : 2014-01-06 16:29:02
    	Server : @10.0.0.122
    	PHP version : 5.4.23
    	Platform : FreeBSD
    	--------------------------------------
    	test_math                 : 32.632 sec.
    	test_stringmanipulation   : 18.053 sec.
    	test_loops                : 6.323 sec.
    	test_ifelse               : 5.504 sec.
    	--------------------------------------
    	Total time:               : 62.512 sec.
    	
    	[2]    Done                          fetch -T 300 -o /dev/stdout http://10.0.0.122
    	[1]  + Done                          fetch -T 300 -o /dev/stdout http://10.0.0.121
    		

    and we can soon observe the following output from top:

    	Iter: 1
    	65101  16 www           1  35  -20 32548K 11456K CPU4    4   0:35  20.56% php-fpm: pool www (php-fpm)
    	65587  17 www           1  52   20 32548K 11456K RUN     4   0:32   0.00% php-fpm: pool www (php-fpm)
    	Iter: 2
    	65101  16 www           1  60  -20 32548K 11456K RUN     4   0:36  25.98% php-fpm: pool www (php-fpm)
    	65587  17 www           1  42   20 32548K 11456K CPU4    4   0:33   2.10% php-fpm: pool www (php-fpm)
    	Iter: 3
    	65101  16 www           1  60  -20 32548K 11456K CPU4    4   0:36  26.27% php-fpm: pool www (php-fpm)
    	65587  17 www           1  94   20 32548K 11456K RUN     4   0:33   8.59% php-fpm: pool www (php-fpm)
    	Iter: 4
    	65101  16 www           1  61  -20 32548K 11456K CPU4    4   0:37  31.69% php-fpm: pool www (php-fpm)
    	65587  17 www           1  95   20 32548K 11456K RUN     4   0:34   9.47% php-fpm: pool www (php-fpm)
    	Iter: 5
    	65101  16 www           1  62  -20 32548K 11456K CPU4    4   0:37  35.60% php-fpm: pool www (php-fpm)
    	65587  17 www           1  95   20 32548K 11456K RUN     4   0:34  11.18% php-fpm: pool www (php-fpm)
    	Iter: 6
    	65101  16 www           1  64  -20 32548K 11456K CPU4    4   0:38  38.96% php-fpm: pool www (php-fpm)
    	65587  17 www           1  96   20 32548K 11456K RUN     4   0:34  12.79% php-fpm: pool www (php-fpm)
    	..
    		

    The jail with jid 16 is getting a larger share of the available CPU cycles and executes almost 1.5 times faster.

    RACCT/RCTL framework

    If your kernel has support RACCT/RCTL, you can set limits on the jail and watch the current statistics for jail resources. To do this, there is a command cbsd jrctl, which arguments

    			% cbsd jrctl mode=apply  ...
    		

    and

    			% cbsd jrctl mode=unset  ...
    		

    automatically called for the install or removal of limits when working jstart or jstop respectively. By command:

    			% cbsd jrctl mode=show
    		

    you can see current statistics on the jail resources consumed, which can be used to generate reports and graphs for loading jail, as well as the CBSD daemon used to generate recommendations on the need to add new resources and for overload warnings.

    By command:

    			% cbsd jrctl
    		

    without arguments you cat see default limits on all jails. Through argument display you can specify the fields for output data. If display is not specified, the value takes from $workdir/etc/defaults/jrctl.conf file, which you can change at its discretion via $workdir/etc/jrctl.conf

    You can edit the limits through

    			% cbsd jrctl-tui jname=jname
    		

    or, if you build a hosting and want to create limits on non-interactively, you can generate a file $workdir/$jname/jail.limits

    By jrctl you can set the following limits jail:

    a) All you can do a framework FreeBSD rctl(8):

    	cputime 	   CPU time, in seconds
    	datasize	   data size, in bytes
    	stacksize	   stack size, in bytes
    	coredumpsize	   core dump size, in bytes
    	memoryuse	   resident set size, in bytes
    	memorylocked	   locked memory, in bytes
    	maxproc 	   number of processes
    	openfiles	   file descriptor table size
    	vmemoryuse	   address space limit, in bytes
    	pseudoterminals    number of PTYs
    	swapuse 	   swap usage, in bytes
    	nthr		   number of threads
    	msgqqueued	   number of queued SysV messages
    	msgqsize	   SysV message queue size, in bytes
    	nmsgq		   number of SysV message queues
    	nsem		   number of SysV semaphores
    	nsemop		   number of SysV semaphores modified in a single semop(2) call
    	nshm		   number of SysV shared memory segments
    	shmsize 	   SysV shared memory size, in bytes
    	wallclock	   wallclock time, in seconds
    		

    expose: tcp/udp port forwarding from master host to jail

    command: expose

    			% cbsd expose jname=test2 mode=add in=200 out=200
    			% cbsd expose jname=test2 mode=delete in=200 out=200
    			% cbsd expose jname=test2 mode=list
    			% cbsd expose jname=test2 mode=clear
    			% cbsd expose jname=test2 mode=flush
    		

    By command cbsd expose you can create forward rule for tcp/udp port from external IP to jail.

    Generation of bootable ISO and USB Memstick from jail

    jail2iso command

    			% cbsd jail2iso
    		
    Description:

    The jail2iso script allows tirmomg a specified jail into bootable images for CD/DVD/Memstick media or the bhyve hypervisor. Despite jail2iso not being used by CBSD itself for its operation or jails, the script can be very useful for easy and comfortable creation of customized LiveCDs, and testing their configuration in a jail environment. This functionality can be useful when:

    • Creating rescue-system with desired tools
    • Building your own FreeBSD distribution
    • Creating images for diskless servers/stations (eg booted from PXE, MicroSD, Flash, CD/DVD, etc.) and have mounted home directories or jails/data files from NFS/FibreChannel/iSCSI/InfiniBand.
    • Transfering environment from jail to a bhyve image to make use of bhyve features

    Script job for iso/memstick

    • Creation of file hierarchy of next image consisting of mounted base and data of a jail
    • Creation of MFS/UZIP of an image which will remain in memory. It is used generally for work acceleration in the LiveCD mode, allowing for "cache" often caused utilities and libraries, for example from / to /usr.
    • Mount over MFS of hierarchy of data from file system of the media through nullfs/RO
    • Mount through tmpfs over RO file system for modification in LiveCD mode.

    If you need a number of directories reload when loading image in RW via tmpfs, before executing jail2iso you needs to be written to a file $systemdir/jail/tmpfsdir all the path. For example rescuebsd jail is a file: /usr/jails/jails-system/rescuebsd/tmpfsdir with content:

    		/root
    		/var/run
    		/var/cache
    		/var/db
    		/var/spool
    		/var/log
    		/usr/home
    		/usr/local/etc
    		

    These entries are processed /etc/rc.d/tmpfsdir, which will save to the image by jail2iso. All contents of these files, which is on the jail, will be moved to tmpfs filesystem. As RW areas are mounted through TMPFS, the quantity of memory available to record will depend on quantity of RAM of servers on which LiveCD is started.

    You may prefer to write your own /boot/loader.conf to created image. To do this, save the file loader.conf in the directory $systemdir/$jname/. Everything you write in this file will be added to loader.conf, generated jail2iso, which has the following mandatory entries:

    			kern.ipc.shmmni=4096
    			kern.ipc.shmseg=4096
    			radeonkms_load="YES"
    			i915kms_load="YES"
    			linux_load="YES"
    		

    You can specify what type of image jail2iso is to create, ISO image with cd9960 for CD/DVD/VMs or Memstick, an image that you can burn to a USB Flash drive.

    Script job for bhyve image

    Since the image will be initially bhyve mode RW, it does not require the creation of RO image with UZIP and support tmpfsdir, so at this stage there is no formation of an image. The script will automatically create a fstab-entry to mount an appropriate device in bhyve machine and updates /etc/ttys as required instructions. When creating an image bhyve, use the parameter:

    			% cbsd jail2iso ... freesize=
    		

    Because without this option, create disk volume equal to the volume of evidence, without reserve. This will make the system broken, so through freesize= further specify the amount of free space in the image there after copying master data.

    For more information

    cbsd jail2iso is not controlled and does not limit the volume of the final result — they are limited only by your destination media:

    For create ISO image use:

    			%  cbsd jail2iso media=iso ..
    		

    For create memstick image use:

    			% cbsd jail2iso media=memstick ..
    		

    For create bhyve image use:

    			% cbsd jail2iso media=bhyve freesize=XXg ..
    		

    Directory where to save the image specified as an argument dstdir.

    By default, the image will be have GENERIC kernel from $workdir/basejail/ directory. You can get this kernel via

    % cbsd repo action=get sources=kernel

    for your version of FreeBSD, or build it yourself through cbsd buildkernel. If the GENERIC kernel is not suitable as an argument name for jail2iso you can specify a different kernel if you have it.

    In addition, to reduce the size of the final image, jail2iso a series of actions:

    • removal .a-archive files in lib directories
    • removing unnecessary data by file list

    By default, this list is located in the $workdir/share/jail2iso-prunelist. You can correct them if you needed, or create your own, specifying the path to it as an argument prunelist

    Example of creating memstick from jail1:

    			% cbsd jail2iso media=memstic jname=jail dstdir=/tmp
    		

    If the file /tmp/jail1.img created, it can be writte to a USB device via the command:

    			dd if=/tmp/mc.img of=/dev/da0 bs="10240" conv="sync"
    		

    , where /dev/da0 — is USB Flash storage.

    Example of creating and launching bhyve image from jail1 with networks via interface em0:

    			% cbsd jail2iso media=bhyve jname=jail1 dstdir=/tmp freesize=1g
    			% kldload vmm
    			% kldload if_tap
    			% sysctl -w net.link.tap.up_on_open=1
    			% ifconfig tap0 create
    			% ifconfig bridge0 create
    			% ifconfig bridge0 addm em0 addm tap0
    			% ifconfig bridge0 up
    			% sh /usr/share/examples/bhyve/vmrun.sh -d /tmp/jail1-10.0_amd64.img vm1 
    		

    where /tmp/jail1-10.0_amd64.img — result from jail2iso.

    Searching for jail in node farm

    jwhereis, jailmapdb commands

    			% cbsd jwhereis
    			% cbsd jailmapdb
    		
    Description:

    Should you manage more than one FreeBSD/cbsd node farm, you can generate a map with jail locations refering to the appropriate servers. The utility jlogin is used in order to search remote jails. This map can be used for a variety of tools, such as a custom admin panel, automatically documentating the farm, as well as a variety of services. For example, a Bacula jail can automatically search the map for new jails and create configuration for their backup; or when a jail migrates from one physical node to another the target host can be reconfigured with jails for backup without the need for manual interaction by the system administrator.

    Copying files from/to jail filesystem

    jailscp command

    			% jail1:remotefile1 localfile1 [ localfile2 jail2@:remotefile2 ]
    		

    The cbsd jailscp command allows you to copy files between your host filesystem that of a jail. The syntax is similar the one used by scp(1)

    Copying files to and from remote jails is only possible if the necessary ssh keys are available. When a rootkeyfile for remote node is present, the connection will default to using root privileges rather than those of the cbsd user. This allow you to write files to any directory in the jail, not those owned by cbsd

    Example:

    			% cbsd jailscp
    		

    A few words about jail traffic counting

    command: fwcounters

    Description:

    Currently, the easiest way to count traffic - use functional of ipfw count on the JID of necessary environment.

    Necessary conditions for the implementation of this:

    • loaded module ipfw.ko;
    • mounted in CBSD (for example via cbsd initenv-tui) ipfw_enable parameters to 1;
    • count the presence of the rules in ipfw before any other firewall rules.

    CBSD starting and stopping the jails and removes automatically sets the rules for traffic jail, using a range of these rules settingCBSD (range by default fwcount_st="99" - fwcount_end="2000" )

    In other words, if you want to count traffic and still have the ability to filter traffic, create filtering rules above 2000 and not taking rules from 99 to 2000.

    Collect of traffic occurs 1 time per hour or when stopping the jail and stored in SQLite3 base, located in the system directory jail: $workdir/jails-system/$jname/traffic/YYYY-MM.sqlite, where YYYY, MM - year and month.

    Example. Enjoying the traffic statistics for the jail kde4. meaning of the fields outgoing and incoming - in bytes:

    root@home:/ # sqlite3 /usr/jails/jails-system/kde4/traffic/2014-09.sqlite 
    SQLite version 3.8.6 2014-08-15 11:46:33
    Enter ".help" for usage hints.
    
    sqlite> .schema traffic
    CREATE TABLE traffic (  dt TIMESTAMP DATE DEFAULT (datetime('now','localtime')) UNIQUE PRIMARY KEY, incoming integer, outgoing integer  );
    
    sqlite> select * from traffic order by dt desc limit 15;
    2014-09-20 15:00:52|142704274|4958246
    2014-09-20 14:00:51|163907026|25242205
    2014-09-20 13:00:49|3894888|182821
    2014-09-20 05:49:53|58329247|41769720
    2014-09-20 05:00:23|24247445|3464945
    2014-09-20 04:00:56|67749758|39433640
    2014-09-20 02:28:36|11767628|438283
    2014-09-20 02:00:57|115675943|10809029
    2014-09-20 01:00:54|279397568|156221677
    2014-09-20 00:00:51|223665101|6232876
    2014-09-19 23:00:50|250122634|8619979
    2014-09-19 22:00:49|221508227|6458218
    2014-09-19 01:00:42|64715837|3443253
    2014-09-19 00:00:38|109266525|8541659
    2014-09-18 23:00:34|99594683|20380707
    sqlite>