FreeBSD virtual environment management and repository

2020-10 upd: we reached the first fundraising goal and rented a server in Hetzner for development! Thank you for donating !

Attention! I apologize for the automatic translation of this text. You can improve it by sending me a more correct version of the text or fix html pages via GITHUB repository.

CBSD API module

Intro

CBSD allows the user to create environments at different levels of abstraction by providing a large number of methods for creating environments - CBSDfile, [jb]create commands, which work both in the configuration file and through command line arguments. Finally, you can use interactive dialog menus ([jb]construct-tui). Despite the variety, all of these options require you to log into the server each time via ssh and enter the appropriate commands. The RestAPI service adds another option for managing environments for you, eliminating the need to use ssh, where you interact with the CBSD framework via HTTP / RestAPI requests.


private FreeBSD multi-node bhyve/jail cloud via API demo:

The service API architecture supports single-site and multi-site coverage, giving you the ability to easily create a private, programmable cloud to run virtualized environments, taking CBSD to a new level of environmental management. In addition to the classic request over HTTP (for example, through the curl utility), the API is supported by a CBSDfile, which in some cases gives a very powerful and more convenient deployment of services in a private cloud. But more on that below.

Let's fix the terminology that will be used in this article.

  • CBSD - a set of scripts that implement virtual machine management at the lowest level. You need CBSD at least 13.0.1. If the version is lower, please upgrade before you start.
  • ZPOOL - ZFS pool in which the CBSD working environment is initialized. By default, the CBSD working directory is initialized to /usr/jails.
  • Broker - the main communication data bus between the cbsd-mq-api service (one) and cbsd-mq-router (can be in any quantity). In this article, beanstalkd serves as the broker.
  • Tube Tube (in beanstald terminology) - a FIFO queue/pipe in the broker, to which cbsd-mq-api writes (publish), and cbsd-mq-router reads (subscribe).
  • Hypervisor (or hoster) (not to be confused with bhyve hypervisor) - a physical node running FreeBSD OS. The number of nodes is unlimited. Each hypervisor has at least 1 ZPOOL. Each pool is served by a cbsd-mq-router process that subscribes to a personal pool queue to process CBSD instructions.
  • API - service (public) that provides RestAPI or json-rpc for interaction between end users (curl / UI frontend). In our case, the simplest implementation will be the cbsd-mq-api service.

Let's start by looking at the simplest single-mode installation.

Standalone API

Installing and configuring the CBSD API in standalone mode.

In this example, all operations are performed on one hypervisor, which will act simultaneously as an API and a broker and a CBSD hoster, where working environments will be launched.

1) Install the message broker:

pkg install -y beanstalkd

Use a flag in rc.conf to specify at which address beanstalkd will wait for connection. Since we are all-in-one box, all connections will not go beyond the localhost boundary, so we specify 127.0.0.1 as the listening address:

sysrc beanstalkd_flags="-l 172.16.0.5 -p 11300"

let's start the service:

service beanstalkd enable
service beanstalkd start

2) At the time of this posting, cbsd-mq-api is not present in the FreeBSD ports system and is not in the pkg repository. We will get the service through source codes. To do this, we need to install the git package (git-lite is enough) and go:

pkg install -y git-lite go

We will clone the repository, build and install the service:

git clone https://github.com/cbsd/cbsd-mq-api.git /root/cbsd-mq-api
/root/cbsd-mq-api/build.sh
cp -a /root/cbsd-mq-api/cbsd-mq-api /usr/local/bin
cp -a /root/cbsd-mq-api/etc/cbsd-mq-api.json /usr/local/etc/
cp -a /root/cbsd-mq-api/rc.d/cbsd-mq-api /usr/local/etc/rc.d/

3) Install additional files in CBSD via the api module:

cbsd module mode=install api
echo 'api.d' >> ~cbsd/etc/modules.conf
cbsd initenv

4) We configure the API:

Let's create the necessary directory structure for the API:

mkdir -p /var/db/cbsd-api ~cbsd/var/db/api/map
chown -R cbsd:cbsd /var/db/cbsd-api ~cbsd/var/db/api/map

Edit the config file /usr/local/etc/cbsd-mq-api.json:

a) The server_url parameter must point to the name or IP address of your API service, available for external connections. This URL as-is will be substituted as the working endpoint. In our case, in the environment, the working external IP address is 172.16.0.5, so the complete configuration line will look like this:

"server_url": "http://172.16.0.5:65531",

let's start the service:

service cbsd-mq-api enable
service cbsd-mq-api start

Copy the configuration file for the service and specify at least one CBSD hoster in the server_list= parameter

cp -a /usr/local/cbsd/modules/api.d/etc/api.conf ~cbsd/etc/
chown cbsd:cbsd ~cbsd/etc/api.conf
vi ~cbsd/etc/api.conf

The server_list parameter must contain the cluster members (usually hostnames), separated by a space. The service uses these names as addressees for sending tasks via beanstalkd.

For example, in our case, the name of the CBSD node is cloud1.my.domain, so api.conf will contain the line:

server_list="cloud1.my.domain"

You can check the correctness of issuing recommendations for the host on which to start the new environment by running the script /usr/local/cbsd/modules/api.d/misc/recomendation.sh: it should return the server name.

5) Setting up cbsd-mq-router.

Install the service:

pkg install -y cbsd-mq-router

Or from the ports tree:

env BATCH=no make -C /usr/ports/sysutils/cbsd-mq-router install

Let's edit the file /usr/local/etc/cbsd-mq-router.json, specifying the correct address of our beanstalkd broker in the "uri" parameter in the beanstalkd section, in our case it is:

    "uri": "127.0.0.1:11300",

We also need to fix the name of the pipe in the broker to which the custom host's cbsd-mq-router service will subscribe. To do this, change the tube and reply_tube_prefix parameters in cbsd_ and cbsd__result_id, respectively, where the server name contains symbols '-' and '.' are converted to '_'.

So, having a CBSD host named cloud1.my.domain, the file /usr/local/etc/cbsd-mq-router.json should contain the following entries:

      "tube": "cbsd_cloud1_my_domain",
      "reply_tube_prefix": "cbsd_cloud1_my_domain_result_id",

If your CBSD working environment is different from the standard /usr/jails, adjust the path in the cbsdenv parameter.

let's start the service:

service cbsd-mq-router enable
service cbsd-mq-router start

Your first standalone private cluster is ready to serve you!

API: jail

Working with JAIL via the API

Let's create a simple payload to create a container, for example, a jail.json file.

jail.json:


{
  "type": "jail",
  "imgsize": "10g",
  "pubkey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILdzM3pm65J28zNt/YcRWSw4NXa8PKU8MhrbA/CShuEm olevole@jhoster1.olevole.ru"
}

Of course, substituting your public key as pubkey. Let's send it to the endpoint for container creation, where /jail at the end means the name of the environment (arbitrary)

curl -X POST -H "Content-Type: application/json" -d @jail.json http://172.16.0.5:65531/api/v1/create/jail

In this case, the API will calculate the md5 hash from your pubkey and the result of the check-sum will automatically become your token, which you can use to control environments and at the same time will be a kind of namespace in the cluster, in which you will see all the resources created with your public key.

So, if you need to destroy the environment, use a GET request for the /destroy endpoint using your hash. Get the md5 sum of your key if you still don't know it;)

md5 -qs "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILdzM3pm65J28zNt/YcRWSw4NXa8PKU8MhrbA/CShuEm olevole@jhoster1.olevole.ru"

use it in your request:

curl -H "cid:678851e352136726da5e6c52126eb906" http://172.16.0.5:65531/api/v1/destroy/jail

Of course, such a mechanism is only suitable for a private cloud. If you plan to raise a service for external users with a similar scheme, use more stringent validation methods to confirm dangerous operations, for example TOTP.

A few more words about jail API configuration. A script working with jail via the API can use a configuration file named jail-api.conf, which must be located in the ~cbsd/etc directory. You can see an example of this file here: /usr/local/cbsd/modules/api.d/etc/jail-api.conf

At the moment the file regulates whether to allow the user to specify in the payload the FreeBSD version of the base for the container, the pool of IP addresses and the external IP address for accessing the container given to the user. By default, we do not skip the FreeBSD base version for the container, no matter what value the user sends: the API will create a container for the version of your hypervisor (ver=native). Since the user of the cloud does not influence the issuance of addresses, and CBSD does this, you can assign any pool, both external addresses and private ones.

API: bhyve

Working with virtual machines bhyve via API

Working with virtual machines has a similar interface and endpoint, as in the case of jail, but has a slightly different payload for creating an environment. In addition, you must configure to match the name of the requested image with the CBSD profile. Like the jail script, the bhyve script also has a configuration file. Copy the example config file:

cp -a /usr/local/cbsd/modules/api.d/etc/bhyve-api.conf ~cbsd/etc/
chown cbsd:cbsd ~cbsd/etc/bhyve-api.conf

As in the case of jail, you can also regulate the rules for issuing IP addresses to virtual machines. And you can control which IP address will be given to the user for SSH access to the created resource - either the direct IP address of the virtual machine will be given, or CBSD will create an expose rule for SSH on any free port. In this case, the user will receive the IP address of the hypervisor and the port that will forward requests to the 22/ssh port of the virtual machine.

Payload example, e.g. centos7.json:

{
  "type": "bhyve",
  "imgsize": "10g",
  "ram": "1g",
  "cpus": "2",
  "img": "centos7",
  "pubkey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILdzM3pm65J28zNt/YcRWSw4NXa8PKU8MhrbA/CShuEm olevole@jhoster1.olevole.ru"
}

And it is created by a similar request as jail:

curl -X POST -H "Content-Type: application/json" -d @centos7.json http://172.16.0.5:65531/api/v1/create/cent7

CBSDfile + API

CBSDfile + API

And for a snack, as a berry on the cake

In some cases curl and RestAPI are much more convenient than ssh + `cbsd bcreate / bconstruct-tui`. If you have to deploy a lot of virtual environments and often, you cannot avoid the need to learn and use some high-level language. CURL and HTTP requests are well suited if you develop a programmable cloud based on CBSD, but it's not very convenient for human. As an intermediate option, there is another hybrid and implicit use case for the API - through the CBSDfile. If you are not familiar with this CBSD feature, be sure to check it out. In order to use the CBSDfile in your private cloud, you just need to add only two parameters to the beginning of the CBSDfile - this is the Endpoint of your API and the key. Having these two options will cause CBSD to switch from a local CBSD installation to the cloud when building. And this can provide a completely new level of capabilities (an example of interaction with a multi-node cluster)!

Multinode API

Multinode API/CBSD cluster

As stated earlier, the current architecture is capable of serving more than one hypervisor. That, with intensive deployment of virtual environments in large numbers, can significantly increase the efficiency and efficiency of your cloud resources.

Our simple API service has the ability to use two external scripts that implement logic for two actions:

  • a) suggest the following free name for the virtual environment. Taking into account several nodes, this solves the problem of possible collisions of names, since it is regulated at one point.
  • b) get recommendations on which host the requested virtual environment should be placed on.

Any flight of imagination is given here for the implementation of logic in any programming language convenient for you and the use of any algorithms. So, for example, if some of your servers have high-performance but low-volume NVME disks and some of your servers have capacious but slow hdd, you can easily implement a simple logic like:

> If a virtual machine with a disk capacity> 500GB is requested, select from the list of servers with hdd.

Together with the API module, two sh scripts are supplied, which, as examples, give a unique ID as a virtual machine name and a round-robin server issuing server. That should be enough to start a full-fledged multinode cluster in 10 minutes. Let's do that.

For simplicity and maximum freedom from dependencies, we will not use additional services or databases, we will manage with ordinary files and directory synchronization using the rsync protocol.

Let's start by setting up the service API. In relation to the standalone configuration, in the ~cbsd/etc/api.conf file, we expand the list by the required number of members in the cluster, for example:

server_list="srv-01.olevole.ru srv-02.olevole.ru srv-03.olevole.ru bob.your-server.de"

You can check that the recommendation script (/usr/local/cbsd/modules/api.d/misc/recomendation.sh) offers hosts in round-robin by default:

WIP...