Deploy Kubernetes Cluster on FreeBSD Bhyve (CBSD)

My goal is to create a small local laboratory with a working k8s cluster, and I have a server at my disposal with the following characteristics:

• 1x3 TB hard drive with FreeBSD 13-CURRENT ( ZFS-on-root )
• RAM: 256 GB
• CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz (56 core)

In other words, we have no shortage of resources. Using the Ubuntu 18 cloud image, I generated a kubernetes cloud image, which differs from the Ubuntu image only in that it is connected to the kubernetes repository, and the apt install kubeadm command was executed, so when deploying the image, you can immediately start working with the kubectl utility.

I uploaded the experimental k8s image to the mirrors of the CBSD project. You can start the virtual machine from this image through the standard CBSD dialog and the bconstruct-tui command:

First of all, we need to choose the type of guest - in the case of k8s, our guest is Linux:

Next, select the name containing kubernetes, in the list of ready-made Linux profiles - this is a cloud image, so look for it at the bottom of the list:

Edit the remaining parameters, if necessary:

For example, we can change the network settings for the cloud-init helper:

After editing the parameters, we create a virtual machine:

As a result, we got a working virtual machine in which kubernetes is already installed, but this is where the main features of CBSD for kubernetes end - it focuses on other tasks.

However, this is already good - thanks to cloud-init, you do not need to install the OS every time using the installer, and you do not need to install kubernetes every time after installing the OS - you just need to start the VM from the gold image. If you like to do manual work in some cases - 20 seconds of starting the virtual machine - and you can continue.

Since creating and configuring k8s is also not a trivial task (primarily because it is a lot of monotonous monkey work in the form of repeating a large number of identical commands), I placed these commands in separate scripts and these scripts have received the status k8s module for CBSD, where all operations of which begin with a prefix with the same name k8s . All CBSD commands display information about their required or not-so arguments with a brief description through the --help argument. The module offers a small number of operations, including bootstrap, join, configuring and receiving a Kubernetes token. You can start these operations in strict order one after another, or you can specify all the necessary arguments one time - the module itself will figure out when and what to do. For example, when you execute this command:

cbsd k8s bootstrap=1 join=1 token=1 setup="master node"

we ask the module to complete the entire cycle of initializing one cluster, with the export of the token for working with kubectl using the CLI through a remote machine:

The CBSD module performs its work in various places - in parallel, in some - sequentially.

For example, each script is executed in parallel, but before the module can proceed to the next step, it needs to make sure that the most recent or slowest node has completed the configuration and is ready for the next step. Thus, the final step of each individual scenario is expectation. For example, we can observe the fact of parallelization and the progress of internal tasks in the output of cbsd taskls :

Sooner or later (in my case - within 20 seconds), the module switched from the initial boot to the configuration phase, when one node is assigned the 'master' role, all the others become 'worker. The purpose of k8s bootstrap is to start a virtual machine with kubernetes from a golden image - which we did above through cbsd bconstruct-tui by starting one virtual machine from the dialog interface. Thus, within 20 seconds we started 10 virtual machines, and now we can observe them through the output of cbsd bls :

I must say that the role of CBSD here is not only launching virtual machines and working with cloud-init, but also automatically searching for free names for new containers (in this case, the kube mask is used as the beginning of the container name) and searching and automatically assigning free IPs addresses that the system takes from the CBSD node ip pool settings. The screenshot shows that the IP addresses 10.0.0.XX that are issued to virtual machines do not always go sequentially - some of them are already on the network.

The configuration stage for master/worker is the longest and takes 3 minutes:

Then comes the join stage, which takes 42 seconds of our life. Finally, the script, exporting the token, succeeds:

Since we exported (token=1) the token for working with k8s to the local machine, we can work with the cluster directly from the FreeBSD host. We can immediately execute the 'kubectl get nodes' command and make sure everything is in place.

So, the cluster is ready to go. To test, you can apply the sample from the official k8s documentation and run your first container in a new cluster:

To delete cluster, there is an destroy=1 argument (either delete with the standard cbsd bremove command). Also, you can reset cluster settings without destroying virtual machines (reset=1). Enjoy!

A small podcast with demos of this article on Youtube: https://youtu.be/ADBuUCtOF1w

Conclusions and Future Plans

In the context of a dynamically developing IT industry, it is extremely important to automate all possible operations and get the result as soon as possible - to value your time and nerves. And use these limited and most valuable resources to accomplish your own tasks. This article demonstrates the collaboration of a number of technologies: FreeBSD. Linux, ZFS, cloud-init, bhyve, CBSD which were used for one thing - to raise a ready-to-use Kubernetes cluster of any configuration as quickly as possible.

Obviously, the longest step (configure/setup) can also be optimized here, since this step performs various actions that work with remote resources (wget .yaml configurations for calico, etc.) - this can also be included in the cloud image of kubernetes in the future.

Also, it seems like an interesting idea to be able to combine several physical servers with FreeBSD/CBSD via VXLAN, creating a single L2 segment for all containers. Using the p9fs , Ceph, NFS or S3 object storage here also has the right to exist for organizing a single disk space for containers on different physical nodes.

Also, with proper refinement, we can get a FreeBSD and ZFS-based, some fault tolerance and scalable Kubernetes cluster in the form of a black box, where black box - does not mean 'closed'. Here, we mean as a result a framework that does not require an IT specialist to immerse in FreeBSD, bhyve and CBSD technologies, but we get a universal API for controlling and managing K8S clusters, as a final product.

Anyone who is interested in FreeBSD / bhyve + kubernetes and wants to run their own SaaS K8S service on their resources using CBSD - write ideas and comments. It would be interesting to provide the public with a simple and cheap way for these operations.