Updating Automatically with Smart Updates

Smart updates build on automatic updates, improving their safety and providing the user with more control. Enabling smart updates means joining Virtuozzo Server nodes to a Kubernetes cluster and creating an update manager entity to update the nodes according to set rules. Enabling smart updates disables automatic updates.

As it is recommended to set up a highly available Kubernetes cluster in production, three or more control plane nodes in the same subnet are required to set up smart updates. In addition, you will need a free IP address at which the control plane will be available.

Note the following:

Smart updates can currently update ReadyKernel patches as well as Virtuozzo Server and VzLinux RPM packages on both standalone nodes and those in Virtuozzo Storage clusters.
Node update policies are initially obtained from the Virtuozzo update server as is with automatic updates. Then, however, a local update schedule, maintenance window, and policies are used for each node. They can be configured by changing the local update manager specification. Nodes with the auto policy are assigned one of the following policies:
- fast, updates installed immediately after publication, assigned to 10% of nodes
- slow, updates installed two weeks after publication, assigned to 40% of nodes
- stable, updates installed six weeks after publication, assigned to 50% of nodes
Updates are first checked for during the setup and then each 24 hours. Also, a check is performed each time the RPM database on the node is modified. This includes modifying the lists of banned RPM packages and ReadyKernel patches, as well as manual installation, updating, or removal of RPM packages. Available updates are installed as soon as the update policy permits. A custom schedule and a maintenance window can be set (see Setting the Maintenance Window and Schedule). If an update fails, the next attempt is made after at least 4 hours.
If a critical service on the node (e.g., VM dispatcher or Virtuozzo Storage mount point) stops working during an update or within 10 minutes after it, the ReadyKernel patch or all packages from the RPM transaction that caused it are added to a ban list. Banned patches and packages will not be installed in the future unless manually removed from the ban list (as explained in Excluding RPM Packages and ReadyKernel Patches from Updates).
The following repositories are checked for updates by default:
- virtuozzolinux-base
- virtuozzolinux-updates
- virtuozzo-os
- virtuozzo-updates
You can change this list manually (see Setting the Repositories).
Nodes are updated one by one. The node that has not been updated the longest is the first one in the queue.
Updating of Virtuozzo Storage nodes starts only if the storage cluster has the HEALTHY status.
Updated nodes need to be rebooted manually. For such nodes, a dedicated Prometheus metric, node_vz_needs_reboot, is set to 1.

Enabling Smart Updates

To set up smart updates with a highly available lightweight Kubernetes cluster, do the following:

Install the required RPM package:
1
# yum install smart-updates
This will install lightweight Kubernetes (k3s) among other dependencies.

Set up a highly available Kubernetes cluster.

Fault tolerance is ensured by keepalived. It provides a virtual IP address that is assigned to a control plane node. If it fails, the IP address migrates to a healthy control plane node.

On each control plane node, prepare a service configuration file. Use a sample supplied by Kubernetes:
1
# cp /usr/share/k3s/k3s_keepalived.conf.sample /etc/keepalived/keepalived.conf
Replace the placeholders in keepalived.conf as follows:
- ${INTERFACE} with the name of the node’s network interface that will negotiate for the virtual IP address. E.g., eth0.
- ${AUTH_PASS} with a password, e.g., mypasswd. It must be the same on all control plane nodes.
- ${APISERVER_VIP} with the virtual IP address at which the control plane will be available. E.g., 192.168.1.2.
On each control plane node, prepare a health check script. It will be called periodically to check that the node holding the virtual IP address is operational. Use a sample supplied by Kubernetes as well:
1
# cp /usr/share/k3s/check_apiserver.sh.sample /etc/keepalived/check_apiserver.sh
Replace the placeholder in check_apiserver.sh as follows:
- ${APISERVER_VIP} with the virtual IP address at which the control plane will be available. E.g., 192.168.1.2.

On each control plane node, configure the firewall:

1
2
3
4
5
6
# firewall-cmd --permanent --add-rich-rule='rule protocol value="vrrp" accept'
# firewall-cmd --permanent \
--add-port=8472/udp \
--add-port=9443/tcp \
--add-port=2379-2380/tcp
# firewall-cmd --reload

On the first control plane node, create the Kubernetes server configuration file and enable the keepalived, docker, and k3s-server services:

1
2
3
# echo "K3S_TOKEN=<my_secret_token>" >> /etc/k3s/k3s-server.env
# echo -e "K3S_EXTRA_FLAGS=\"--cluster-init\"" >> /etc/k3s/k3s-server.env
# systemctl enable keepalived docker k3s-server --now

Where <my_secret_token> is an arbitrary secret string.

On the second and third control plane nodes, create the Kubernetes server configuration files and enable the keepalived, docker, and k3s-server services:
1 2 3
# echo "K3S_TOKEN=<my_secret_token>" >> /etc/k3s/k3s-server.env # echo "K3S_URL=https://<virt_IP>:9443" >> /etc/k3s/k3s-server.env # systemctl enable keepalived docker k3s-server --now
Where:
- <my_secret_token> is the secret string from the previous step.
- <virt_IP> is the virtual IP address from /etc/keepalived/keepalived.conf.

On each agent node, open the required port, create the Kubernetes agent configuration file, and enable the docker and k3s-agent services:

1
2
3
4
# firewall-cmd --permanent --add-port=8472/udp && firewall-cmd --reload
# echo "K3S_TOKEN=<my_secret_token>" >> /etc/k3s/k3s-agent.env
# echo "K3S_URL=https://<virt_IP>:9443" >> /etc/k3s/k3s-agent.env
# systemctl enable docker k3s-agent --now

Where:

<my_secret_token> is the secret string from the previous step.
<virt_IP> is the virtual IP address from /etc/keepalived/keepalived.conf.

You can repeat this step later to add new nodes to the Kubernetes cluster.

Check that the cluster is healthy. For example:

1
2
3
4
5
# kubectl get nodes
NAME                  STATUS   ROLES         AGE     VERSION
node1.example.local   Ready    etcd,master   3h36m   v1.19.7-k3s1
node2.example.local   Ready    etcd,master   3h33m   v1.19.7-k3s1
node3.example.local   Ready    etcd,master   3h33m   v1.19.7-k3s1

Deploy the maintenance operator in the Kubernetes cluster:

1
# kubectl apply -f /usr/share/smart-updates/maintenance-operator.yaml

Wait until the maintenance operator is ready:

1
2
3
4
# kubectl wait --namespace=vz-maintenance-operator \
--for=condition=Complete job vz-maintenance-deployment-job-v<version> --timeout=1h

job.batch/vz-maintenance-deployment-job-v<version> condition met

Where <version> is the version of the smart-updates package, e.g., 1.0.0. You can also obtain the full job name from the maintenance operator configuration file. For example:

1
2
3
# grep "vz-maintenance-deployment-job-v" /usr/share/smart-updates/maintenance-operator.yaml

name: vz-maintenance-deployment-job-v1.0.0

Create an UpdateManager instance to initiate synchronization of updates:
1
# kubectl apply -f /usr/share/smart-updates/update-manager.yaml

Checking Node Status

You can now check the state of updates on nodes in the Kubernetes cluster with

1
# kubectl -n vz-maintenance-operator get updatemanager -o <yaml|json>

A typical response (abridged) may look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
<...>
"status": {
  "nodes": [
    {
      "id": "node1.example.local",
      "kernel": "3.10.0-1160.21.1.vz7.174.10",
      "readyKernel": {
        "status": "UpToDate"
      },
      "rpmPackages": {
        "status": "UpToDate"
      },
      "stageName": "stable"
    },
    {
      "id": "node2.example.local",
      "kernel": "3.10.0-1160.21.1.vz7.174.10",
      "readyKernel": {
        "status": "UpToDate"
      },
      "rpmPackages": {
        "numberOfPackagesToUpdate": 1,
        "status": "UpToDate"
      },
      "stageName": "fast"
    },
    {
      "id": "node3.example.local",
      "kernel": "3.10.0-1160.21.1.vz7.174.10",
      "readyKernel": {
        "status": "UpToDate"
      },
      "rpmPackages": {
        "status": "UpToDate"
      },
      "stageName": "slow"
    }
  ]
}
<...>

The sample above shows that all nodes are up-to-date, as per their update policies. It also reports that one RPM package has recently been installed on the second node, node2.example.local.

Setting the Maintenance Window and Schedule

You can set one or more maintenance windows during which updating is allowed to start. The format is 0h00m, e.g., 1h30m for a window that is one hour and 30 minutes long.

You can also set an update schedule similar to that used by automatic updates. The format is standard Unix cron format, e.g., 0 1 * * 1-5 means “each day at 01:00 from Monday to Friday”.

Do the following:

Open the update manager specification for editing:

1
# kubectl -n vz-maintenance-operator edit updatemanagers default

Add the required parameters to the spec part. For example:

1
2
3
4
5
6
spec:
  <...>
  maintenanceWindows:
  - duration: 1h30m
    schedule: 0 1 * * 1-5
  <...>

Save and close the specification.

Setting the Repositories

RPM package updates are obtained from default repositories listed early in this section. You can, however, change them separately for each policy. If you do so, only the repositories that you provide will be used by smart updates. You will still be able to manually install RPM packages from unlisted repositories.

Do the following:

Open the update manager specification for editing:

1
# kubectl -n vz-maintenance-operator edit updatemanagers default

Add the needed repository IDs under the repositories key for the desired policy. Add the key if needed.

You can get repository IDs from yum repolist.

For example, to use only VzLinux repositories for the fast policy:

1
2
3
4
5
6
7
8
9
 spec:
   stages:
   - name: fast
     nodeSelector:
       matchLabels:
         maintenance.virtuozzo.io/auto-updates-policy: fast
     repositories:
     - virtuozzolinux-base
     - virtuozzolinux-updates

Save and close the specification.

Excluding RPM Packages and ReadyKernel Patches from Updates

You can prevent specific RPM packages and ReadyKernel patches from being installed during updating.

The format for RPM packages is <name>-<version>-<release>.<arch>, e.g., bash-4.2.46-34.vl7.x86_64. The format for ReadyKernel patches is X-Y-rZ, e.g., 123-4-r5.

Do the following:

Open the update manager specification for editing:

1
# kubectl -n vz-maintenance-operator edit updatemanagers default

List the RPM packages and ReadyKernel patches in the spec section, under bannedPackages and bannedPatches keys, respectively. For example:

1
2
3
4
5
6
7
spec:
  <...>
  bannedPackages:
  - bash-4.2.46-34.vl7.x86_64
  bannedPatches:
  - 123-4-r5
  <...>

Save and close the specification.

To allow installation of banned patches and packages again, remove them from the list.

Changing Node Update Policies

To change a node’s update policy, re-label the node in the update manager specification as follows:

1
# kubectl label node <node> maintenance.virtuozzo.io/auto-updates-policy=<policy> --overwrite

For example:

1
# kubectl label node node1.example.local maintenance.virtuozzo.io/auto-updates-policy=fast --overwrite

Such re-labelling does not overwrite node update policies on the Virtuozzo update server. If smart updates are disabled, automatic updates will continue working as before.

Suspending Updates for All or Specific Nodes

You can prevent updates from being installed either across the entire Kubernetes cluster or on particular nodes. It may be useful to prepare the nodes for maintenance, for example.

To suspend updating cluster-wide, set suspend: true in the update manager specification:

Open the update manager specification for editing:

1
# kubectl -n vz-maintenance-operator edit updatemanagers default

Add the required parameters to the spec part. For example:

1
2
3
4
spec:
  <...>
  suspend: true
  <...>

Save and close the specification.

To suspend updates on a specific node, assign the suspend label to it:

1
# kubectl label nodes node1.example.local maintenance.virtuozzo.io/suspend=""

The change will be reflected in the node status. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# kubectl -n vz-maintenance-operator get updatemanager -o json
   "status": {
     "nodes": [
       {
         "id": "node1.example.local",
         "kernel": "3.10.0-1160.21.1.vz7.174.10",
         "readyKernel": {
           "reason": "ManuallySuspended",
           "status": "Suspended"
         },
         "rpmPackages": {
           "reason": "ManuallySuspended",
           "status": "Suspended"
         },
         "stageName": "stable"
       },
     <...>

To resume updates for a suspended node, remove the suspend label. For example:

1
# kubectl label nodes node.example.local maintenance.virtuozzo.io/suspend-

Again, the change will be reflected in the node status. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# kubectl -n vz-maintenance-operator get updatemanager -o json
   "status": {
     "nodes": [
       {
         "id": "node1.example.local",
         "kernel": "3.10.0-1160.21.1.vz7.174.10",
         "readyKernel": {
           "status": "UpToDate"
         },
         "rpmPackages": {
           "status": "UpToDate"
         },
         "stageName": "stable"
       },
     <...>

Disabling Smart Updates

To disable smart updates and revert to automatic updates, do the following:

Delete the Kubernetes entities related to the maintenance operator:

1
# kubectl delete -f /usr/share/smart-updates/maintenance-operator.yaml

If you deployed Kubernetes solely for smart updates and no longer need it, disable its server and agent services:
1
# systemctl disable --now k3s-*
Run this command on each node.