How to use Velero for Kubernetes backups

This guide describes how to deploy Velero for Kubernetes and manage backups for Kubernetes-based applications on Virtuozzo Infrastructure.

About Velero

Velero is an open-source tool to backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.

Velero lets you:

Take backups of your cluster and restore in case of loss
Migrate cluster resources to other clusters
Replicate your production cluster to development and testing clusters

Velero consists of:

A server that runs on your cluster
A command-line client that runs locally

Prerequisites

1. Deploy a Virtuozzo Infrastructure cluster.

2. Create the compute cluster with the Kubernetes and load balancing services.

3. Configure a storage policy named standard for boot volumes on Kubernetes master nodes. Ensure that the selected policy is available for all projects where you are planning to deploy Kubernetes.

4. Create a Kubernetes cluster.

5. Ensure that you have the credentials (the access key and secret key) to the object storage service. In this guide, we will use the S3 object storage and a bucket provided by Virtuozzo Infrastructure.

6. Create a storage class with the snapshot functionality enabled:

6.1. Create the default storage class with the storage policy standard. The storage policy must be available in your project.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# cat > storage-class.yaml <<\EOT
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: default
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: cinder.csi.openstack.org
parameters:
  type: standard
EOT

Apply the configuration file:

1
# kubectl create -f storage-class.yaml

6.2. Install the required custom resource definitions (CRD) for volume snapshots:

6.2.1. Check the existing CRDs:

1
2
3
4
# kubectl api-resources | grep volumesnapshot
volumesnapshotclasses    snapshot.storage.k8s.io/v1beta1        false        VolumeSnapshotClass
volumesnapshotcontents   snapshot.storage.k8s.io/v1beta1        false        VolumeSnapshotContent
volumesnapshots          snapshot.storage.k8s.io/v1beta1        true         VolumeSnapshot

6.2.2. Install the missing CRDs (install only one version):

1
2
3
4
5
6
7
8
# git clone https://github.com/kubernetes-csi/external-snapshotter/
# cd ./external-snapshotter
# git checkout release-5.0
# kubectl apply -f client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
# kubectl apply -f client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
# kubectl apply -f client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
# kubectl apply -f deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml -n kube-system
# kubectl apply -f deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml -n kube-system

Important: The CSI snapshotter version must not be higher than release-5.0.

6.3. Create a snapshot class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# cat > snapshotclass.yaml <<\EOT
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: snapclass
driver: cinder.csi.openstack.org
deletionPolicy: Delete
parameters:
  force-create: "true" 
EOT

Apply the configuration file:

1
# kubectl apply -f snapshot-class.yaml

Installing Velero in a Kubernetes cluster

1. Download the required configuration files to your local machine:

1
2
# wget https://virtuozzo-k8s-demo.s3.amazonaws.com/credentials-velero-example
# wget https://virtuozzo-k8s-demo.s3.amazonaws.com/velero.yaml

2. Install the Velero command-line client on your local machine.

3. Connect to the Kubernetes cluster:

1
# export KUBECONFIG=<your_k8s_kubeconfig_file>

4. In the Velero configuration file velero.yaml, change the following parameters:

bucket to the name of your object storage bucket
s3Url to the address of your object storage endpoint

5. In the object storage credentials file credentials-velero-example, change the following parameters:

aws_access_key_id to your S3 access key
aws_secret_access_key to your S3 secret key

5. Add the repository where the Velero Helm chart is located:

1
# helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts

6. Install the chart from the added repository:

1
2
# helm upgrade --install velero vmware-tanzu/velero --create-namespace --namespace velero \
--set-file credentials.secretContents.cloud=credentials-velero-example -f velero.yaml

7. Verify the Velero installation:

1
2
3
4
5
6
# kubectl get backupstoragelocations -n velero
NAME      PHASE       LAST VALIDATED   AGE   DEFAULT
default   Available   19s              20h   true
# kubectl get volumesnapshotlocations -n velero
NAME      AGE
default   20h

The output shows that the backup and snapshot destinations (your S3 bucket) are properly configured.

Creating a test application

We will be using WordPress as a test application deployed using the Bitnami template:

1
# helm install wp-test bitnami/wordpress

Check that you have the following components deployed:

1
2
3
4
5
6
7
8
# kubectl get deployments -n default
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
wp-test-wordpress   1/1     1            1           12h
# kubectl get service -n default
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
kubernetes          ClusterIP      10.254.0.1      <none>           443/TCP                      40h
wp-test-mariadb     ClusterIP      10.254.28.108   <none>           3306/TCP                     13h
wp-test-wordpress   LoadBalancer   10.254.74.125   188.42.240.165   80:30964/TCP,443:30591/TCP   13h

In the output, the EXTERNAL-IP of your wp-test-wordpress load balancer is the public IP address of your test WordPress instance that you can use to access the application.

Note: The WordPress template consists of the MariaDB database and the WordPress application itself.

Creating the first backup

To create a backup, run the following command:

1
# velero backup create wp-test-b1 --selector app.kubernetes.io/instance=wp-test --snapshot-volumes --snapshot-move-data

Where:

wp-test-b1 is the name of your backup.
--selector app.kubernetes.io/instance=wp-test specifies the selector to identify Kubernetes entities for backup. All WordPress application components have the instance=wp-test tag, so Velero will back up all of them, including persistent volumes.
--snapshot-volumes specifies the snapshot approach that will be used to back up your persistent volumes.
--snapshot-move-data specifies that snapshots will be moved to your backup destination and not stored in the same Kubernetes cluster.

To check that the backup was successfully created and the snapshots were uploaded to S3, run:

1
2
3
4
5
6
7
# kubectl get backups -n velero
NAME                            AGE
wp-test-b1                      13h
# kubectl get datauploads -n velero
NAME                 STATUS      STARTED   BYTES DONE   TOTAL BYTES   STORAGE LOCATION   AGE     NODE
wp-test-b1-tcr98     Completed   13h       155618227    155618227     default            13h     k8s-103-mab4wz56ljff-node-0
wp-test-b1-x6m97     Completed   13h       167394550    167394550     default            13h     k8s-103-mab4wz56lj

Additionally, go to your S3 bucket and verify that your file structure is as follows:

You can also check the backup task status by running:

1
2
3
4
5
6
7
8
9
# velero backup describe wp-test-b1 --details
Name:         wp-test-b1
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.25.7
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=25
Phase:  Completed

Phase should have the status Completed.

Restoring the application

To test the restore functionality, let’s destroy the WordPress application. To do this, you can delete the WordPress deployment:

1
2
3
4
# kubectl delete deployments wp-test-wordpress
deployment.apps "wp-test-wordpress" deleted
# kubectl get deployments -n default
No resources found in default namespace.

You can check that the WordPress application is no longer available by opening its public IP address in the browser.

You can also delete the WordPress persistent volume:

1
2
3
4
5
6
7
# kubectl get pvc
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-wp-test-mariadb-0   Bound    pvc-c8134c2b-9f0b-424f-a42a-63b25cfcf21c   8Gi        RWO            standard       12h
pvc01                    Bound    pvc-35e8f714-5c8f-4c6e-920e-230d1b421f0e   20Gi       RWO            standard       40h
wp-test-wordpress        Bound    pvc-eef40f1c-dff7-4317-be74-f49a0df5feec   10Gi       RWO            standard       12h
# kubectl delete pvc wp-test-wordpress
persistentvolumeclaim "wp-test-wordpress" deleted

To restore the application from the created backup, run:

1
# velero create restore wp-test-restore1 --from-backup wp-test-b1

Where:

wp-test-restore1 is the name of a restore task
wp-test-b1 is the name of your backup

Open the WordPress public IP address in the browser, to verify that the WordPress application is available again.

Creating a backup schedule

For managing backup schedules, Velero uses cron. To learn more, refer to the Velero documentation.

To create backups every hour and send backup files, including snapshots, to your S3 storage, run:

1
# velero schedule create wp-test-hourly --schedule="0 * * * *" --selector app.kubernetes.io/instance=wp-test --snapshot-volumes --snapshot-move-data

Creating a file system-based backup

Velero also supports file system backup. To learn more about this approach and how it is different from the snapshot-based backup, refer to the Velero documentation.

To create a file system backup, run:

1
# velero backup create wp-test-b2 --selector app.kubernetes.io/instance=wp-test --default-volumes-to-fs-backup

Enjoy!