Managing Virtual Machine and Container Backups

Backing up your virtual machines and containers on a regular basis is essential for system reliability. Virtuozzo Server allows you to back up and restore virtual machines and containers on the local node with the prlctl and prlsrvctl utilities.

Creating Virtual Machine and Container Backups

You can create backups of virtual machines and containers with the prlctl backup command. The command is executed on the local node where the virtual machines or containers are located. The resulting backups can be stored on either the local node or a remote one (e.g., a dedicated backup server).

By default, an incremental backup is created that contains only the files changed since the previous full or incremental backup. If no previous backups exist, a full backup is created. You can, however, forcibly create a full backup with the -f option.

For example, to create a backup of the virtual machine MyVM and store it on the local node, run the following command:

1
2
3
4

# prlctl backup MyVM
...
The VM has been successfully backed up with \
backup ID {746dba2a-3b10-4ced-9dd6-76a2blcl4a69}

The backup UUID, like the one shown above, will be required to manage the backup in the future.

To create a backup of the virtual machine MyVM and store it on a remote node, specify the remote node’s IP address or hostname with the -s option, for example:

1

# prlctl backup MyVM -s user:passwd@192.168.0.10

The root account is used to log in to the remote node by default, so you will be asked for the root password. You can also provide different credentials and port in the format [<user>[:<passwd>]@]<server>[:<port>]. Alternatively, you can set up RSA authentication and not use passwords at all (see Setting Up RSA Authentication Between Nodes).

By default, backups are placed in the /vz/vmprivate/backups directory. To set another default backup directory for the node, use prlsrvctl. For example:

1
2
3

# prlsrvctl set --backup-path "/vz/mybackupdir"
# prlsrvctl info | grep "Backup path"
Backup path: /vz/mybackupdir

To change the default backup directory for a specific virtual machine or container, use prlctl. For example:

1
2
3

# prlctl set MyVM --backup-path "/vz/myvmbackupdir"
# prlctl list -i MyVM | grep "Backup path"
Backup path: /vz/myvmbackupdir

Now the command prlctl backup MyVM will back up MyVM to /vz/myvmbackupdir by default.

In general, you can use the --backup-path option to back up VEs to any existing directory, even if it is different from VE’s default backup directory.

You can also combine the -s and --backup-path options to back up VEs to any existing directory on a remote node. For example:

1

# prlctl backup MyVM -s user:passwd@192.168.0.10 --backup-path "/vz/mybackupdir"

If, however, the remote node runs an older version of Virtuozzo Server that does not support the --backup-path option, the backup will be placed in the default backup directory on the remote node.

Note the following:

• You can back up both running and stopped virtual machines and containers.
• Creating a consistent backup of a running virtual machine requires the Virtuozzo guest tools to be installed in that virtual machine.
• You cannot back up virtual machines with attached physical HDDs, mounted ISO or floppy disk images, etc.
• If you restore a VM or container from backup and back it up again, a full backup will be created. The following backups will be incremental.
• Incremental backups will not be created if you manage backups with more than one solution that uses the Virtuozzo API, e.g., prlctl backup and some other.
• Container images are compacted at 02:00 every night by pcompact launched by a cron job. This process locks container images and prevents any other processes from opening them for writing. As a result, backup processes on locked images may fail. To avoid this, you can either reschedule backups or the pcompact job.

Improving Backup Performance

A virtual environment with a write-intensive workload may write to its snapshot blocks faster than they are backed up, so writing is slowed down to the backup speed. Besides, if the backup process stalls, the virtual environment also stalls. To avoid this, the original block can be copied to a temporary intermediate snapshot called a reversed delta and then rewritten without delay.

To switch to backups with reversed delta, run

1

# prlsrvctl set --backup-mode push-with-reversed-delta

Temporary snapshots will be saved to the virtual environment’s home directory by default. To set a different location, use the command

1

# prlsrvctl set --backup-tmpdir <path>

Temporary snapshots will be written with the O_DIRECT flag to avoid wasting node’s cache, so the directory must allow O_DIRECT writes.

To revert to direct write backups, run

1

# prlsrvctl set --backup-mode push

Or you can back up a virtual environment with the --no-reversed-delta option. For example:

1

# prlctl backup MyVM --no-reversed-delta

When direct write backups are created, the amount of page cache available to them is limited to 512 MB to reduce wasting node’s cache.

Listing Virtual Machine and Container Backups

You can list backups on the node with the prlctl backup-list command. For example:

1
2
3

# prlctl backup-list
              ID  Backup_ID         Node        Date                 Type  Size
{c1dee22f-<...>}  {209d54a0-<...>}  test.local  09/09/2020 10:19:32     f  411566405

If no options are specified, the command lists backups that reside in the default backup directory set for the VE or, if that has not been done, for the node. To list backups residing in a custom directory, add the --backup-path option to the command. For example:

1
2
3
4

# prlctl backup-list --backup-path "/vz/mybackupdir"
              ID Backup_ID        Node        Date                 Type       Size
{c1dee22f-<...>} {0d882beb-<...>} test.local  09/09/2020 14:03:25     f  411566405
{eab1e7fd-<...>} {33002958-<...>} test.local  09/09/2020 14:03:41     f     229376

To list backups for a specific VE, add the VE name or UUID to the command. For example:

1
2
3

# prlctl backup-list MyCT --backup-path "/vz/mybackupdir"
              ID Backup_ID        Node        Date                 Type       Size
{c1dee22f-<...>} {0d882beb-<...>} test.local  09/09/2020 14:03:25     f  411566405

You can also add the -s option to list backups residing on a remote node. For example:

1
2
3
4

# prlctl backup-list MyCT -s user:passwd@192.168.0.10 --backup-path "/vz/myremotebackupdir"
              ID Backup_ID          Node         Date                 Type       Size
{c1dee22f-<...>} {c91b57a6-<...>}   test.remote  09/09/2020 14:03:25     f  411566405
{c1dee22f-<...>} {c91b57a6-<...>}.2 test.remote  09/09/2020 18:43:40     i     250016

If, however, the remote node runs an older version of Virtuozzo Server that does not support the --backup-path option, the command will list backups residing in the default backup directory on the remote node.

In any case, the command shows the following information on the listed backups:

If required, you can filter the backup list with the --vmtype ct|vm|all option that only shows backups of containers, virtual machines, or both. To list only backups created on the local node, use the --localvms option.

Restoring Virtual Machines and Containers from Backups

Local or remote backups of virtual machines and containers can be restored with the prlctl restore command.

The following rules and considerations apply:

• Restore commands are run on the node where the backups will be restored.

• Only stopped virtual machines and containers can be restored from backup.

• Virtuozzo 6 backups can be restored to Virtuozzo Server 7 nodes (with conversion to Virtuozzo Server 7 format).

  • Backups of virtual machines and containers with guests unsupported in Virtuozzo Server 7 may not be restored correctly (see Virtuozzo Server Supported Guest Operating Systems Guide).
  • VZFS-based containers must be converted to ploop format and backed up again before they can be restored to Virtuozzo Server 7.

For example, to restore a backup of the virtual machine MyVM with the UUID a53f1184-333e-41cf-b410-2ec8ffea67d4, run either command:

1
2

# prlctl restore MyVM
# prlctl restore a53f1184-333e-41cf-b410-2ec8ffea67d4

If multiple backups of a VE exist, the latest one will be restored. To restore a particular backup, specify its UUID with the -t option. For example:

1

# prlctl restore -t 24a3011c-8667-4870-9e11-278f1398eab0

By default, the command looks for backups in the default backup directory set for the VE or, if that has not been done, for the node. To restore a VE from a backup residing in a custom directory, use the --backup-path option. For example:

1

# prlctl restore MyVM --backup-path "/vz/myvmbackupdir"

Add the -s option if the backup is on a remote node. For example:

1

# prlctl restore MyVM -s user:passwd@192.168.0.10 --backup-path "/vz/myvmbackupdir"

If, however, the remote node runs an older version of Virtuozzo Server that does not support the --backup-path option, the command will restore VE’s latest backup from the default backup directory on the remote node.

If backups are stored remotely and the VE to be restored does not exist on the node, you can restore it by specifying its UUID as well as backup server’s IP address or hostname. For example:

1

# prlctl restore -t 24a3011c-8667-4870-9e11-278f1398eab0 -s 192.168.0.10

If the VM or container exists on the node (e.g., has been restored from a remote backup once), you can restore it from the latest backup by its name or UUID.

If necessary, you can transfer remote backups to the local node and restore them locally. To do this:

1. Find out the backup directories on the source and destination servers.

   If the VE has its own backup directory, locate it with prlctl list. For example:

   1 2	# prlctl list -i MyVM | grep "Backup path" Backup path: /vz/myvmbackupdir

   Otherwise, locate node’s default backup directory. For example:

   1 2	# prlsrvctl info | grep "Backup path" Backup path: /vz/mybackupdir

2. Copy backups to a directory on the local node. Or, if you keep backups on a network storage, mount it to a local directory.

3. Restore the backups with the prlctl restore -t command as shown in the example above. Specify the local directory where you have copied the backups with --backup-path.

To reduce downtime of a restored virtual machine, you can restore it live. In this case, the restored virtual machine is started right after launching the restore process. To do it, use the prlctl restore --live command. For example:

1

# prlctl restore MyVM --live

Deleting Virtual Machine and Container Backups

You can delete backups with the prlctl backup-delete command.

To delete all backups of a VM or container, specify only the VM or container name or UUID:

1

# prlctl backup-delete MyVM

To remove a specific VM or container backup, provide the VM or container name or UUID as well as the backup ID (you can find out these IDs with the prlctl backup-list command). For example:

1
2
3
4

# prlctl backup-list MyVM
              ID  Backup_ID         Node        Date                 Type  Size
{c1dee22f-<...>}  {209d54a0-<...>}  test.local  2011-05-30 10:19:32     f  411566405
# prlctl backup-delete MyVM -t 209d54a0-e3b8-4a03-9ca8-d4cc7a2a27ca --keep-chain

With the --keep-chain option specified, the remaining backup chain will be preserved after specific backups are deleted from it.

To delete backups stored on a remote node, use the -s option. For example:

1

# prlctl backup-delete MyVM -s user:passwd@192.168.0.10

To delete backups stored in a custom directory, use the --backup-path option. For example:

1

# prlctl backup-delete MyVM --backup-path "/vz/myvmbackupdir"

You can also combine these two options to remove backups from a custom directory on a remote node.

Backing up Entire Servers

In addition to backing up single virtual machines and containers, you can create backups of all virtual environments on the node with the prlsrvctl backup command. For example:

1
2
3
4
5
6
7
8
9

# prlsrvctl backup -f
Backing up the CT MyCT
...
The CT has been successfully backed up with backup id {b14ec76d-c0e2-432f-859a- \
4727c0042065}
Backing up the VM MyVM
...
The VM has been successfully backed up with backup id {746dba2a-3b10-4ced-9dd6- \
76a2blcl4a69}.

Note the following:

• You can back up both running and stopped virtual machines and containers.
• Creating consistent backups of running virtual machines requires the Virtuozzo guest tools to be installed in those virtual machines.
• You cannot back up virtual machines with attached physical HDDs, mounted ISO or floppy disk images, etc.

Attaching Backups to Virtual Machines and Containers

To read the contents of a virtual machine or container backup, you can attach it to a virtual machine or container as a virtual hard disk.

Note the following:

• Only local backups can be attached.
• The attached backup is writable so that the file system can process its journal on mount. However, all changes will be discarded when the backup is detached. The amount of data that can be written to the attached backup is limited to 256MB.
• Attached backups are not preserved during clone, backup, and snapshot operations.
• Use the nouuid mount option to attach backups to VMs with XFS partitions (e.g., VzLinux 8.4). Doing so will help avoid XFS UUID collisions.

Attaching Backups to Linux Virtual Machines

1. Make sure that the prl_backup and kpartx utilities are installed in the virtual machine the backup will be attached to. The prl_backup utility is provided by Virtuozzo guest tools.

2. Obtain the ID and file name of the backup to attach. You can do this with the prlctl backup-list command. For example:

   1 2 3 4 5

   # prlctl backup-list vm2 -f ... Backup_ID: {0fcd6696-c9dc-4827-9fbd-6ee3abe017fa} ... Name: harddisk.hdd.qcow2

3. Attach the backup as an HDD to the Linux VM you will access the backup from. You can do this with the prlctl set --backup-add command. For example:

   1 2 3 4 5

   # prlctl set vm1 --backup-add {0fcd6696-c9dc-4827-9fbd-6ee3abe017fa} \ --disk harddisk.hdd.qcow2 Creating hdd1 (+) sata:2 real='backup:///{0fcd6696-c9dc-4827-9fbd-6ee3abe017fa}/ \ harddisk.hdd.qcow2' backup='{0fcd6696-c9dc-4827-9fbd-6ee3abe017fa}' \ disk='harddisk.hdd.qcow2'

   If the backup contains multiple disks and you need to connect them all, omit the --disk option.

4. Obtain the name of the newly attached device, which is disabled at the moment, using the prl_backup list command. For example:

   1 2 3 4

   # prlctl exec vm1 prl_backup list ... List of disabled attached backups: [1] /dev/sdc

5. Enable the backup with the prl_backup enable command. For example:

   1	# prlctl exec vm1 prl_backup enable /dev/sdc

6. Optionally, make sure the backup is now enabled, using the prl_backup list -e command. For example:

   1 2 3 4 5 6 7 8

   # prlctl exec vm1 prl_backup list -e List of enabled attached backups: [1] /dev/sdc (/dev/mapper/backup1) NAME TYPE SIZE FSTYPE UUID MOUNTPOINT backup1 (dm-3) dm 64G |-backup1p1 (dm-4) part 500M ext4 1ac82165-113d-40ee-8ae2-8a72f62d95bf -backup1p2 (dm-5) part 63.5G LVM2_mem Zw9QiY-BiU5-o8dn-ScTK-vOZx-KujW-wbgmS3

Now you can mount the required backup part as a file system.

Use mount to mount the ext4 part. For example:

1

# prlctl exec vm1 mount /dev/mapper/backup1p1 /mnt/backup1p1

You can now access the backup part contents at /mnt/backup1p1.

To avoid an XFS UUID collision, mount the XFS partitions with the nouuid mount option:

1

prlctl exec vm1 mount -o nouuid /dev/mapper/backup1p1 /mnt/backup1p1

Do the following to mount the LVM2_member part:

1. Assign the new volume group a new name so it can coexist with other volume groups. You can do this with the vgimportclone command. For example:

   1 2 3 4 5 6

   # prlctl exec vm1 vgimportclone -n backup1p2 /dev/mapper/backup1p2 ... Volume group "VolGroup" successfully renamed to "backup1p2" ... Found volume group "backup1p2" using metadata type lvm2 ...

   If the guest operating system is RockyLinux 9.3, use an additional option --importdevices when running the vgimportclone command. For example:

   1	prlctl exec vm1 vgimportclone --importdevices -n rl-backup /dev/mapper/backup1p2

2. Obtain the list of mountable logical volumes with the lvs command. For example:

   1 2 3 4

   # prlctl exec vm1 lvs | grep backup1p2 lv_home backup1p2 -wi-------11.54g lv_root backup1p2 -wi------- 50.00g lv_swap backup1p2 -wi-------1.97g

3. Activate the required logical volume with the lvchange -ay command. For example:

   1	# prlctl exec vm1 lvchange -ay /dev/backup1p2/lv_root

4. Mount the logical volume as a file system. For example:

   1	# prlctl exec vm1 mount /dev/backup1p2/lv_root /mnt/backup1p2

You can now access the backup part contents at /mnt/backup1p2.

Attaching Backups to Windows Virtual Machines

1. Obtain the backup ID and file name. For example:

   1 2 3 4 5

   # prlctl backup-list vm2 -f ... Backup_ID: {cff742a9-f942-41c5-9ac2-ace3b4eba783} ... Name: harddisk.hdd.qcow2

2. Attach the required backup as an HDD to the Windows virtual machine you will access the backup from. For example:

   1 2 3 4 5

   # prlctl set vm1 --backup-add {cff742a9-f942-41c5-9ac2-ace3b4eba783} \ --disk harddisk.hdd.qcow2 Creating hdd1 (+) sata:2 real='backup:///{cff742a9-f942-41c5-9ac2-ace3b4eba783}/ \ harddisk.hdd.qcow2' backup='{cff742a9-f942-41c5-9ac2-ace3b4eba783}' \ disk='harddisk.hdd.qcow2'

The attached backup will appear as a ready-to-use disk in the Windows virtual machine.

Attaching Backups to Linux Containers

1. Obtain the backup ID and file name with the prlctl backup-list -f command. For example:

   1 2 3 4 5 6

   # prlctl backup-list 102 -f ... Backup_ID: {d70441dd-f077-44a0-8191-27704d4d8fdb} ... Name: root.hdd.qcow2c ...

2. Attach the backup as an HDD to the Linux container you will access the backup from. You can do this with the prlctl set --backup-add command. For example:

   1 2 3 4 5

   # prlctl set MyCT --backup-add {d70441dd-f077-44a0-8191-27704d4d8fdb} \ --disk root.hdd.qcow2c Creating hdd1 (+) sata:0 real='backup:///{d70441dd-f077-44a0-8191-27704d4d8fdb}/ \ root.hdd.qcow2c' backup='{d70441dd-f077-44a0-8191-27704d4d8fdb}' \ disk='root.hdd.qcow2c'

3. Using the backup ID, identify the ploop device corresponding to the backup. For example:

   1 2 3

   # ploop list | grep {d70441dd-f077-44a0-8191-27704d4d8fdb} ploop28261 /buse/{8417a267-0919-4c8f-a31d-68671358d6a8}_ \ {d70441dd-f077-44a0-8191-27704d4d8fdb}_root.hdd.qcow2c/content

4. Mount the logical volume as a file system. For example:

   1	# prlctl exec MyCT mount /dev/ploop28261p1 /mnt/backup1

You can now access the backup contents at /mnt/backup1.

Detaching Backups from Virtual Machines and Containers

• To detach all virtual disks from all backups attached to a virtual machine or container, use the prlctl set --backup-del all command. For example:

  1	# prlctl set vm1 --backup-del all

• To detach all virtual disks from a specific backup attached to a virtual machine or container, use the prlctl set --backup-del <backup_ID> command. For example:

  1	# prlctl set vm1 --backup-del {e13561bb-5676-49bd-a935-ae0145eb0229}

• To detach a specific virtual disk from any of the backups attached to a virtual machine or container, delete the disk with the prlctl set --device-del hdd<N> command. For example:

  1	# prlctl set vm1 --device-del hdd1

Best Practices for Handling Page Cache When Using Third-Party Backup Software

Performing Backups from the Page Cache Angle

Backup software by its nature must, first, read all files from a disk or directory to back up and, second, write that data somewhere. If no precautions are taken, both steps generate huge amounts of page cache, as both file reads and writes send their data to the page cache.

As mentioned before, because RAM size is normally much smaller than disk size, backup forces the memory management subsystem to drop the existing page cache used by processes and replace it with the page cache generated by backup. At that, files read and written during backup are unlikely be read by other processes anytime soon. As a result, node’s RAM fills up with useless page cache, while the useful page cache is dropped.

Thus when writing backup software, one needs to make sure to keep the useful page cache on the node. That is, drop page cache generated by backup. This can be done in two ways:

1. Have backup software read and write files in the O_DIRECT mode.

   Always consider this when creating new backup tools. If you are writing a simple script, this may sometimes be possible as well.

   • If you need to copy a large file, e.g., a snapshot, use dd with the O_DIRECT flag instead of cp:

     1	# dd if=/path/to/src_file of=/path/to/dst_file bs=8M iflag=nocache oflag=direct

   • If you need to create an archive and write it somewhere, most often you can avoid page cache generation for the write part if the archiver can write to stdout. Just write the archive to disk with dd:

     1	# archive_and_write_to_stdout | dd of=/path/to/dst_file bs=8M oflag=direct

     It may not be that easy with the read part, however. For example, tar found in the mainstream tree still does not support the --o_direct option, although patches exist that implement this functionality. It is like that with rsync too. There are patches, pull requests, and even custom builds that add fadvise (FADV_DONTNEED) support to it, but none of those have been accepted to the rsync mainstream tree.

     One interesting tool is nocache. It tries to minimize the effect of applications on the Linux file system cache. It intercepts open and close system calls and calls posix_fadvise with the POSIX_FADV_DONTNEED parameter. For example:

     1	# nocache tar cz -O /path/to/backup_dir | dd of=/path/to/dst_file bs=8M oflag=direct

     or

     1	# nocache rsync -aiv --rsync-path='nocache rsync' /src/ host:/dest/

     The --rsync-path option makes the remote side use a nocache wrapper as well.

     The nocache tool provides a workaround for small and medium-sized files (note that by default it only handles files up to 1MB in size). However by the time tar, rsync, or another utility closes a large source file and nocache calls fadvise() to drop it, it is too late, as the file contents have already reached node’s page cache, replacing useful data in it. More details are provided in this post.

2. Run backup software in a cgroup with a limited page cache (Virtuozzo Server 7).

   If your backup tools do not support the O_DIRECT file access mode, you can use a feature of Virtuozzo Server 7: limiting of page cache for particular memory cgroups (see Limiting Memory Pressure from Node Processes).

Limiting Memory Pressure from Node Processes

To limit memory pressure on the hardware node resulting from fast page cache generation by node processes (like backups), one needs to do the following:

1. Create a separate memory cgroup.
2. Place the desired process into the cgroup.
3. Disable tcache for the cgroup.
4. Limit page cache for the cgroup (a feature of Virtuozzo Server 7).
5. After the process that generates page cache fast ends, remove the memory cgroup.

The easiest way is to use the script nocache.sh that performs all these steps.

Usage:

1
2
3
4

# nocache.sh
[-limit LIMIT]
[-pcgroup PARENT_CGROUP]
<command>

Where:

• -limit LIMIT is the page cache limit in megabytes to apply to the process. The default is 256MB.
• -pcgroup PARENT_CGROUP is the parent memory cgroup, inside which the cgroup with the limited page cache will be created. The current memory cgroup is used by default.
• <command> is the command with options (or script) that generates page cache.

Examples:

1
2
3

# nocache.sh backup_script.sh

# nocache.sh -limit 128 -pcgroup /machine.slice backup_script.sh

If you cannot use the script for some reason, you can perform the same steps manually:

1. Create a separate memory cgroup.

   1	# mkdir /sys/fs/cgroup/memory/user.slice/limited_pagecache

   You can replace limited_pagecache with any unique string.

2. Place the desired process with PID $PID into the cgroup.

   1	# echo $PID >> /sys/fs/cgroup/memory/user.slice/limited_pagecache/tasks

1

# echo $$ >> /sys/fs/cgroup/memory/user.slice/limited_pagecache/tasks

3. Disable tcache for the cgroup.

   1	# echo 1 > /sys/fs/cgroup/memory/user.slice/limited_pagecache/memory.disable_cleancache

4. Limit page cache for the cgroup to $LIMIT bytes (a feature of Virtuozzo Server 7):

   1	# echo $LIMIT > /sys/fs/cgroup/memory/user.slice/limited_pagecache/memory.cache.limit_in_bytes

   The limit depends on the application. For example, 256MB is enough for tar + gzip.

5. After the process that generates page cache fast ends, remove the memory cgroup.

   1	# rmdir /sys/fs/cgroup/memory/user.slice/limited_pagecache

   Note that a cgroup with processes cannot be removed. If on step 2 you put the process PID into the cgroup, first remove it from the cgroup with:

   1	# echo $$ >> /sys/fs/cgroup/memory/user.slice/tasks

Frequenty asked questions:

• Q: May my application fail to work due to memory allocation issues if run with nocache.sh?

  A: No. The wrapper does not limit the size of anonymous memory allowed for the application, so memory allocation will not fail.

• Q: May nocache.sh affect the performance of my application?

  A: It depends on the application. Backup-like programs that read each file only once will not slow down. Applications that read same files two or more times may slow down.