Configuring Automatic Memory Management Policies

Virtuozzo Server can automatically manage VM and container memory. It uses the memory management service VCMMD that does the following:

• Manages kernel same-page merging (KSM) for VMs
• Tunes huge pages transparently
• Balances containers and VMs between NUMA nodes
• Adjusts memory limits and guarantees for containers, VMs, and Virtuozzo Storage services

All of these optimizations are applied by means of memory management policies. Setting the correct policy may provide performance and density benefits to your Virtuozzo Server installation.

Two policies are available:

• density (default): A universal policy whether memory is overcommitted or not. It provides better density than the performance policy. It also provides higher performance if memory is overcommitted or the same performance if it is not. This policy does the following:

  • Sets VM memory guarantees to 40%, allowing more memory to be overcommitted (see Configuring Virtual Machine Memory Guarantees). The container memory guarantees remain at 0% (see Configuring Container Memory Guarantees).
  • Binds VMs and containers to server’s NUMA nodes. Rebalances them based on their memory and CPU usage when the difference in loads on the busiest and freest nodes exceeds 50%.
    This policy is set automatically if a license is not installed or has expired.

• performance: Best for servers with virtual environments that can fit into NUMA nodes. This policy does the following:

  • Sets VM memory guarantees to 80% (see Configuring Virtual Machine Memory Guarantees). The container memory guarantees remain at 0% (see Configuring Container Memory Guarantees).
  • Binds VMs and containers to server’s NUMA nodes. Rebalances them between NUMA nodes every 5 minutes, based on each node’s free RAM and CPU load as well as each VE’s CPU and RAM usage. Doing so enables the benefits of the NUMA architecture.
    This policy does not allow overcommitting memory.

You can switch policies on a server as follows:

1. Change the policy and check the result using the following commands:

   1 2 3

   # prlsrvctl set --vcmmd-policy density # prlsrvctl info | grep "policy" Vcmmd policy: density config=density

2. If you stopped the virtual environments or migrated them to another server, start the virtual environments or migrate them back to the server.

Restarting VCMMD

In certain situations, you may need to restart the VCMMD daemon. For example:

• When you activate a license after installing Virtuozzo Server or update an expired license.
• To permanently apply custom limits and guarantees for services residing in memory slices. See Managing Host Services with VCMMD and Managing Virtuozzo Storage Services with VCMMD.
• If you deploy Virtuozzo Storage on top of Virtuozzo Server and start its services manually without rebooting the server. See Managing Virtuozzo Storage Services with VCMMD.
• To apply changes after tweaking the VCMMD configuration file. See Managing Virtuozzo Storage Services with VCMMD.

If virtual environments are running on the server, do the following:

1. Stop all virtual environments on the server or temporarily live-migrate them to another server.

2. Restart vcmmd:

   1	# systemctl restart vcmmd

   The default performance policy will be enabled.

3. Start the virtual environments or live-migrate them back to the server, depending on what you did in step 1.

If no virtual environments are running on the server, just restarting the service is enough.

Optimizing Virtual Machine Memory Usage with Kernel Same-Page Merging

To optimize memory usage by virtual machines, Virtuozzo Server uses a Linux feature called kernel same-page merging (KSM). The KSM daemon periodically scans memory for pages with identical content and merges those into a single page. That page is marked as copy-on-write (COW). When a virtual machine needs to change that page’s contents, the kernel creates a copy of the page for that VM and changes it.

KSM enables the server to:

• Avoid swapping
• Run more virtual machines
• Overcommit virtual machine memory
• Speed up RAM and hence certain applications and guest operating systems

In Virtuozzo Server 7, pages are not merged by default. In general, merging starts when the amount of free RAM drops below 20% and stops when free RAM goes above this threshold again. It works differenly, however, depending on the policy:

• With the density policy enabled, merging works as described on server’s entire RAM, disregarding NUMA nodes.
• With the performance policy enabled, pages are merged independently within each NUMA node. This slows KSM by as many times as there are NUMA nodes on the server (e.g., by four times if there are four NUMA nodes). Unlike density, however, pages are unmerged as soon as they can fit into RAM again, leaving 20% of it free. This is done to avoid COW operations and increase server performance.

KSM is enabled by default. If you need to disable it for some reason, set KSM to false in /etc/vz/vcmmd.conf. Add the parameter if needed.

1
2
3
4
5

"LoadManager": {
    "Controllers": {
        "KSM": false,
        <...>
    },

Restart vcmmd to apply changes. See Restarting VCMMD.

Managing Host Services with VCMMD

You can set memory limits and guarantees managed by vcmmd for host services. The daemon will take them into account when allocating resources for virtual environments.

For example, to have a service named service managed by vcmmd, do as follows:

1. Place the service processes in /sys/fs/cgroup/memory/service.slice directory.

2. Register service.slice in vcmmd, specifying the desired guarantee and limit in bytes (you can also use K for kibibytes, M for mebibytes, or G for gibibytes). For example:

   1	# vcmmdctl register SRVC service.slice --guarantee 100000000 --limit 100000000 --swap 100000

   These settings will last until a server reboot. A way to set permanent limits is provided further in this section.

3. Activate the service:

   1	# vcmmdctl activate service.slice

List memory slices to check that the service is managed with vcmmd. For example:

1
2
3
4

# vcmmdctl list
name                                     type  active  guarantee     limit    swap
<...>
service.slice                            SRVC     yes      97656     97656      97

You can also temporarily change the service memory parameters at runtime using the vcmmdctl update command.

For example, to change the service.slice parameters, run:

1

# vcmmdctl update service.slice --guarantee 200000000 --limit 200000000

List memory slices to check that the parameters have been applied. For example:

1
2
3
4

# vcmmdctl list
name                                     type  active  guarantee     limit    swap
<...>
service.slice                            SRVC     yes     195312    195312     max

To unregister a service from vcmmd at runtime, use the vcmmdctl unregister command.

Guarantees and limits set using the instructions above will be removed on host reboot. You can, however, set permanent limits and guarantees for services in system and user memory slices. For example, to make sure that you can still access the host via SSH if it becomes heavily overcommitted. Do as follows:

1. If needed, dump the current values to the VCMMD configuration file:

   1	# vcmmdctl config -f > /etc/vz/vcmmd.conf

2. Specify the minimum and maximum values in bytes in /etc/vz/vcmmd.conf (-1 stands for unlimited memory). For example:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

   { "VStorage": { <...> }, "System": { "Path": "system.slice", "Limit": { "Max": -1, "Min": 0, "Share": 0.7 }, "Guarantee": { "Max": 1073741824, "Min": 536870912, "Share": 0.25 }, "Swap": { "Max": 0, "Min": 0, "Share": 0 } }, "User": { "Path": "user.slice", "Limit": { "Max": -1, "Min": 0, "Share": 0.7 }, "Guarantee": { "Max": 536870912, "Min": 268435456, "Share": 0.25 }, "Swap": { "Max": 0, "Min": 0, "Share": 0 } } }

3. Restart vcmmd to apply changes. See Restarting VCMMD.

4. Check that the parameters were applied. The actual guarantees and limits are calculated based on the minimum and maximum values specified in /etc/vz/vcmmd.conf.

   1 2 3 4 5

   # vcmmdctl list name type active guarantee limit swap system.slice SRVC yes 1048576 5605796 0 user.slice SRVC yes 524288 5605796 0 <...>

Adjusting the Page Cache Limit for user.slice

In addition to the memory limits and guarantees for host services, you can set the overall page cache limit for user.slice. If the node’s file system is actively used, doing so may increase read/write performance at the expense of virtual environments’ RAM speed. The caveat is that the optimal value has to be determined experimentally.

By default, the user.slice page cache limit is set to whichever value is smaller: 10% of RAM or 10 GB.

To set a different limit, change the UserCacheLimitTotal parameter in /etc/vz/vcmmd.conf. It will adjust the /sys/fs/cgroup/memory/user.slice/memory.cache.limit_in_bytes value.

Do the following:

1. Dump the current values to the VCMMD configuration file:

   1	# vcmmdctl config -f > /etc/vz/vcmmd.conf

2. Add UserCacheLimitTotal to the LoadManager section. For example, to set it to 20 GiB, in bytes:

   1 2 3 4

   "LoadManager": { <...> "UserCacheLimitTotal": 21474836480 },

3. Restart vcmmd to apply changes. See Restarting VCMMD.

Managing Virtuozzo Storage Services with VCMMD

On servers running Virtuozzo Storage, vstorage.slice/vstorage-services.slice is automatically created in the memory cgroup on server start.

If, however, you manually deployed Virtuozzo Storage on top of Virtuozzo Server and started the vstorage* services without rebooting the server, restart vcmmd to create this memory slice. Consult Restarting VCMMD.

List memory slices to check that Virtuozzo Storage services are managed by VCMMD. For example:

1
2
3
4

# vcmmdctl list
name                                     type  active  guarantee     limit    swap
<...>
vstorage.slice/vstorage-services.slice   SRVC     yes    4194304 103854973       0

You can temporarily change memory limits and guarantees for Virtuozzo Storage services at runtime using the vcmmdctl update command. For example:

1

# vcmmdctl update vstorage.slice/vstorage-services.slice --guarantee 100000000 --limit 100000000

The parameters will be reset to the default values on the next vcmmd restart or node reboot.

To configure limits and guarantees for Virtuozzo Storage services permanently, do as follows:

1. If needed, dump the current values to the VCMMD configuration file:

   1	# vcmmdctl config -f > /etc/vz/vcmmd.conf

2. Specify the values in bytes in /etc/vz/vcmmd.conf (-1 stands for unlimited memory). For example:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

   { "VStorage": { "Path": "vstorage.slice/vstorage-services.slice", "Limit": { "Max": -1, "Min": 0, "Share": 0.7 }, "Guarantee": { "Max": 1073741824, "Min": 536870912, "Share": 0.25 }, "Swap": { "Max": 0, "Min": 0, "Share": 0 } }, "System": { <...> }, "User": { <...> } }

3. Restart vcmmd to apply changes. See Restarting VCMMD.

List memory slices to check that the parameters were applied. The actual guarantees and limits are calculated based on the minimum and maximum values specified in /etc/vz/vcmmd.conf.

1
2
3
4

# vcmmdctl list
name                                   type active guarantee     limit      swap
<...>
vstorage.slice/vstorage-services.slice   SRVC    yes   1048576   5605796         0

If you use large CS journals, you may also attempt to improve storage write performance by tweaking the VCMMD configuration.

Initially, storage services can use two thirds of host’s memory. As soon as you create a virtual environment, the memory available to storage services becomes limited to 512MB per CS, which may throttle their performance. You can, however, let these services use more of host’s RAM by adjusting the StorageCacheLimitTotal value. In general, set it to at least 4 GB per CS with a journal.

For example, if you have 10 CSes with journals, set the parameter to 40 GiB by adding it to the LoadManager section of /etc/vz/vcmmd.conf:

1
2
3
4
5
6
7
8

"LoadManager": {
    "Controllers": {
        <...>
        "StorageCacheLimitTotal": 42949672960,
        <...>
    },
    <...>
},

Restart vcmmd to apply changes. See Restarting VCMMD.

Check that the changes were applied:

1
2

# vcmmdctl config | grep StorageCacheLimitTotal
        "StorageCacheLimitTotal": 42949672960,

Other parameters that you may see in the Controllers section of /etc/vz/vcmmd.conf include:

• NUMA, which switches balancing of virtual environments between NUMA nodes. It is only used when the performance policy is enabled.
• StoragePolicy, which is enabled if the node is in a Virtuozzo Storage cluster. It adjusts swappiness, manages memory limits for Virtuozzo Storage services mentioned above, and protects Virtuozzo Storage services from the OOM killer.

Both of these parameters are set automatically and need not be tweaked.