CommVault snapshots (avhdx) hanging on Hyper-V

Recently I came across an interesting application consistent backup issue of Hyper-V virtual machines in CommVault. The backup jobs would partly run or complete with one or more errors. According to the error log shown in CommVault, I would start by checking the free space on the Hyper-V host and within the virtual machines. However, this was not the case here as both machines had enough free space. Hence I started digging further into CommVault snapshots.

Error Code: [91:155]
Description: The system or provider has insufficient storage space. If possible delete any old or unnecessary persistent shadow copies and try again.
Source: <server name>, Process: vsbkp


Checking CommVault logs under %Programfiles%\CommVault\Simpana\Log Files couldn’t shed light on the root cause. The next thing I had in mind was to check for any old checkpoints in Hyper-V Manager. An easy way to collect all the available checkpoints on a Hyper-V server is by using the following Powershell command:

Get-VMSnapshot -VMName *

Or even easier, to collect all checkpoint available on all cluster nodes you could use the following command.

Get-VM -ComputerName (Get-ClusterNode -Cluster %clustername%) | Get-VMSnapshot
Powershell command to get available snapshots on all cluster nodes
Powershell command to get available snapshots on all cluster nodes

From my experience with backups, I can say that CommVault sometimes can struggle taking backups of virtual machines that have checkpoints available.  You could remove unused checkpoints, however, it might happen in very rare situations.

Digging further into the Windows Logs of the Hyper-v host (Microsoft-Windows-Hyper-V-VMMS/Admin) I noticed a ‘sneaky and innocent’ error referring to a .avhdx file of a snapshot that does no longer exist.

The absolute path ‘C:\ClusterStorage\<CSVname>\<VMName><DiskGUID>.avhdx’ is valid for the “Hard Disk Image pool, but references a file that does not exist.

Event log showing the missing reference of the file
Event log showing the missing reference of the file

The missing file could be seen on the Hyper-V Manager, the virtual machine was still referring to inexisting CommVault snapshots in Hyper-V.

Hyper-V VM referring to a missing file
Hyper-V VM referring to a missing file

Following event ID’s from Windows Event Viewer could be interesting to look for: 10172, 32902


Restarting Hyper-V Virtual Machine Management (vvms) service will solve the problem. I did it using graphical interface, and a scary warning appeared asking me if I wanted to ‘Turn Off’ the service. Finally, it just halts the management service and does not affect the running VM’s.

Stop VMM service warning
Stop VMM service warning

The same operation can be accomplished using the following Powershell commands:

stop-service -name vmms
start-service -name vmms

Any suggestion or question? Leave a reply below, or feel free to contact us. Make sure to subscribe to our mailing list to get the latest.

Share Article on:


One Response

  1. Oh boy, automatic all caps 😉

    Great post, ran into the same issue: no checkpoints, but .avhd(x) files were filling up the clustered volumes. Restarting the VM MgmT service caused the merge process to start!

Leave a Reply

Your email address will not be published. Required fields are marked *