Tag Archive : CommVault

/ CommVault

CommVault Shadow Copy Errors

CommVault backups might fail from time to time due to a bad VSS functioning. There is plenty of Microsoft articles on how to deal with VSS Snapshot failures. In this article we will focus on a particular error that occurs while a VSS snapshot fails to be taken.

Job error description

Error Code: [19:857]
Description: The job has failed because the VSS snapshot could not be created
Source: spcvma54, Process: clBackup

Troubleshooting

First of all I will make sure that the following is ticked off:

  1. No errors are seen in the VSS Providers list
    vssadmin list providers
  2. No errors are found in the VSS Writers list
    vssadmin list writers
  3. There’s enough free space to create a Shadow Copy
    You can test by creating a shadow copy manually. Eventually set the Maximum size to “No Limit”.
  4. Check Event Viewer > Windows Logs > Application.

In my case, it was the Event Viewer that shed a light to the issue resolution. Among other VSS events, there was this little “innocent” error indicating a problem with one of the Shadow Copy Providers, eventhough this was not listed in the command line.

Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} [0x80040154, Class not registered
].

Operation:
Obtain a callable interface for this provider
Add a Volume to a Shadow Copy Set

Context:
Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
Snapshot Context: 25
Execution Context: Coordinator

Solution

From the error logs above we see that there’s an issue with the Shadow Copy Provider with CLSID: 3e02620c-e180-44f3-b154-2473646e4cb8

  1. Open Registry Editor and navigate to: http://hkey_local_machine/SYSTEM/CurrentControlSet/services/VSS/Providers/
  2. It is always recommended to back up the registry keys, do so for your safety ūüėČ
  3. Find the registry key which corresponds with the CLSID seen from the Event Viewer. In my case it was Hyper-V IC Software Shadow Copy.
  4. Delete this key (or any other that reflects with erros in the Event Logs)
  5. Reboot the server and backups should work again.

You can check the creation of a shadow copy by using the following command:

C:\windows\system32>vssadmin list shadows

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2013 Microsoft Corp.

Contents of shadow copy set ID: {9d7b650a-377f-4e49-8e92-06d025abcf8b}
   Contained 1 shadow copies at creation time: 23/12/2020 10:39:21
      Shadow Copy ID: {d6995554-1430-4eb8-b7fa-e454cabaeeb5}
         Original Volume: (E:)\\?\Volume{1fe0ecbf-65fc-45cb-9f3e-891a63280da6}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1

         Originating Machine: STORSERV01.STORCOM.COM
         Service Machine: STORSERV01.STORCOM.COM
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: DataVolumeRollback
         Attributes: Persistent, No auto release, No writers, Differential

 

Any suggestion or question? Leave a reply below, or feel free to contact us. Also make sure to subscribe to our mailing list to get the latest updates.

Recently I came across an interesting application consistent backup issue of Hyper-V virtual machines in CommVault. The backup jobs would partly run or complete with one or more errors. According to the error log shown in CommVault, I would start by checking the free space on the Hyper-V host and within the virtual machines. However, this was not the case here as both machines had enough free space. Hence I started digging further into CommVault snapshots.

Error Code: [91:155]
Description: The system or provider has insufficient storage space. If possible delete any old or unnecessary persistent shadow copies and try again.
Source: <server name>, Process: vsbkp

Diagnosis

Checking CommVault logs under %Programfiles%\CommVault\Simpana\Log Files couldn’t shed light on the root cause. The next thing I had in mind was to check for any old checkpoints in Hyper-V Manager. An easy way to collect all the available checkpoints on a Hyper-V server is by using the following Powershell command:

Get-VMSnapshot -VMName *

Or even easier, to collect all checkpoint available on all cluster nodes you could use the following command.

Get-VM -ComputerName (Get-ClusterNode -Cluster %clustername%) | Get-VMSnapshot
Powershell command to get available snapshots on all cluster nodes
Powershell command to get available snapshots on all cluster nodes

From my experience with backups, I can say that CommVault sometimes can struggle taking backups of virtual machines that have checkpoints available.  You could remove unused checkpoints, however, it might happen in very rare situations.

Digging further into the Windows Logs of the Hyper-v host (Microsoft-Windows-Hyper-V-VMMS/Admin) I noticed a ‘sneaky and innocent’ error referring to a .avhdx file of a snapshot that does no longer exist.

The absolute path ‘C:\ClusterStorage\<CSVname>\<VMName><DiskGUID>.avhdx’ is valid for the “Hard Disk Image pool, but references a file that does not exist.

Event log showing the missing reference of the file
Event log showing the missing reference of the file

The missing file could be seen on the Hyper-V Manager, the virtual machine was still referring to inexisting CommVault snapshots in Hyper-V.

Hyper-V VM referring to a missing file
Hyper-V VM referring to a missing file

Following event ID’s from Windows Event Viewer could be interesting to look for: 10172, 32902

Solution

Restarting Hyper-V Virtual Machine Management (vvms) service will solve the problem. I did it using graphical interface, and a scary warning appeared asking me if I wanted to ‘Turn Off’ the service. Finally, it just halts the management service and does not affect the running VM’s.

Stop VMM service warning
Stop VMM service warning

The same operation can be accomplished using the following Powershell commands:

stop-service -name vmms
start-service -name vmms

Any suggestion or question? Leave a reply below, or feel free to contact us. Make sure to subscribe to our mailing list to get the latest.