Category: Backup

Home / Category: Backup

CommVault

This article covers the implementation of CommVault archiving for the O365 Exchange Online environment. As is generally known, in these times, more and more companies are moving their applications to Cloud, including Sharepoint and Exchange.

In a traditional Exchange environment (on-prem), CommVault software is used to perform backups of Exchange databases (DAG’s). A database can contain up to a hundred or thousand mailboxes. The downside of such an approach is that when performing a restore of a single mailbox, the whole database needs to be restored.

Unlike the case with Exchange backups on-prem, archiving of Exchange Online mailboxes is done in a more granual way. In short, each mailbox / e-mail content is archived using CommVault access nodes. Hence, restores are also more granular, meaning you can restore a single mail or mailbox.

Exchange Online Requirements

At this time of writing, I’m using CommVault version 11 Service Pack 24. CommVault documentation page provides an extended explanation of pre-requisites and best-practices, however in this article highlights the most important ones. I’ll be covering every step using CommCell Console, however there is also Command Center available with most of the tasks automated.

Basically, what you’ll need to set up the environment is the following:

  1. An CommServe environment (assuming you already haver one, as you landed on this page)
  2. Media Agent(s)
  3. Access Node(s) or proxy servers that establish the connection with the O365/Exchange Online API’s
  4. An Index Server – preferably on separate VM
  5. A Storage Policy for your Exchange Online archives

In short, the flow between your backup target location and O365 API’s goes as following:

O365 (Cloud) API’s > CV Access Nodes > (optionally Media Agents) > Storage Library

Service Accounts and Application ID’s

In order to connect, discover and perform archive operations of Exchange Online we need to have some administrative & privileged accounts.

  • Local service account: This is a local admin account used on the access nodes
  • Exchange Online Service account: This is an Exchange Online (O365), global admin account and is used to discover mailboxes from the tenant
  • Application ID‘s are used to perform archive operations in Modern Authentication mode. When using Command Center, you can register application ID’s automatically by using the custom toolkit or you can register them manually by following Registering Exchange Online with Azure.

Ports

Assuming your O365 environment is hosted in public (All Regions) hosting at Microsoft. You’ll need to have the following ports and URL’s allowed on your proxy/firewall (if any in-between) devices:

Protocol: https

Ports: 80, 443

URLs:

  • https://www.office.com/
  • https://outlook.office365.com/Powershell-LiveID
  • https://outlook.office365.com/EWS/Exchange.asmx
  • https://login.microsoftonline.com/
  • https://graph.microsoft.com/

CommVault Online Access Nodes

An Access or Proxy node is the edge server used to establish the connection with O365 API’s and perform the archiving jobs. Depending on your needs, you can always start off with a single access node, then scale out to more nodes. Find below recommendations of an Access Node in a setup using Content Indexing.

EnvironmentMediumLarge
Mailboxes5,00010,000
Messages per day500,0001 million
Guidelines
Access nodesNormal availability: 1High availability: 2 or moreNormal availability: 1High availability: 2 or more
CPU or vCPU for the access nodes8 cores16 cores
RAM for the access nodes16 GB32 GB
Streams per access node1020
Azure apps510
Content Indexing on Access Nodes

CommVault Index Server

An index server is used to store metadata, perform content indexing and search mailboxes for message when requesting a restore. You can install the indexing feature on an access node, but it is highly recommended that you create the index cache in a separate server.

EnvironmentSmallMediumLarge
Application5 TB15 TB25 TB
Mailboxes4002,0005,000
Objects per node (estimated)50 million150 million250 million
Guidelines
CPU or vCPU8 cores16 cores16 cores
RAM16 GB32 GB64 GB
Index disk2 TB6 TB10 TB
Index disk IOPSMinimum: 800Minimum: 1,600Minimum: 2,400
Index Server using Content Indexing

Archiving Policy

  • In CommCell console go to Policies > Configuration Policies > Exchange Policies
  • Right click on it and select New Archiving Policy
  • Fill in the name and the desired information and finish by clicking OK.

CommVault Exchange Online software installation

Make sure to download the latest CommVault software before starting with the installation and configuration of Exchange Online nodes.

Access Node

The setup on the access node is an easy and straight forward installation. The only thing that you need to pay attention to is the selection of CommVault roles. When prompted, select Office365 as seen in the figure below.

O365 Role CommVault software installation

Index Server

The installation of the Index Server is pretty much the same as the access node, however, there are some things to pay attention to when installing it. Make sure to create an extra disk on the server of 2 TB.

  • When prompted if you want to Create a new CommCell, Join and existing CommCell or Advanced selection select the last option Advanced selection.
  • Under Window Packages, tick off the following features: Index Store, Index Gateway and Content Extractor.
  • The Installation path can be left as default.
  • Index Cache path: select the extra disk you created previously, in my case the path becomes: E:\Program Files\CommVault\ContentStore\IndexCache.

Creating and configuring Exchange Online client

Before we create the Exchange Online Client, we need to have the Indexing Server ready.

Creating Index Server

In CommCell Console, expand Client Computer Groups, right click on Index Server group and select New Index Server. Under General tab, give the index a name, e.g.: O365-ExchangeOnline
– Check Enable Cloud and enter Index Directory, e.g.: D:\Index\
– Uncheck Enable Cloud

Under Roles tab select Exchange Index and click on Include >
Finally, under Nodes tab, click on Add and and add the server name of the Index Server.

Creating Exchange Online client

  • In CommCell Console, right click on Client Computers and select New Client
  • Under Application, click on Exchange Mailbox and select User Mailbox.

The wizard consists of different settings, which can be defined according to your needs.

General tab

  • Fill in the client name, e.g.: STORCOM_EXO
  • Storage Policy: An earlier defined storage policy. See CommVault documentation on how to create one.
  • Index Server: The name of the Index Server you created previously
  • Job Results Directory: This is a shared network path. I usually create a shared folder on the first access node, for example: \\STORCOMSRV01\JobResults\

Access Nodes

  • Click on Add to add one or more access nodes

Environment Type

  • I assume you have a O365 environment which is independent of your on-prem environment, in this case select Exchange Online (Access through On-premises Active Directory), in other cases use Exchange Hybrid (or Exchnage On-Premises)
  • Select Use Modern Authentication

When using Modern Authentication, Application ID that you will create are used to perform archive operations.

Azure App Details

  • Leave Cloud Region as Default (Global Service)
  • Click on Add and start adding Application ID’s that you created previously.

Service Account Settings

At this step you need at least two service accounts. One will be a local admin account on your access nodes, and the second one is a Exchange global admin account in Azure.

  • Click on Add and start adding your service accounts

Other tabs (Other/Advanced, AD Server, Security, Activity Control) are optional and to be used in specific environments.

Discovering Exchange Online mailboxes and running the first Archiving job

Once you have completed the above steps, we are one step closer to starting the first archiving job. In order to launch the backup (archiving job), we first need to tell CommVault what mailboxes to archive.

Discovering mailboxes

  • Go to the Exchange client (named: O365-ExchangeOnline) we created previously
  • Expand Exchange Mailbox, and User Mailbox and select the subclient, named by default usermailbox
  • On the bottom click on Mailboxes
  • Right click on the page and select New Association > User
  • Click Configure and then Discover
    At this point the discovery of mailboxes will start running in the background. In my case it took 10 to 15 minutes for the discovery process to finish.
  • Repeat same steps after 10 to 15 minutes until the mailboxes are discovered.
  • Once done, select them all, uncheck Perform discover operation in cached mode and click OK
  • Finally under Policies tab, select the archiving policy we created previously and we are done.

First archiving job

After a successful completion of the above steps, we are ready to start the first archiving job.

  • In CommCell Console click on the Exchange Online client, under User Mailbox select the subclient
  • Right click on the subclient usermailbox and click on Archive
  • The first archiving operation will be always a full
  • Finally click on OK to start the job.

Any suggestion or question? Leave a reply below, or feel free to contact us. Subscribe to our mailing list for the latest updates.

CommVault

CommVault backups might fail from time to time due to a bad VSS functioning. There is plenty of Microsoft articles on how to deal with VSS Snapshot failures. In this article we will focus on a particular error that occurs while a VSS snapshot fails to be taken.

Job error description

Error Code: [19:857]
Description: The job has failed because the VSS snapshot could not be created
Source: spcvma54, Process: clBackup

Troubleshooting

First of all I will make sure that the following is ticked off:

  1. No errors are seen in the VSS Providers list
    vssadmin list providers
  2. No errors are found in the VSS Writers list
    vssadmin list writers
  3. There’s enough free space to create a Shadow Copy
    You can test by creating a shadow copy manually. Eventually set the Maximum size to “No Limit”.
  4. Check Event Viewer > Windows Logs > Application.

In my case, it was the Event Viewer that shed a light to the issue resolution. Among other VSS events, there was this little “innocent” error indicating a problem with one of the Shadow Copy Providers, eventhough this was not listed in the command line.

Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {3e02620c-e180-44f3-b154-2473646e4cb8} [0x80040154, Class not registered
].

Operation:
Obtain a callable interface for this provider
Add a Volume to a Shadow Copy Set

Context:
Provider ID: {74600e39-7dc5-4567-a03b-f091d6c7b092}
Class ID: {3e02620c-e180-44f3-b154-2473646e4cb8}
Snapshot Context: 25
Execution Context: Coordinator

Solution

From the error logs above we see that there’s an issue with the Shadow Copy Provider with CLSID: 3e02620c-e180-44f3-b154-2473646e4cb8

  1. Open Registry Editor and navigate to: http://hkey_local_machine/SYSTEM/CurrentControlSet/services/VSS/Providers/
  2. It is always recommended to back up the registry keys, do so for your safety 😉
  3. Find the registry key which corresponds with the CLSID seen from the Event Viewer. In my case it was Hyper-V IC Software Shadow Copy.
  4. Delete this key (or any other that reflects with erros in the Event Logs)
  5. Reboot the server and backups should work again.

You can check the creation of a shadow copy by using the following command:

C:\windows\system32>vssadmin list shadows

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

(C) Copyright 2001-2013 Microsoft Corp.

Contents of shadow copy set ID: {9d7b650a-377f-4e49-8e92-06d025abcf8b}
   Contained 1 shadow copies at creation time: 23/12/2020 10:39:21
      Shadow Copy ID: {d6995554-1430-4eb8-b7fa-e454cabaeeb5}
         Original Volume: (E:)\\?\Volume{1fe0ecbf-65fc-45cb-9f3e-891a63280da6}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1

         Originating Machine: STORSERV01.STORCOM.COM
         Service Machine: STORSERV01.STORCOM.COM
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: DataVolumeRollback
         Attributes: Persistent, No auto release, No writers, Differential

 

Any suggestion or question? Leave a reply below, or feel free to contact us. Also make sure to subscribe to our mailing list to get the latest updates.

Recently I came across an interesting application consistent backup issue of Hyper-V virtual machines in CommVault. The backup jobs would partly run or complete with one or more errors. According to the error log shown in CommVault, I would start by checking the free space on the Hyper-V host and within the virtual machines. However, this was not the case here as both machines had enough free space. Hence I started digging further into CommVault snapshots.

Error Code: [91:155]
Description: The system or provider has insufficient storage space. If possible delete any old or unnecessary persistent shadow copies and try again.
Source: <server name>, Process: vsbkp

Diagnosis

Checking CommVault logs under %Programfiles%\CommVault\Simpana\Log Files couldn’t shed light on the root cause. The next thing I had in mind was to check for any old checkpoints in Hyper-V Manager. An easy way to collect all the available checkpoints on a Hyper-V server is by using the following Powershell command:

Get-VMSnapshot -VMName *

Or even easier, to collect all checkpoint available on all cluster nodes you could use the following command.

Get-VM -ComputerName (Get-ClusterNode -Cluster %clustername%) | Get-VMSnapshot

Powershell command to get available snapshots on all cluster nodes
Powershell command to get available snapshots on all cluster nodes

From my experience with backups, I can say that CommVault sometimes can struggle taking backups of virtual machines that have checkpoints available.  You could remove unused checkpoints, however, it might happen in very rare situations.

Digging further into the Windows Logs of the Hyper-v host (Microsoft-Windows-Hyper-V-VMMS/Admin) I noticed a ‘sneaky and innocent’ error referring to a .avhdx file of a snapshot that does no longer exist.

The absolute path ‘C:\ClusterStorage\<CSVname>\<VMName><DiskGUID>.avhdx’ is valid for the “Hard Disk Image pool, but references a file that does not exist.

Event log showing the missing reference of the file
Event log showing the missing reference of the file

The missing file could be seen on the Hyper-V Manager, the virtual machine was still referring to inexisting CommVault snapshots in Hyper-V.

Hyper-V VM referring to a missing file
Hyper-V VM referring to a missing file

Following event ID’s from Windows Event Viewer could be interesting to look for: 10172, 32902

Solution

Restarting Hyper-V Virtual Machine Management (vvms) service will solve the problem. I did it using graphical interface, and a scary warning appeared asking me if I wanted to ‘Turn Off’ the service. Finally, it just halts the management service and does not affect the running VM’s.

Stop VMM service warning
Stop VMM service warning

The same operation can be accomplished using the following Powershell commands:

stop-service -name vmms
start-service -name vmms

Any suggestion or question? Leave a reply below, or feel free to contact us. Make sure to subscribe to our mailing list to get the latest.