Backup by Zextras

This chapter provides an overview of Zextras Backup, the element of the Zextras Suite in charge of data backup. A summary of the most frequent job is provided at the beginning of the chapter, along with connections to more technical sources.

The design of Zextras Backup is then explained, along with some crucial previous knowledge that will be expanded upon later in the chapter.

The choices for regularly storing and checking the backed-up data are then offered. The relevant Command Line Reference is provided for each section.

Common Zextras Backup Tasks

In addition to connections to technical resources, this section includes instructions for the most often job requested by users.

Activating Zextras Backup

A few additional steps are required once your servers are set up before you can configure the Backup module and have all of your data automatically backed up.

  • Install a storage device where you want it. In this step, the default path is /opt/zimbra/backup/zextras; make sure to change it to the one you choose.
  • Set the backup path’s correct permissions: /opt/zimbra/backup/zextras chown zimbra:zimbra

The backup route must be initialised once you have defined it, which may be done through the admin panel or the command line. In the first scenario, press the Initialise NOW! button in Fig. 6’s upper right corner. Initialization is carried out through the CLI by simply launching SmartScan for the first time: backup by zxsuite start doSmartScan

The design of Zextras Backup

The key ideas necessary to comprehend Zextras Backup’s design are introduced in this part, along with an overview of how they relate to one another. Each idea is then further explored in its own area.

RPO and RTO are two broad techniques that are taken into consideration when developing a backup plan, which we will discuss before moving on to the architecture of Zextras Backup.

The Recovery Time Objective (RTO) is the maximum length of time a stakeholder is willing to wait to recover its data, whereas the Recovery Point Objective (RPO) is the maximum quantity of data that a stakeholder is willing to lose in the event of a disaster.

These definitions state that the ideal acceptable value is zero, although practical values typically fall within a range of zero, depending on the amount of the data. Real Time Scan and SmartScan work together in Zextras to ensure that both RTO and RPO numbers are very low: While the SmartScan replicates all objects that have been updated, the Real Time Scanner guarantees that all metadata changes are recorded as soon as they occur. As a result, the potential loss of data is minimal and often restricted to those items that have changed between two consecutive runs on the SmartScan.

The notion of ITEM serves as the foundation for Zextras Backup’s whole architecture: The smallest thing that may be kept in the backup is called an item. For instance:

  • email communication
  • a person or a group of people
  • a binder
  • an arrangement
  • a task
  • a document on Drive
  • a user account and its settings
  • a list of recipients
  • an area
  • a category of services

As a result, they will never be checked for changes by the Real Time Scan or included in a restore. Examples of such objects include the following:

  • Server configuration, or the parameters for each server
  • Global Zextras product settings
  • Any modifications (Postfix, Jetty, etc.) added to the programme

Every change in the accompanying information for each item handled by Zextras Suite is tracked and preserved, enabling its restoration at a later time. In other words, if one of an item’s associated information changes, a transaction-based “photograph” of the entire thing is taken and saved with a timestamp. Metadata that describes an item includes, for instance:

  • when an email was opened, read, deleted, and transferred to a folder.
  • a change to a contact’s name, address, or employment
  • the removal of a file from a folder or its insertion
  • the modification of an item’s status (such as an account).

Technically, all modifications made to an object over its lifespan are kept in the form of a JSON Array. Read more about this in the section on item structure.

An object that has been designated for deletion is known as a deleted item.

A transaction is a change in the state of an object. Change of status refers to when a user modifies one of the item’s related metadata. As a result, a Transaction may be thought of as a snapshot of the metadata taken at a certain point in time. A Transaction ID serves as a unique identifier for every transaction. An item can be returned to any previous transaction. Zextras Backup Restore Strategies has further information.

Real-Time Scan and SmartScan

The actual content of a mailbox is read and utilised to create the backup as part of the Initial Scan, which is carried out by the SmartScan. If the Scan Operation Scheduling is activated in the Administration Zimlet, the SmartScan is then run daily and at each module start.

The main function of SmartScan is to look for objects that have changed since its last run and to update the database with any new data.

Every action that occurs on the system is recorded in real time by the Real Time Scan, enabling recovery with split-second accuracy. Since nothing in the backup is overwritten by the Real Time Scanner, each object has its own entire history. Additionally, it has the capacity to recognise other modifications pertaining to the same item at the same time and record them all as a single metadata change.

Real Time Scan and SmartScan are both by default enabled. While each one can be stopped (on their own), it is advised to leave them both running because they are meant to work best together.

When to Turn Off Scanning Operations
Since backups are stored on disc, access to the disc is required during scan procedures. There are so many circumstances in which either the SmartScan or Real Time Scan could (or should), even temporarily, be deactivated. For instance:
  • You often work with Drive documents or have a large volume of daily transactions, and you see that the server is using a lot of resources. In this situation, the Real Time Scan can be temporarily disabled.
  • You begin a migration: In this situation, it is advised to suspend the SmartScan since it would cause several I/O operations on the disc and maybe even cause the server to become unresponsive. In fact, it would regard each object that was moved or restored as a brand-new one.
  • Your daily email volume is heavy, both inbound and outbound. In this situation, you should always keep the Real Time Scan on since otherwise, the SmartScan will be the only backup for all transactions, and because I/O operations need a lot of resources, it might not be able to finish in a timely manner.
The location on a filesystem where all the data pertaining to the backup and archives is kept is known as the backup path. There is only one backup path available per server; many servers cannot share the same backup path. It is organised as a hierarchy of directories, with /opt/zimbra/backup/zextras/ by default at the top. The following significant files and folders are located under this directory:
  • Map files, also known as map_[server_ID] files, are used to indicate if a backup has been imported from an external backup and include the server’s unique ID in the filename.
  • All of the accounts defined in the Mailbox’s definitions are listed in the directory called accounts. You may find there, in particular, the following significant files and directories:
  • The file account_info contains all of the account’s details, such as the password, signature, and preferences.
  • account_stat is a file that contains statistics about the account, such as the ID of the most recent content that SmartScan saved.
  • The timestamp of the first run is one of the general data about the backup that are maintained in the file called backupstat.
  • Drive items are kept in a directory called drive_items that has up to 256 subfolders, each of which is named using two lowercase hexadecimal letters. Drive items are placed here according to the last two letters of their UUID.
  • items is a directory that has up to 100 subfolders. Items are kept in these subfolders according to the last two digits of their IDs.
  • servers is a directory that holds archives, one every day up to the chosen server retention duration, of the conversation, Zextras configuration, and server configuration and customizations.
  • items is a directory with a name made up of two hexadecimal (uppercase and lowercase) characters that may hold up to 4096 extra folders. The directory whose name contains the final two characters of each item’s ID will hold the items in the Mailbox.
  • A map between the original object and the restored item is contained in the user object ID mapping file called id_mapper.log. The file may be found in /backup/zextras/accounts/xxxxx-xxxx-xxxx-xxxxxxxxxx/id_mapper.log. Only in the event of an external restoration is this file present.
Choosing a backup route
Both the GUI and the CLI may be used to establish the Backup Path:
  • Via GUI: Under “Backup Path” in the “Backup” section of the Zextras Administration Zimlet.
  • Using the CLI: changing the ZxBackup_DestPath config key using the zxsuite config server command.
Retention Guidelines
How many days after being designated for deletion an object is actually erased from the backup depends on the retention policy (also known as retention period). The Backup’s retention guidelines are as follows:
  • The default setting for the data retention policy for single items is 30 days.
  • The account retention policy relates to the accounts and is set to 30 days by default.
The Backup Purge won’t execute if any retention times are set to zero (zero), which means archives will be retained indefinitely (infinite retention).
It will still be possible to recover all items up to the Account retention time if an account is deleted and needs to be restored after the Data retention time has passed because in that case, even if all the metadata have been deleted, the digest may still contain the information needed to restore the item.
 
Backup Purge: The Backup Purge is a cleanup procedure that eliminates any deleted items that have outlived the retention period allowed by the Data Retention Policy and the Account Retention Policy from the Backup Path.
Check for Coherence
The Coherency Check examines a Backup Path more thoroughly than SmartScan does in order to identify faulty information and BLOBs.
 
The Coherency Check thoroughly examines all of the information and BLOBs in the Backup Path, whereas the SmartScan simply checks objects that have changed since the last SmartScan run.
 
Use the zxsuite backup doCoherencyCheck command to launch a Coherency Check from the CLI.
The Operation of Zextras Backup
Zextras Backup is a tool that can store all iterations of an ITEM. It is compatible with many OS architectures and Zimbra versions because it is not designed to be a system or Operating System backup.
 
Administrators may use Zextras Backup to generate an atomic backup of each item in a mailbox account and then restore various things to various accounts or even separate servers.
 
All backup files are by default saved in the local directory /opt/zimbra/backup/zextras/ by Zextras Backup. A directory has to meet certain requirements in order to be utilised as the Backup Path.
  • able to be read and written by users of Zimbra.
  • Use a filesystem that recognises case.
When Zextras Backup is first launched, a SmartScan is launched to get all data from the mailbox and build the first backup structure. Each item is stored along with all of its information as a JSON array on a case-sensitive filesystem. After the first start, the backup may be kept current and synced with the account by using either the Real Time Scanner, the SmartScan, or both.
 
The Composition of an Item
The item’s basic structure is a JSON Array that keeps track of all changes made to it over its lifetime, including information about emails (such as tags, visibility, or moving an email to a folder), contacts, tasks, individual folders, groups, or drive documents, as well as user preferences (such as password hashes and general settings).
For example, it is not useful to store the user’s last login time or the IMAP and Activesync states because if the account will be restored on a new one, the values of those attributes would be related to the old account. This is done to improve performance.
 
We may recover data at a certain point in time by gathering the transaction’s timestamp.
 
The engine examines all valid transactions while performing the restoration, assessing the “start-date” and “end-date” characteristics.
The same reasoning is applied for retrieving deleted things: when an item is removed, the timestamp is stored, allowing us to recover deleted items that occurred within a certain time period.
 
The metadata keeps track of the validity of the old and new digests even if the blob associated with the item changes and, as a result, so does its digest (as is the case for Drive Document).
Team database backup
A platform for instant messaging called Zextras Team offers a variety of capabilities, such as file sharing, Web conferencing, and more. Team’s database may quickly expand to a huge size due to the fact that it keeps track of everything (uploaded files, conversation, and so on). This slows down backup procedures and renders the database unusable for restore operations.
 
Because of this, Team’s DB backup has been set to be deactivated by default. Theoretically, an Administrator may allow it, but only after getting in touch with a TSE (Technical Support Engineer).
What is a SmartScan, exactly?
The SmartScan only runs on accounts that have changed since the last SmartScan; as a result, it may dramatically increase system performance and cut down on scan time.
 
If Scan Operation Scheduling is enabled in the Administration Zimlet’s Zextras Backup area, a SmartScan is automatically scheduled to run every night. Every week on a day chosen by the user, Zextras Backup does a Purge along with a SmartScan to purge its datastore of any deleted items that have outlived their retention duration.
How does it function?
Looking for things updated since the last SmartScan, the Zextras Backup engine analyses every item on the Zimbra Datastore. When an item is detected in the backup but not in the Zimbra datastore, it flags it as deleted and updates any outdated entries as well as adds any items that aren’t yet there.
 
The backup is then updated with all configuration metadata, resulting in the storage of domains, accounts, COSs, and server settings in addition to a dump of all configuration.
 
When LDAP is included in the setup, SmartScan will save a compressed LDAP dump in the Backup Path that may be used independently to fix LDAP configuration issues.
For every account when the External Restore feature is turned on, SmartScan prepares a single (daily) archive that contains all of the account’s metadata and saves it on the external volume. See the section on backup on external storage for further details.
How Often is a SmartScan Run?
  • Upon launching the Zextras Backup module.
  • Whenever the Scan Operation Scheduling option in the Administration Zimlet is activated, every day
  • after being previously deactivated, when the Real Time Scanner is once again enabled through the Administration Zimlet
Checking the Status of a Running Scan and Running a SmartScan
It is advised to check how many activities are currently running in order to identify the proper id before actually doing this check. The zxsuite backup getAllOperations command may be used to do this.
Use the zxsuite backup monitor command in the CLI to examine the progress of a current scan.
Live Scanning
What exactly is a real-time scanner?
The Real Time Scan, an engine that is closely linked to the Mailbox, intercepts all transactions that occur on each user’s mailbox and records them in order to preserve an item’s complete history for its entire lifespan.
 
Any object may be recovered at any moment with the help of the Real moment Scan.\
How does it function?
The Real Time Scanner nearly instantly receives every event from the mail server and then “replicates” those actions on its own data structure by adding items or changing their information. Since nothing is ever overwritten, each object in the backup has a full history of its own.
Why Should I Disable the Real Time Scanner? Managing the Real Time Scanner Enabling the Real Time Scanner
Only while conducting an external restore of many domains should you turn off the real-time scanner. This is a precaution to prevent your server from being overloaded. Re-enable the Real Time Scanner after the import, and when requested, do a SmartScan.
Safety Scan and Limitations
The following are the key restrictions for recovering data obtained with the Real Time Scanner:
  • When a user selects the Empty Folder option from the right-click context menu, the folder is empty.
 
An Account Scan on the specified account is initiated PRIOR TO the restoration in this scenario and any other time Zextras Backup is unable to detect the condition of an item by reading the information captured by the Real Time Scan.
 
This sanitises the mailbox backup information and corrects any mismatched data.
 
Backup removal
What is the Backup Purge? It is a cleanup procedure that eliminates from the Backup Path any deleted item that has been there longer than the Retention Policy-specified retention period.
 
How does it function?
When an item designated for deletion is discovered whose last update is older than the retention time limit, the purge engine checks the metadata of all the deleted items and deletes the item from the backup.
 
However, because of Zextras Backup’s integrated deduplication, a BLOB item will not be erased if it is still referenced by one or more valid metadata files.
 
Zextras Backup also adheres to the Backup Path’s delete policies while backing up customizations. By deselecting the Purge old customizations checkbox in the Administration Zimlet’s Zextras Backup section, this may be adjusted.
 
When is a Backup Purge Executed? 
  • Weekly if the Administration Zimlet’s Scan Operation Scheduling is activated.
  • when manually initiated using the CLI or the Administration Console
Since no deleted item will ever go beyond the retention duration while infinite retention is active (i.e., the Data Retention Policy is set to 0), the Backup Purge will end instantly.
Checking the Status of a Running Backup Purge and Running a Backup Purge
Use the zxsuite backup monitor command on the CLI to check the progress of a running Purge:
Limitations and Unique Situations for the Backup
  1. In a few instances, the backup does not function properly. Here, we go over such situations.
  2. The most recent state should NOT be used when restoring an active account to a new account. Imagine if a person accidentally deletes all of his emails or that the emails in a user’s account are lost for any other cause (such, for example, a server failure). The user begs the administrator to give them back. The outcome of restoring the account status to the most recent state accessible is that the new account will include that state, which is an empty account because the emails had already been erased in the most recent state. Therefore, it is important to restore the account to a point in time that is before the emails were erased in order to do so appropriately.
  3. If an email client is set up to download emails and promptly delete them from the server while utilising the POP3/POP3S protocol, these emails might not be backed up. If the Zextras Powerstore component is installed, this does not take place.
  4. The backup does not include emails sent directly over an SMTP connection (such as when using a multifunctional device or telnet to connect to the STMP server).
  5. If an IMAP/SMTP client is used to send email, the IMAP client must be set up to store the email in a remote folder (using the IMAP STORE command) after the send operation in order for the email to be backed up.
Fixing LDAP Backup issues
The backup of just the LDAP data may occasionally fail while backing up a mailbox server and end with the following warning:
In this part, we offer some recommendations about how to approach this issue.
 
Amplify Log Verbosity
The log file contains a variety of log messages, depending on the configuration of the mailbox server. A first line of defence is to make the log file more verbose in the event that an LDAP backup fails and the log file does not record enough messages to pinpoint the reason of the failure.
Now, use the following command (which just backs up the LDAP data) to do a backup, and then recheck the log file.
Restore the verbosity to the previous level when the command completes and you have done reviewing the log file:
 
Root credentials are missing
Zextras Suite must connect remotely to the LDAP server using LDAP root credentials in order to back up LDAP data.
 
The password is preserved specifically in the Zimbra localconfig, however the LDAP root password is blank on mailbox servers without the LDAP component installed. As a result, the LDAP connection fails with an error stating that the credentials were incorrect, and no LDAP data backup is created.
 
On a mailbox server, use the following series of commands to confirm this situation:
 
The final command should provide results.
The output of the command should now be dn:cn=config. If not, either the LDAP root password is incorrect or it has not been saved in the local settings.
 
Follow these three steps to resolve the issue.
Disable LDAP Backup You may completely turn off LDAP data backup with the Zextras suite if you don’t want to. Run this command to turn off LDAP Backup on each mailbox server.
Use external storage as a backup
Zextras Backup is made up of metadata and blobs (compressed and deduplicated), which are by default stored on the same folder—or mounted volume—specified in the Backup Path, as explained in the Architecture of Zextras Backup. The Backup Path must be quick enough to prevent queuing activities and/or risk data loss in order to provide real-time backup.
 
S3 buckets, NFS shares, and other storage mounted using Fuse, on the other hand, can be extremely sluggish and might not be suitable for mounting on the Backup Path.
The concept behind Backup on External Storage is to employ two distinct storages: one local (and generally fast) for metadata and cache and one external (local network or cloud) for the blobs and a copy of the metadata. This is because the metadata is the most crucial component of backups.
 
Multiple updates will be combined and transferred together if the external storage is remote, whereas bigger, slower, and less expensive storage can be used if it is nearby.
How backups on external storage are performed
Metadata are locally preserved in the Backup Path, BLOBs are briefly cached on the local disc, and they are sent as quickly as possible to the distant storage.
 
When an account has been updated since the last scan, the SmartScan locally updates the metadata for such accounts and archives them on the remote storage.
 
Running one of the following commands with the remote_metadata_upload true option will manually start the remote metadata archiving:
  • backup zxsuite doSmartScan
  • backup zxsuite doAccountScan
  • doBackupServerCustomizations in zxsuite backup
  • backup with zxsuite doBackupLDAP
  • backup using zxsuite doBackupCluster
Better reliability and backup resilience are provided by dividing the I/O-intensive metadata folder from the BLOBs one. This also ensures that the backup continues to function even if the remote storage is briefly unavailable due to network problems or ongoing maintenance.
Objects and advantages
The following are the top two benefits of backing up to external storage:
 
  • Only metadata that statistically makes up less than 10% of the overall backup size requires fast IOPS storage.
  • The majority of the time, backups are kept outside, away from the local infrastructure, and are therefore reachable from disaster recovery sites.
You may use the zxsuite backup setBackupVolume Default command to deactivate the External Storage.
Information kept on the external storage
Using a structure very similar to the Backup Path, data is kept on external storage:
Only the $BACKUP_PATH/items are stored on the external drive; however, the metadata, which is stored in the $BACKUP_PATH/accounts, continues to utilise the local disc as a working directory to keep the modified metadata.
There are a number of commands specifically designed to download metadata from external storage, reconstruct the account’s structure and content in the event of a disaster, or update/correct local information.
For instance, this programme downloads to the Backup Path the most recent metadata that is accessible in the remote storage.
For further details, refer to the zxsuite backup retrieveMetadataFromArchive S3 documentation.
 
External storages: There are two types of supported external volumes: NFS external volumes and Fuse external volumes. Shared volumes mounted either at the OS level or totally controlled by Zextras are both examples of supported external volumes.
External NFS/Fuse storage
Because no existing volume may be reused, it is important to configure the new volume(s) that will contain the backup before utilising the NFS/Fuse sharing. The actions you take vary depending on the strategy you decide on. Here, we simply cover the simpler and more dependable option
S3 outside storage
It is necessary to construct a specific Zextras bucket before utilising an ObjectStorage.
 
Zextras Backup and Zextras Powerstore buckets are incompatible despite having a similar idea. It is not feasible to store Backup data on the same bucket as Powerstore data, and vice versa.
 
The zxsuite core listBuckets command provides use information for buckets, for instance:
Given that each Zextras Bucket has a prefix identifying it, you may utilise the combination of S3 bucket credentials and the prefix of a Zextras Bucket to identify and store several Zextras Buckets separately within a single S3 Bucket.
 
In other words, several Zextras Buckets might be defined in the same Amazon S3 Bucket to be utilised for both Powerstore HSM and Backup.
A multi-mailbox scenario with S3 Backup
Multiple buckets are not required in multi-mailbox environments: When configuring remote backup on the first server, you just enter the bucket configuration data. The data from additional servers can then be kept in a different directory on the same storage by using the bucket_configuration_id and prefix arguments.
Switch on the external storage’s backup function.
It is required to permit Zextras Suite to utilise the external storage after it has been configured. Depending on whether the new storage must be accessible from a freshly installed server or whether current local backups must be moved to the external storage, the process is slightly different.
CLI for Zextras Backup
The index of all zxsuite backup commands may be found in this section. ZxBackup CLI Commands has a section dedicated to it called Full Reference.
BackupAuthToken, doAccountScan, backup chat, backup cluster, backup LDAP, and backup server customizations  check shares, check coherenceVerify doEnableDisableCOS  doPurge doRaw doExport doExternalRestore doFixShares doItemRestore doItemSearchRestore, restart services, restore blobs, restore on new accounts, run smart scans, stop all operations, stop operation, stop service, and delete  getAccountInfo  Obtain All Operations and Obtain Available Accounts  getAvailableDomains  getBackupInfo getsCOSBackupStatus  getItem  getMap  getProperty  getServerConfig  migrateBackupVolume Default getServices migrateBackupVolume a local migrationBackupDefault setBackupVolume Local setBackupVolume S3 setProperty update Volume S3 monitor retrieveMetadataFromArchive Local retrieveMetadataFromArchive S3 setBackupVolumeS3 BackupVolume
 

Leave a Reply

Your email address will not be published. Required fields are marked *