Carbonio Email Service provider

Logo
Menu
Logo
Menu
Have a Question?

This chapter discusses Zextras Backup, the Zextras Suite component in charge of backing up all data. The chapter is organised into many sections: first, an overview of the most typical task is provided, followed by links to more technical literature.

Following that, the architecture of Zextras Backup is defined, which includes crucial ideas to understand ahead of time; the concepts will be explored in the rest of the chapter.

Finally, the choices for periodically storing and checking the backed-up data are offered. The relevant Command Line Reference is included with each section.

Backup Common Tasks using Zextras

This section offers guidance for the most typical tasks encountered by users, as well as access to technical information.

How to Use Zextras Backup

After you’ve done configuring your servers, you’ll need to take a few more steps to configure the Backup module and have all of your data automatically backed up.

Install a storage device at the desired place. Throughout this part, we will use the default location /opt/zimbra/backup/zextras; remember to change it with the path you choose.

Once the backup route has been defined, it must be initialised, which may be done using the admin panel or the command line. In the first scenario, in the upper right corner of Fig. 6, click the Initialise NOW! button. From the command line, initialization is accomplished by simply running SmartScan for the first time: backup using zxsuite begin SmartScan

Zextras Backup Architecture
This section presents and describes the fundamental ideas required to understand the architecture of Zextras Backup; each concept is then described in its own section.
 
Before delving into the architecture of Zextras Backup, we need review two broad ways to creating a backup strategy: RPO and RTO.
 
The Recovery Point Objective (RPO) is the most data that a stakeholder is ready to lose in the event of a disaster, whereas the Recovery Time Objective (RTO) is the most time that a stakeholder is prepared to wait to retrieve its data.
These definitions state that the ideal acceptable value is zero, although reality values are generally close to zero, depending on the amount of the data. The combination of Real Time Scan and SmartScan in Zextras ensures that both RTO and RPO numbers are relatively low: The Real Time Scanner guarantees that all metadata changes are recorded as soon as they occur, whereas the SmartScan copies all objects that have been modified, limiting the possibility of data loss to items that have changed between two consecutive SmartScan runs.
Zextras Backup’s whole architecture centres around the concept of ITEM: An item is the smallest thing preserved in the backup, for example:
  • a message sent through email
  • a person or a group of people
  • a file folder
  • a scheduled appointment
  • a task
  • a Google Drive file
  • an account (together with its settings)
  • a list of recipients
  • a website
  • a type of service (COS)
There are other non-item objects that will never be checked for changes by the Real Time Scan and will never be part of a restore:
  • Server setup, or the configuration of each server
  • Zextras product parameters at the global level
  • Any software modifications (Postfix, Jetty, etc…)
Every alteration in the accompanying information for each object handled by Zextras Suite is recorded and kept, allowing its restoration at any point in time. In other words, if one of an item’s information changes, a “photograph” of the entire thing is taken and recorded with a timestamp through a transaction. The following are some examples of metadata connected with an item:
  • after an email has been read, deleted, or moved to a folder
  • a change in a contact’s name, address, or employment
  • the removal or addition of a file from a folder
  • a change in the status of an object (for example, an account)
An item is technically saved as a JSON Array that contains all modifications made over the object’s lifespan. More on this in the section on Item Structure.
 
A Deleted Item is one that has been designated for deletion.
A transaction is a change in the status of an object. A change of status occurs when a user modifies one of the metadata associated with an object. As a result, a Transaction may be viewed as an image of the metadata at a certain point in time. A Transaction ID is assigned to each transaction. It is possible to return an item to a previous transaction. More information may be found in Zextras Backup Restore Strategies.
 
Real-Time Scan and SmartScan
The basic structure of the backup is constructed during the SmartScan’s basic Scan: the content of a Mailbox is read and utilised to build the backup. If Scan Operation Scheduling is enabled in the Administration Zimlet, the SmartScan is then conducted at the start of the module and on a daily basis.
SmartScan’s primary function is to check for objects that have been updated since the last run and to update the database with any new information.
 
The Real Time Scan records every event that occurs on the system in real time, allowing for split-second recovery. Because the Real Time Scanner does not erase data in the backup, each object has its own entire history. Furthermore, it may identify other modifications to the same item at the same time and record them all as a single metadata update.
SmartScan and Real Time Scan are both turned on by default. While both may be stopped (independently), it is recommended that they remain running because they are meant to compliment each other.
When Should Scan Operations Be Disabled?
Because backups are written to disc, Scan activities require I/O disc access. As a result, there are a variety of instances in which either the SmartScan or Real Time Scan may (or should) be deactivated, even if only momentarily. As an example:
  • You have a high volume of transactions every day (or frequently interact with Drive documents) and observe a heavy burden on the server’s resources. You can temporarily disable the Real Time Scan in this scenario.
  • You begin a migration: In this scenario, it is recommended that you halt the SmartScan since it will generate a large number of I/O operations on disc and may even cause the server to crash. It would, in fact, consider every migrated or restored object as if it were a new one.
  • You receive and send a large number of emails each day. In this situation, you should always have the Real period Scan enabled, because otherwise, all transactions would be backed up solely by the SmartScan, which may be unable to finish in a reasonable period due to the resources required for I/O operations.
Backup Path The backup path is the location on a disc where all backup and archive information is saved. Each server has a unique backup path; no two servers can share the same backup path. It is organised as a hierarchy of directories, with the topmost folder being /opt/zimbra/backup/zextras/ by default. This directory contains the following key files and directories:
  • map_[server_ID] are so-called map files that indicate if the Backup was imported from an external backup and include the server’s unique ID in the filename.
  • accounts is a directory containing information for all accounts defined in the Mailbox. There are particularly significant files and folders to be found there:
  • account_info is a file that contains all of the account’s metadata, such as the password, signature, and preferences.
  • account_stat is a file that contains different statistics about the account, such as the ID of the most recently saved material by SmartScan.
  • backupstat is a file that stores general backup statistics, such as the timestamp of the first run.
  • drive_items is a directory that contains up to 256 subfolders (named with two hexadecimal lowercase letters) that hold Drive items based on the last two letters of their UUID.
  • items is a directory with up to 100 subfolders (the names of which are made up of two digits, and in which things are placed according to their ID’s last two digits).
  • servers is a directory that stores daily archives of the server setup and customizations, Zextras configuration, and chat, up to the configured server retention duration.
  • items is a directory that may hold up to 4096 extra folders and is named with two hexadecimal (uppercase and lowercase) characters. Items in the Mailbox will be saved in the directory with the last two characters of their ID as the name.
  • id_mapper.log is a user object ID mapping file that contains a link between the original and restored objects. It may be found in /backup/zextras/accounts/xxxxx-xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/id_mapper.log. This file is only present in the event of an external restoration.
Configuring the Backup Path
Both the GUI and the CLI may be used to configure the Backup Path:
  • Via GUI: under “Backup Path” in the “Backup” section of the Zextras Administration Zimlet.
  • Change the ZxBackup_DestPath config key using the zxsuite config server command.
Policy on Retention
The Retention Policy (also known as the retention period) specifies how many days an object marked for deletion gets deleted from the backup. The backup retention policies are as follows:
  • The data retention policy only applies to single items and is set to 30 days by default.
  • Account retention policy applies to the accounts and is set to 30 days by default.
All retention durations are adjustable; if set to 0 (zero), archives are retained indefinitely (infinite retention), and the Backup Purge is disabled.
If an account is deleted and must be restored after the Data retention time has expired, all items up to the Account retention time can still be recovered, because even if all metadata has been purged, the digest can still contain the information needed to restore the item.
Backup Cleanup
The Backup Purge is a cleanup procedure that removes from the Backup Path any deleted item that has outlived the retention duration specified in the Data Retention Policy and Account Retention Policy.
Check for Coherence
The Coherency Check is especially designed to detect faulty information and BLOBs, and it examines a Backup Path in more depth than SmartScan.
 
While the SmartScan checks just objects that have been updated since the last SmartScan run, the Coherency Check checks all metadata and BLOBs in the Backup Path.
 
Use the zxsuite backup doCoherencyCheck command to initiate a Coherency Check from the command line:
How Does Zextras Backup Work?
Zextras Backup was created to store every version of an ITEM. Because it is not meant as a system or operating system backup, it can function with many OS architectures and Zimbra versions.
Administrators may use Zextras Backup to establish an atomic backup of each item in the mailbox account and restore various things on separate accounts or even other servers.
 
By default, Zextras Backup saves all backup files in the local directory /opt/zimbra/backup/zextras/. A directory must meet the following requirements in order to be considered for usage as the Backup Path:
  • The zimbra user should be able to read and write to it.
  • Make use of a case-sensitive filesystem.
When Zextras Backup is first launched, it does a SmartScan to get all data from the mailbox and generate the first backup structure, in which each item is stored along with all of its information as a JSON array on a case sensitive filesystem. Following the first start, the Real Time Scanner, SmartScan, or both can be used to maintain the backup up to date and synchronised with the account.
The Structure of an Item
The item’s basic structure is a JSON Array that records all changes that occur during the item’s lifetime, such as information about emails (e.g., tags, visibility, email moved to a folder), contacts, tasks, single folders, groups, or drive documents, and user preferences (e.g., password hash, general settings).
To improve performance, only the changes required to restore the items are recorded: for example, it is not useful to store the user’s last login time or the IMAP and Activesync state because the values of those attributes would be related to the old account if the account is restored on a new one.
 
We can recover data at a certain point in its existence by collecting the transaction’s timestamp.
 
During the restoration, the engine evaluates the “start-date” and “end-date” characteristics of all valid transactions.
 
The same principle is used to recover deleted things: when an item is removed, we save the timestamp and may thus restore anything deleted within a certain time window.
 
Even if the blob connected with the item changes, and therefore its digest changes (like with Drive Document), the metadata records the validity of both the old and new digests.
 
Team Database Backup
Zextras Team is an instant messaging platform that includes file sharing, Web conferencing, and other services. Because Team keeps track of everything (uploaded files, conversations, and so on), its database may soon expand to a big size, slowing down Backup processes and rendering it unusable for restoration operations.
As a result, backups of Team’s database are now deactivated by default. In principle, an Administrator can activate it, but only after consulting with a TSE (Technical Support Engineer).
 
What exactly is the SmartScan?
Because the SmartScan only functions on accounts that have been updated since the previous SmartScan, it can enhance system performance and reduce scan time significantly.
A SmartScan is conducted every night by default (if Scan Operation Scheduling is enabled in the Zextras Backup section of the Administration Zimlet). A Purge is done once a week, on a user-specified day, together with the SmartScan to clean Zextras Backup’s datastore of any deleted item that has surpassed the retention term.
What is the procedure?
The Zextras Backup engine searches the Zimbra Datastore for things that have been updated since the last SmartScan. It updates any outdated entries and produces any items that are not yet present in the backup, while marking any item discovered in the backup but not in the Zimbra datastore as destroyed.
 
The backup’s configuration information is then changed, resulting in domains, accounts, COSs, and server configurations being saved with a dump of all configuration.
 
When LDAP is used in the setup, SmartScan will save a compressed LDAP dump in the Backup Path, which may also be used independently to recover a damaged LDAP configuration.
When the External Restore feature is enabled, SmartScan builds a single (daily) archive for each account that contains all of the account’s metadata and saves it on an external drive. More information may be found in the section Backup on external storage.
 
When is a SmartScan performed?
  • When you launch the Zextras Backup module.
  • If the Administration Zimlet’s Scan Operation Scheduling is activated, this will happen every day.
  • When the Real Time Scanner is re-enabled via the Administration Zimlet after having been deactivated earlier
Checking the Status of a Running Scan Running a SmartScan
It is recommended that before doing this check, you verify how many activities are occurring in order to find the right id. This may be accomplished by using the zxsuite backup getAllOperations command.
Use the zxsuite backup monitor command to verify the status of an ongoing scan from the command line:
 
Real-Time Scanner What is a Real-Time Scanner?
The Real Time Scan is an engine that is intimately linked to the Mailbox that intercepts all transactions that occur on each user’s mailbox and records them in order to save the whole history of an item for its entire existence.
 
Any object may be recovered at any moment in time thanks to the Real Time Scan.
What is the procedure?
The Real Time Scanner receives all mail server events nearly in real time, then’replicates’ them on its own data structure, producing items or modifying their information. Because no data is ever overwritten in the backup, each object has its own entire history.
 
Why Should I Disable the Real Time Scanner? Managing the Real Time Scanner Enabling the Real Time Scanner
Only while running an External Restore of many domains should you disable the Real Time Scanner. This is a precautionary approach to avoid overloading your server. Re-enable the Real Time Scanner and do a SmartScan when requested after the import.
Limitations and a Safety Check
When recovering data gathered using the Real Time Scanner, the fundamental constraint is:
  • Emptied Folder – when the Empty Folder button in the right-click context menu is used.
If Zextras Backup cannot detect the condition of an item by reading the information captured by the Real Time Scan in this circumstance, an Account Scan on the provided account is initiated BEFORE the restoration.
 
This corrects any mismatched data and sanitises the mailbox’s backed up information.
 
Backup Cleanup
What exactly is the Backup Purge?
The Backup Purge is a cleanup procedure that removes from the Backup Path any deleted item that has surpassed the Retention Policy’s retention time.
What is the procedure?
The Purge engine checks the information of all removed items and deletes any objects marked for deletion whose last update is older than the retention time range.
 
However, if an item BLOB is still referenced by one or more valid metadata files, the BLOB will not be erased owing to Zextras Backup’s built-in deduplication.
 
Customizations backed up by Zextras Backup adhere to the purging policies of the Backup Path. This may be modified in the Administration Zimlet’s Zextras Backup section by unchecking the Purge old modifications option.
When is a backup purge performed? Weekly, if the Scan Operation Scheduling option in the Administration Zimlet is activated.
 
When launched manually through the Administration Console or the CLI
 
When you enable infinite retention (i.e., set the Data Retention Policy to 0), the Backup Purge will instantly end since no removed item will ever exceed the retention duration.
 
Running a Backup Purge Checking the Status of a Backup Purge that is Running
Use the zxsuite backup monitor command to verify the status of an ongoing Purge from the command line:
Backup Limitations and Special Cases
There are a few instances where the backup does not function properly. These cases are discussed in this section.
 
  1. The newest state available should NOT be used to restore an active account on a new account. Assume that a person deletes all of his emails by accident, or that the emails in an account are lost for whatever cause (for example, a server failure). The user requests their return and approaches the administrator. If the admin restores the account’s status to the most recent accessible state, the new account will include the most recent available state, which is an empty account, because the email was previously erased in the most recent state.
  2. As a result, in order to successfully recover the account, it must be restored at a period in time before the emails were deleted.
  3. If the email client is configured to download email messages and delete them immediately from the server while utilising the POP3/POP3S protocol, these messages may not be included in the backup. If the Zextras Powerstore component is installed, this does not occur.
  4. When an email is sent directly over an SMTP connection (e.g., using a multifunctional device or connecting to the STMP server through telnet), it is not included in the backup.
  5. When sending email using an IMAP/SMTP client, the IMAP client must be set to store the send email in a remote folder (through the IMAP STORE command) after the send operation; otherwise, the email may not be included in the backup.
Troubleshooting LDAP Backup
When backing up a mailbox server, the backup of merely the LDAP data may fail and finish with a warning:
We offer various solutions to this problem in this section.
Boost Log Verbosity
A number of log messages are recorded in the log file depending on the mailbox server setup. If an LDAP backup fails and the log file does not include enough messages to determine the underlying cause of the problem, one option is to raise the log file’s verbosity.
Now, execute the backup command (which just backs up the LDAP data) and check the log file again.
 
When the command is done and you have finished reviewing the log file, remember to return the verbosity to its previous level:
Root credentials are missing.
Zextras Suite must create a remote connection to the LDAP server using LDAP root credentials in order to back up LDAP data.
 
The password is kept in the Zimbra localconfig, however on a mailbox server without the LDAP component, the LDAP root password is empty. As a result, the LDAP connection fails with an incorrect credentials error, and the LDAP data backup is not performed.
 
This scenario may be confirmed by running the following commands on a mailbox server:
The final command should result in output.
 
Now, execute the command
 
Should return dn:cn=config. If this is not the case, the LDAP root password is either incorrect or does not exist in the local setup.
 
Follow this three-step approach to resolve the issue.
Disable LDAP Backup If you do not want Zextras suite to backup LDAP data, you can turn it off completely. Run this command on each mailbox server to deactivate LDAP Backup.
External storage backup
Zextras Backup is made of metadata and blobs (compressed and deduplicated), which are stored by default on the same folder — or mounted drive — indicated in the Backup Path. The Backup Path must be quick enough to avoid queuing processes and/or data loss during real-time backup.
 
However, S3 buckets, NFS shares, and other storage mounted using Fuse can be very sluggish and may not be suitable for backup path storage.
 
Because the metadata is the most significant aspect of backups, the concept behind Backup on External Storage is to employ two distinct storages: one local (and often fast) for metadata and cache and one external (local network or cloud) for blobs and a copy of metadata.
Multiple updates will be combined and transmitted together if the external storage is remote, however if it is local, bigger but slower and cheaper storage can be used.
How External Storage Backup Works
Metadata is preserved locally in the Backup Path, and BLOBs are cached on the local disc for a short time before being sent to remote storage as quickly as feasible.
 
SmartScan refreshes the information for accounts that have been updated since the previous scan and archives it on remote storage.
 
Remote metadata archiving may also be manually activated by running one of the following commands with the remote_metadata_upload true parameter:
  • backup zxsuite doSmartScan
  • backup zxsuite doAccountScan
  • BackupServerCustomizations in zxsuite
  • backup zxsuite doBackupLDAP
  • backup doBackupCluster zxsuite
By separating the I/O heavy metadata folder from the BLOBs folder, it is also assured that the backup will run even if the remote storage is briefly unavailable (for example, due to network difficulties or ongoing maintenance chores), resulting in improved dependability and backup resilience.
 
Benefits and objectives
It is worthwhile to mention the two primary benefits of backup on external storage:
 
Fast IOPS storage is only required for metadata that accounts for less than 10% of the overall backup size.
 
Backups are often kept outside of the local infrastructure and therefore available from disaster recovery sites
You may deactivate External Storage by using the zxsuite backup setBackupVolume Default command.
Data kept on external storage
Data is saved in external storage using a structure that is fairly similar to the Backup Path:
For additional details, see the zxsuite backup retrieveMetadataFromArchive S3 documentation.
 
External volumes Supported external volumes, i.e. shared volumes mounted at the OS level or object storage wholly controlled by Zextras, are of two types: NFS external volumes and Fuse external volumes, which are explained further in this section.
External NFS/Fuse storageBefore utilising the NFS/Fuse sharing, setup the new volume(s) that will hold the backup, as no existing volume may be reused. The actions to take varies depending on the strategy you pick. Only the simplest and most dependable method is described here.
External S3 storage
A specific Zextras bucket must be built before utilising an ObjectStorage.
 
Zextras Backup and Zextras Powerstore buckets are not compatible with each other, despite their similarity in principle. If Powerstore data is saved in a bucket, Backup data cannot be placed in the same bucket, and vice versa.
 
The bucket utilisation is reported by the zxsuite core listBuckets command, for example:
Because each Zextras Bucket is defined by a prefix, you may uniquely identify and store numerous Zextras Buckets within a single S3 Bucket by combining S3 bucket credentials and Zextras bucket prefix.
 
In other words, you might establish numerous Zextras Buckets within the same Amazon S3 Bucket to be utilised for both Powerstore HSM and Backup.
 
Backup to S3 in a multi-mailbox environment
It is not essential to construct numerous buckets in multi-mailbox environments: When you enable remote backup on the first server, you just enter the bucket setup information. The bucket_configuration_id and prefix parameters are then used to store data from other servers in a different directory on the same storage.
Enable backup on external storage.
Once the external storage has been configured, Zextras Suite must be allowed to utilise it. The technique differs somewhat depending on whether the new storage must be accessible from a freshly installed server or whether existing local backups must be moved to the external storage.
Backup CLI Zextras
The index of all zxsuite backup commands may be found in this section. Full documentation is available in the dedicated section ZxBackup CLI Commands.
doAccountScan doBackupAuthToken doBackupChat doBackupCluster doBackupLDAP doBackupServerCustomizations  doCheckShares doCoherencyDoEnable should be checked.DisableCOS  doExport doExternalRestore doFixShares doItemRestore doItemSearch doPurge doRawRestore doRestartService doRestoreBlobs doRestoreOnNewAccount doSmartScan doStartService doStopAllOperations doStopOperation doStopService doUndelete  getAccountInfo  getAllOperations getAvailableAccounts  getAvailableDomains  getBackupInfo getCOSBackupStatus  getItem  getMap  getProperty  getServerConfig  getServices migrateBackupVolume Default migrateBackupVolume Locals migrateBackupVolume S3 monitor retrieveMetadataFromArchive Local retrieveMetadataFromArchive S3 setBackupVolume Default setBackupVolume Local setBackupVolume S3 setProperty updateBackupVolume S3