Have a Question?
What Can Go Wrong During Disaster Recovery?
One or more of the following events must occur for a situation to be classified as a Disaster:
- Failure of one or more critical filesystems (such as / or /opt/zimbra/) due to hardware failure
- Internal or external circumstances (such as a thoughtless rm * or an external incursion) render the contents of a critical disc useless.
- Failure of the physical computer hosting the Zimbra service or the associated virtualization infrastructure
- A significant failure in a software or operating system update/upgrade
Taking Fewer Chances
Some ideas for reducing the likelihood of a disaster:
- Always store critical filesystems on separate discs (e.g., /opt/zimbra/ or your Zextras Backup Path).
- Use a server monitoring/alerting tool to detect problems as soon as they emerge.
- Plan your upgrades and migrations thoroughly.
How to Recover Your System The Recovery
A system’s recovery is broken into two steps:
- Base system recovery (installation and configuration of the operating system, installation and configuration of Zimbra)
- Data recovery (reimporting the most recent data accessible to the Zimbra server, including domain and user configurations, COS data, and mailbox contents)
How Can Zextras Backup Aid Recovery?
Zextras Backup’s Import Backup functionality makes it simple and secure to conduct step 2 of a recovery.
Using the backup path of the old server as the import path allows you to restore a basic Zimbra installation to the last valid moment of your previous server.
The Rehabilitation Procedure
- Configure the Server and Global settings of Zimbra on a new server.
- Install Zextras Suite on the newly installed server.
- Mount the old server’s backup folder on the new one. If this is not accessible, utilise the most recent external backup or copy of either.
- Begin an External Restore on the new server with the CLI command:
- The External Restore procedure will immediately generate the domains, accounts, and distribution lists, so the system will be available for your users as soon as the first portion of the Restore is done (see your Zextras Suite Notifications). After that, emails and other inbox objects will be recovered.
Configurations & Settings
Server and global settings are backed up but not immediately restored. The high-level integration of Zextras Backup with Zimbra enables you to restore your data to a server with a different OS/Zimbra Release/Networking/Storage configuration with no limits other than the minimal Zimbra version necessary to operate Zextras Suite.
Whether you want to produce an exact clone of the previous server or just adapt the settings from the old server to a new environment, Zextras Backup includes a very useful CLI command:
Snapshots and VMs
Virtual machines are now the most frequent option to install server solutions such as Zimbra Collaboration Suite, thanks to the introduction of highly advanced virtualization technologies in recent years.
Most hypervisors have configurable snapshot capabilities as well as snapshot-based VM backup mechanisms. In the event of a disaster, you may always roll back to the most recent snapshot and import the missing data using Zextras Backup’s External Restore option, which uses the server’s backup path as the import route.
Disaster Recovery from a Previous Virtual Machine State
Snapshot-based backup methods enable you to preserve a valid frozen copy of a VM and revert to it at any time. It is preferable to take snapshot copies of switched off VMs to maintain data consistency, however this is not required.
To restore from a prior machine state using Zextras Backup, you must first:
- Restore the most recent valid backup to a different (clone) VM in an isolated network, ensuring that no users can access it and that no incoming or outgoing emails are sent.
- Turn on the clone and wait for Zimbra to load.
- Disable the RealTime Scanner in Zextras Backup.
- Connect the unaltered Backup directory Virtual Disc to the clone and mount it (on a different directory).
- Begin an External Restore by specifying the Backup Path as the Import Path.
This will analyse all objects in the Backup Path and import any missing ones, accelerating catastrophe recovery. These processes can be performed indefinitely as long as user access and mail traffic are restricted.
After the restoration is complete, ensure that everything is operational and that user access and mail traffic are restored.
What Will Happen Next?
If you need to restore content from before the disaster, just create a new Backup Path and delete the old one.
Items That Cannot Be Restored
How can I tell whether all of my belongings have been restored?
It’s quite simple. Check the procedure Completed notice you received as soon as the restoration procedure was completed. It may be read in the Administration Zimlet’s Notifications part, and it is also emailed to the address you provided in the Administration Zimlet’s Core section as the Notification E-Mail recipient address.
The skipped items section includes a list of unrestored things for each account, as illustrated in the following excerpt:
Unrestored Items vs. Skipped Items
Item omitted
An object that has previously been restored, either during this or a prior restoration.
Item that has not been repaired
An object that was not restored owing to a problem with the restoration procedure.
Why haven’t some of my belongings been restored?
There are several potential reasons, the most prevalent of which are:
Error in Reading
Because of an I/O exception or a permission issue, either the raw item or the metadata file is unreadable.
Item that has been broken
Zextras Backup can read both the raw item and the metadata file, however their content is broken/corrupted.
Item is not valid.
The raw item and the metadata file are both readable, and the content is correct, yet Zimbra will not inject the item.
How Can I Spot Unrestored Items?
There are two methods for doing so: the CLI and the Zimbra Web Client. The first method allows you to search for the item inside the backup/import path, while the second allows you to examine the objects on the source server.
How Can I Restore Items That Have Not Been Restored?
An item that cannot be restored indicates that there is a problem, either with the item or with your present Zimbra configuration. In rare circumstances, it is possible to restore an object even if it was not restored on the first try.
The next paragraphs include a variety of suggestions and tactics that might be useful when dealing with various types of unrestorable goods.
Items were not restored due to a read error.
A careful distinction must be made between the read mistakes that might result in objects not being restored:
Hard mistakes, hardware failures, and any other destructive errors that result in unrecoverable data loss are all examples of destructive errors.
Soft mistakes
Non-destructive mistakes include incorrect permissions, filesystem faults, RAID difficulties (for example, defective RAID1 mirroring), and so on.
While there isn’t much you can do about hard mistakes, you can avoid or reduce soft errors by following these guidelines:
- Perform a filesystem check.
- Check the array for potential difficulties (depending on RAID level) if utilising a RAID disc arrangement.
- Check that the ‘zimbra’ user has read/write access to the backup/import path, all subfolders, and all files included inside.
- Examine the connection quality of network-shared filesystems carefully. If the network quality is bad, try using rsync to transmit the data.
- If you’re using SSHfs to remotely mount the backup/import path, use the -o allow_other option to perform the mount command as root.
Items that were identified as broken were not restored.
Unfortunately, this is the least salvageable category of unrestored objects.
Depending on the degree of corruption, it may be feasible to recover either a former state or the raw object (this is only applicable to emails). Use the getItem CLI command to determine the level of corruption:
If the item is an email, you can recover a standard.eml file by following these steps:
Determine the most recent valid state
- Consider the following from the above snippet:
- Determine the blob path
- Take the previous step’s blob path:
- Uncompress the BLOB file into an.eml file using gzip.
- Done! You may now use your preferred client to import the.eml file into the relevant mailbox.
An item is marked as Invalid if, although being formally accurate, it is rejected by Zimbra’s LMTP Validator upon injection. This is typical when transferring items produced on an earlier version of Zimbra to a later version. Validation criteria are frequently revised, thus not all messages declared legitimate by one Zimbra version are still considered valid by another.
- If you encountered a high number of unrestored items during an import, you should temporarily disable the LMTP validator and rerun the import:
- Run the following command as the Zimbra user to deactivate the LMTP Validator:
After the import is finished, you may activate the LMTP validator by running:
Taking Additional and Offsite Backups of Zextras Backup’s DatastoreWho Keeps an Eye on the Watchmen?
Backup systems are an excellent safeguard against data loss, but each backup system must be part of a larger backup plan to provide the maximum degree of dependability. A lack of a good backup plan creates a false sense of security while really causing even the greatest backup solutions in the world to fail.
Creating a backup strategy is not straightforward, and you will almost certainly be presented with the following question: “What if I lose the data I backed up?” The likelihood of this occurring is ultimately determined by how you create and handle backups. For example, if you keep both your data and your backups on the same SATA-II disc, you are more likely to lose all of your backed up data than if you store your backed up data on a separate SAN using a RAID 1+0 arrangement.
Here are some tips and best practises for improving your backup strategy by backing up and storing the Backup NG datastore elsewhere.
Creating a Second Backup of Zextras Backup’s Datastore
To reduce the possibility of data loss, a backup might make use of the well-known database features known as ACID, which ensure data validity and integrity.
By adhering to these principles, it is quite simple to create a backup of the Datastore and ensure the content’s integrity and authenticity. The best (and simplest) method to do so is to use the rsync programme, which is built on a technique that only transmits deltas (what really changed) rather than the entire data set, and operates progressively. Specific choices and parameters are determined by several aspects, including the quantity of data to be synchronised and the storage used, while connecting to an rsync daemon rather than a remote shell as a transport is generally more quicker in transmitting data.
You won’t have to stop Zimbra or the Real moment Scanner to make an additional backup of Zextras Backup’s datastore using rsync, and owing to the ACID characteristics, you’ll always be able to pause and resume the sync at any moment.
Offsite Storage of Your Zextras Backup’s Datastore Backup
As seen in the preceding section, backing up Zextras Backup’s Datastore is simple, and using rsync makes it just as simple to store your backup in a remote place.
The following best practises are advised to optimise your backup plan when working with this type of setup:
- If you plan your rsync backups, be sure that there is adequate time between each rsync instance for the transfer to finish.
- To avoid discrepancies, use the –delete options to ensure that files destroyed on the source server are also deleted on the destination server.
- If you discover that using the –delete option takes too long, plan two separate rsync instances, one with –delete and one without, to run after the weekly purge.
- Transfer the whole folder tree recursively, beginning with Zextras Backup’s Backup Path. This contains backups of server configuration and mapfiles.
- Check that the destination filesystem is case sensitive (exactly like Backup NG’s Backup Path).
- If you intend to restore directly from a remote location, ensure sure the zimbra user on your server has read and write access to the transferred data.
- Expect latency if your transfer speed is significantly higher than your storage throughput (or vice versa).
Additional/Offsite Backup F.A.Q.
Multistore Data Backup in a Multistore EnvironmentMultistore Command Execution in a Multistore Environment
The Network Administration Zimlet makes it easier to administer many servers: Even if you are signed into the Zimbra Administration Console of another server, you may pick a server from the Zextras Backup tab and execute all backup actions on that server.
The following are the key distinctions between Singlestore and Multistore environments:
- Restore on New Account activities in a Multistore system ALWAYS create the new account in the Source account’s mailbox server.
- All operations are recorded on the destination server rather than the server that initiated the action.
- If the incorrect destination server for an operation is selected, Zimbra immediately redirects the operation request to the correct server.
Backup and restoration
Backup and restore operate precisely the same in a Multistore environment as they do in a Singlestore scenario.
The various servers will be configured and maintained independently using the Administration Zimlet, but some actions, such as Live Full Scan and Stop All actions, may be ‘broadcast’ to all mailstores via the zxsuite CLI with the –hostname all_servers option. This also applies to Zextras Backup settings.
Backup and restore activities are managed in the following manner:
- Smartscans may be run on a single server using the Administration Zimlet or across several servers using the CLI.
- Restores may be initiated using the CLI, the Accounts tab in the Zimbra Admin Console, each server tab in the Zextras Backup menu of the Administration Zimlet, or the Accounts tab in the Zimbra Admin Console. The following are the distinctions between these methods:
When working in a Multistore environment, the Export and Import functions are the most distinct. The following are the basic scenarios:
The commands doCheckShares and doFixShares
The doCheckShares command will parse all local account share information and report any errors:
doFixShares will use a migration to correct all share inconsistencies:
The Operation Queue and Queue Management of Zextras Backup
When a Zextras Backup operation is launched, whether manually or automatically, it is enqueued in a dedicated, unprioritized FIFO queue. Each operation is conducted as soon as any previous operation is dequeued (either completed or ended).
The queuing system has an impact on the following operations:
- External storage
- Every restoration operation
- SmartScan
Changes to the configuration of Zextras Backup are not queued and are implemented instantly.
Queue Management in Operation
Backup Management at the COS Level
What is COS-level Backup Management? COS-level Backup Management enables administrators to stop ALL Zextras Backup functionalities for an entire Class of Service in order to save storage use.
What Is the Process of COS-Level Backup Management?
What happens if I turn off the Zextras Backup Module for a certain Class of Service?
- All accounts in the COS will be ignored by the Real Time Scanner.
- The Export Backup feature DOES NOT EXPORT COS accounts.
- The backup system will treat accounts in the COS as Deleted. This implies that after the data retention period expires, all data for those accounts will be removed from the backup repository. Re-enabling a Class of Service’s backup will reset this.
How is the information about backup enabled/disabled saved?
When a Class of Service’s backup is disabled, the following marker is added to the Class of Service’s Notes field: ${ZxBackup_Disabled}
While the Notes field remains completely editable and useful, updating or removing this marking will re-enable the COS backup.