The component that is in charge of backing up all the data is Carbonio Backup, which is covered in this chapter. A summary of the most frequent job is provided at the beginning of the chapter, along with connections to more technical sources.
The design of Carbonio Backup is then explained, along with some crucial preparatory knowledge that will be expanded upon in the next sections.
Finally, the options for backing up objects and accounts are described along with the appropriate CLI instructions. Tasks that can be completed using the GUI may be found in the Backup area of the Admin Panel, while those that can be completed using both the CLI and GUI are cross-referenced between the two sections so you can select your preferred method of execution.
As a result, the Backup documentation is divided into four primary sections:
- The current page for backup (of an AppServer) contains information on the architecture of the backup modules, a glossary of terminology that are important, the most frequent backup actions, and how to perform each one using the CLI.
- Restore Techniques for the Backup: CLI-based methods for restoring specific objects, accounts, or whole AppServers
- Advanced Backup Techniques, including Disaster Recovery, a group of last-ditch recovery options in the event of hardware or software issues (unrelated to Carbonio)
- All tasks that can only be completed through the GUI are included in the backup of the Admin Panel.
Common Tasks for Carbonio Backup
In addition to connections to technical resources, this section includes instructions for the most often job requested by users.
How to Run the Carbonio Backup Programme
The Backup component must be configured after you have done setting up your server in order to have all of your data automatically backed up.
- Install a storage device where you want it. In this step, the default path is /opt/zextras/backup/zextras; make sure to change it to the one you selected.
- Set the backup path’s correct permissions: /opt/zextras/backup/zextras chown zextras:zextras
Building Blocks of Carbonio Backup
The key ideas necessary to comprehend the architecture of Carbonio Backup are introduced in this section, along with an overview of how they relate to one another. Each idea is then further explored in its own area.
RPO and RTO are two broad techniques that are taken into consideration when developing a backup plan, which we will discuss before moving on to the architecture of Carbonio Backup.
The Recovery Time Objective (RTO) is the maximum length of time a stakeholder is willing to wait to recover its data, whereas the Recovery Point Objective (RPO) is the maximum quantity of data that a stakeholder is willing to lose in the event of a disaster.
These definitions state that the ideal acceptable value is zero, although practical values typically fall within a range of zero, depending on the amount of the data. Realtime Scanner and SmartScan work together in Carbonio to ensure that both RTO and RPO numbers are very low: While the SmartScan replicates all objects that have been updated, the Real Time Scanner guarantees that all metadata changes are recorded as soon as they occur. As a result, the potential loss of data is minimal and often restricted to those items that have changed between two consecutive runs on the SmartScan.
The idea of ITEM serves as the foundation for Carbonio Backup’s whole architecture. The smallest thing that may be kept in the backup is called an item. For instance:
- email communication
- a person or a group of people
- a binder
- an arrangement
- a task
- a file from the Carbonio Files
- a user account and its settings
- an email list
- an area
- a category of services
The Realtime Scanner will never check for changes on these objects and will never include them in a restore since they are not items:
- Server configuration, or the parameters for each server
- Global Carbonio product settings
- Any modifications (Postfix, Jetty, etc.) added to the programme
Every change in the related information for each object maintained by Carbonio is tracked and preserved, enabling its restoration at a later time. In other words, if one of an item’s associated information changes, a transaction-based “photograph” of the entire thing is taken and saved with a timestamp. Metadata that describes an item includes, for instance:
- when an email was opened, read, deleted, and transferred to a folder.
- a change to a contact’s name, address, or employment
- the removal of a file from a folder or its insertion
- the modification of an item’s status (such as an account).
Technically, all modifications made to an object over its lifespan are kept in the form of a JSON Array. Read more about this in the section on item structure.
An object that has been designated for deletion is known as a deleted item.
A transaction is a change in the state of an object. Change of status refers to when a user modifies one of the item’s related metadata. As a result, a Transaction may be thought of as a snapshot of the metadata taken at a certain point in time. A Transaction ID serves as a unique identifier for every transaction. An item can be returned to any previous transaction. Restore Strategies has further information.
Realtime Scanner and SmartScan
The SmartScan’s Initial Scan builds the backup’s initial structure by reading an AppServer’s actual content and using it to create the backup. If the Scan Operation Scheduling is activated in the Carbonio Admin Panel, the SmartScan is then run each time the Carbonio Backup starts and once per day.
The main function of SmartScan is to look for objects that have changed since its last run and to update the database with any new data.
A recovery with split-second precision is feasible because to the Realtime Scanner’s live recording of every system event. Every item has its own entire history since the Realtime Scanner does not erase any data in the backup. Additionally, it has the capacity to recognise other modifications pertaining to the same item at the same time and record them all as a single metadata change.
Realtime Scanner and SmartScan are both by default activated. While each one can be stopped (on their own), it is advised to leave them both running because they are meant to work best together.
When to Turn Off Scanning Operations
Since backups are stored on disc, access to the disc is required during scan procedures. There are so many situations in which either the SmartScan or Realtime Scanner could (or should), even temporarily, be deactivated. For instance:
- You frequently deal with Carbonio Files documents or have a large volume of daily transactions, and you see that the server is using a lot of resources. In this situation, the Real Time Scan can be temporarily disabled.
- You begin a migration: In this situation, it is advised to suspend the SmartScan since it would cause several I/O operations on the disc and maybe even cause the server to become unresponsive. In fact, it would regard each object that was moved or restored as a brand-new one.
- Your daily email volume is heavy, both inbound and outbound. In this situation, you need always have the Realtime Scanner running because otherwise, the SmartScan would be the only backup for all transactions, and because I/O operations demand a lot of resources, it might not be able to finish in a timely manner.
Backup Route
The location on a filesystem where all the data regarding the backup and archives is saved is known as the backup path. There is only one backup path available per server; many servers cannot share the same backup path. It is organised as a hierarchy of directories, with /opt/zextras/backup/zextras/ by default at the top. The following significant files and folders are located under this directory:
- Map files, also known as map_[server_ID] files, are used to indicate if a backup has been imported from an external backup and include the server’s unique ID in the filename.
- All of the accounts defined in the AppServer’s definitions are listed in the directory called accounts. You may find there, in particular, the following significant files and directories:
- The file account_info contains all of the account’s details, such as the password, signature, and preferences.
- account_stat is a file that contains statistics about the account, such as the ID of the most recent content that SmartScan saved.
- The timestamp of the first run is one of the general data about the backup that are maintained in the file called backupstat.
- The directory drive_items has up to 256 subfolders, each of which is named with two lowercase hexadecimal letters. These subfolders are where Carbonio Files items are saved, according to the last two letters of their UUID.
- items is a directory that has up to 100 subfolders. Items are kept in these subfolders according to the last two digits of their IDs.
- servers is a directory that holds archives, one per day up to the chosen server retention duration, containing the server configuration and customizations, the Carbonio configuration, and the chat.
- items is a directory with a name made up of two hexadecimal (uppercase and lowercase) characters that may hold up to 4096 extra folders. The final two characters of each item’s ID will be used as the name of the directory where it will be kept in the AppServer.
- A map between the original object and the restored item is contained in the user object ID mapping file called id_mapper.log. The file may be found in /backup/zextras/accounts/xxxxx-xxxx-xxxx-xxxxxxxxxx/id_mapper.log. Only in the event of an external restoration is this file present.
Choosing a backup route
Every object and its associated information are preserved in a backup path. Each server is required to establish a single backup path that is exclusive to that server and is never reused. In other words, attempting to designate a Backup Path as the current Backup Path on a separate server would result in an error. Altering the backup file in an attempt to force this condition will damage both the old and the new backup data.
You may use the command to get the Backup Path’s current value.
Use the set sub-command to modify the Backup Path rather than get and append the new path.
The notification “OK” will appear after a successful procedure.
Retention Guidelines
How many days after being designated for deletion an object is actually erased from the backup depends on the retention policy (also known as retention period). The Backup’s retention guidelines are as follows:
- The default setting for the data retention policy for single items is 30 days.
- The account retention policy relates to the accounts and is set to 30 days by default.
The Backup Purge won’t execute if any retention times are set to zero (zero), which means archives will be retained indefinitely (infinite retention).
By utilising the appropriate, you may determine the Retention Policy’s current value.
Use 0 for unlimited retention or any integer value for the amount of days to update either value. For instance, to set the account and data retention to 15 days, use the following syntax: If an account is deleted and needs to be restored after the Data retention time has passed, it will still be possible to recover all items up to the Account retention time because in that scenario, even if all the metadata have been purged, the digest may still contain the data needed to restore the item.
Backup removal
The Backup Purge is a cleansing procedure that eliminates any deleted item that has outlived the retention period stipulated by the Data Retention Policy and Account Retention Policy from the Backup Path.
Check for Coherence
The Coherency Check examines a Backup Path more thoroughly than SmartScan does in order to identify faulty information and BLOBs.
The Coherency Check thoroughly examines all of the information and BLOBs in the Backup Path, whereas the SmartScan simply checks objects that have changed since the last SmartScan run.
Use the carbonio backup doCoherencyCheck carbonio_backup_docoherencycheck> command to launch a Coherency Check from the CLI:
How Carbonio Backup Operates
Every single variant of an ITEM may be stored via Carbonio Backup. It is compatible with many OS architectures and Carbonio versions because it is not designed to be a system or Operating System backup.
Administrators may recover various items on various accounts or even on other servers using Carbonio Backup, which enables them to generate an atomic backup of every item in the AppServer account.
All backup files are by default saved in the local directory /opt/zextras/backup/zextras/ by Carbonio Backup. A directory has to meet certain requirements in order to be utilised as the Backup Path.
- by the zextras user to be able to read and write to.
- Use a filesystem that recognises case
When Carbonio Backup is first launched, a SmartScan is launched to get all data from the AppServer and build the first backup structure. Each item is stored along with all of its information as a JSON array on a case-sensitive filesystem. After the first start, the backup may be kept current and synced with the account by using either the Real Time Scanner, the SmartScan, or both.
The Composition of an Item
A JSON Array serves as the item’s basic structure and keeps track of all changes made to it over its lifetime. Examples of such changes include information about emails (such as tags, visibility, or emails moved to a folder), contacts, tasks, single folders, groups, or Carbonio Files documents, as well as user preferences (such as password hashes and general settings).
For example, it is not useful to store the user’s last login time or the IMAP and Activesync states because if the account will be restored on a new one, the values of those attributes would be related to the old account. This is done to improve performance.
We may recover data at a certain point in time by gathering the transaction’s timestamp.
The engine examines all valid transactions while performing the restoration, assessing the “start-date” and “end-date” characteristics.
The same reasoning is applied for retrieving deleted things: when an item is removed, the timestamp is stored, allowing us to recover deleted items that occurred within a certain time period.
The metadata maintains a record of the validity of the old and new digests even if the blob linked with the item changes and, as a result, its digest does too (as is the case for Carbonio Files Document).
SmartScan
The SmartScan only runs on accounts that have changed since the last SmartScan; as a result, it may dramatically increase system performance and cut down on scan time.
If Scan Operation Scheduling is enabled in the Carbonio Backup section of the Carbonio Admin Panel, a SmartScan is automatically scheduled to run each night. A Purge is done together with a SmartScan once a week on a day chosen by the user, clearing the volume on which the Carbonio Backup is kept of any deleted items that have outlived their retention term.
Searching for things updated since the last SmartScan, the Carbonio Backup engine checks every item in the Carbonio mailbox. It generates any items that are missing from the backup and updates any entries that are out of date, while marking as destroyed any items that are discovered in the backup but not in the Carbonio mailbox.
The backup is then updated with all configuration metadata, resulting in the storage of domains, accounts, COSs, and server settings in addition to a dump of all configuration.
When the backup includes LDAP data, SmartScan will create a compressed dump in the Backup Path that may be used independently to fix a faulty setup.
For every account when the External Restore feature is turned on, SmartScan prepares a single (daily) archive that contains all of the account’s metadata and saves it on the external volume. Information regarding backup on external storage may be found there.
Admin Panel Global Server Settings Server Config is where Smartscan’s configuration and manual execution are done.
Realtime Scanner is an engine that is closely linked to the AppServer. It intercepts all transactions that occur on each user mailbox and records them in order to preserve an item’s whole history for its full lifespan.
Any object may be recovered at any moment with the help of the Realtime Scanner.
The AppServer’s events are all read by the Realtime Scanner in practically real-time, and after that, it duplicates those actions on its own data structure by adding objects and modifying their information. Since nothing is ever overwritten, each object in the backup has a full history of its own.
Safety Scan and Limitations
The Empty Folder item in the right-click context menu is the key restriction when recovering data obtained with the Realtime Scanner.
In this situation, as well as any other time Carbonio Backup is unable to detect an item’s state from the information stored by the Realtime Scanner, a Smartscan on the specified account is initiated prior to the restore. This process corrects any misaligned data and sanitises the backed up metadata.
Block-free Backup Mode
Blobless Backup Mode is a feature of Carbonio that protects all other item-related data while forgoing backing up item blobs.
This mode is intended to maximise the backup module’s use of disc space and recovery speed while utilising advanced storage features of the storage solution, such as built-in backup or data replication.
To activate Blobless Backup Mode, there is just one prerequisite.
- There must be no independent third-party volumes: Only local volumes and centralised third-party volumes may be used with Blobless Backup Mode.
No matter whether storage vendor is used, Blobless Backup Mode can be activated on any server or infrastructure that satisfies the aforementioned conditions.
Blobless Backup Mode functions exactly like its default counterpart: RealTime Scanner handles backing up item changes, while SmartScan handles domain/COS/account consistency. The only difference between the two is that in Blobless Backup Mode, there are no items of type blob in the backup, but all metadata and transaction history are still saved.
It is crucial to keep in mind that, once activated, Blobless Backup Mode impacts the whole server and, regardless of the target volume and HSM settings, no blobs get backed up.
Blobless Backup Mode is a CLI-only feature that may be turned on or off using the backup command line interface.To activate it worldwide, for instance, use the BloblessMode configuration element at the global and server levels:
Or you might restrict its use to the domain mail.example.com:
Backup Purge:
This cleansing procedure removes from the Backup Path any deleted items that have been removed for a longer period of time than allowed by the Retention Policy.
When an item designated for deletion is discovered whose last update is older than the retention time limit, the purge engine checks the metadata of all the deleted items and deletes the item from the backup.
Notably, thanks to Carbonio Backup’s integrated deduplication, the blob of an item won’t be removed if it is still referenced by one or more valid metadata files.
Both manually starting the Backup Purge from the CLI and scheduling it through the Admin Panel (Admin Panel + Global Server Settings Server Config) are options.
However, take notice that the Backup Purge will end right away if infinite retention is enabled (i.e., the Data Retention Policy is set to 0), as no deleted item will ever go beyond the retention duration.
Limitations and Unique Situations for the Backup
- In a few instances, the backup does not function properly. Here, we go over such situations.
- The most recent state should NOT be used when restoring an active account to a new account. Imagine if a person accidentally deletes all of his emails or that the emails in a user’s account are lost for any other cause (such, for example, a server failure). The user begs the administrator to give them back. The outcome of restoring the account status to the most recent state accessible is that the new account will include that state, which is an empty account because the emails had already been erased in the most recent state. Therefore, it is important to restore the account to a point in time that is before the emails were erased in order to do so appropriately.
- an email client is set up to download emails and promptly delete them from the server while utilising the POP3/POP3S protocol, these emails might not be backed up. If the Carbonio Advanced component is installed, this does not take place.
- The backup does not include emails sent directly over an SMTP connection (such as when using a multifunctional device or telnet to connect to the STMP server).
- If an IMAP/SMTP client is used to send email, the IMAP client must be set up to store the email in a remote folder (using the IMAP STORE command) after the send operation in order for the email to be backed up.
Use external storage as a backup
Carbonio Backup is made up of metadata and blobs (compressed and deduplicated), which are by default stored on the same folder—or mounted volume—specified in the Backup Path, as explained in the Architecture of Carbonio Backup. The Backup Path must be quick enough to prevent activities from queuing and/or risk data loss in order to provide real-time backup.
S3 buckets, NFS shares, and other storage mounted using Fuse, on the other hand, can be extremely sluggish and might not be suitable for mounting on the Backup Path.
The idea behind Backup on External Storage is to employ two distinct storage: one local (and often fast) for metadata and cache and one external (local network or cloud) for the blobs and a copy of the metadata. This is because the metadata is the most crucial component of backups.
Multiple updates will be combined and delivered together if the external storage is remote. If the external storage is local, bigger, slower, and less expensive storage can be used.
Metadata are locally preserved in the Backup Path, BLOBs are briefly cached on the local disc, and they are sent as quickly as possible to the distant storage.
When an account has been updated since the last scan, the SmartScan locally updates the metadata for such accounts and archives them on the remote storage.
Running one of the following commands with the remote_metadata_upload true option will manually start the remote metadata archiving:
- backup carbonio doSmartScan
- backup carbonio doAccountScan
- backup using carbonio, doBackupServerCustomizations
- backup using carbonio doBackupLDAP
- backup carbonio doBackupCluster
Better reliability and backup resilience are provided by dividing the I/O-intensive metadata folder from the BLOBs one. This also ensures that the backup continues to function even if the remote storage is briefly unavailable due to network problems or ongoing maintenance.
Goals and Advantages
The following are the top two benefits of backing up to external storage:
- Only metadata that statistically makes up less than 10% of the overall backup size requires fast IOPS storage.
- The majority of the time, backups are kept outside, away from the local infrastructure, and are therefore reachable from disaster recovery sites.
You may use the carbonio backup setBackupVolume Default command to deactivate the External Storage.
Information Kept in External Storage
External storage uses a structure very similar to the Backup Path to store data.
Only the $BACKUP_PATH/items are stored on the external drive; however, the metadata, which is stored in the $BACKUP_PATH/accounts, continues to utilise the local disc as a working directory to keep the modified metadata.
There are a number of commands specifically designed to download metadata from external storage, reconstruct the account’s structure and content in the event of a disaster, or update/correct local information.
For instance, this programme downloads to the Backup Path the most recent metadata that is accessible in the remote storage.
Types of External Storage Supported external volumes, or shared volumes mounted either at the OS level or object storage completely handled by Carbonio, are of two types: NFS or Fuse external volumes, which are discussed in the remaining sections of this chapter.
External NFS/Fuse Storage
Because no existing volume may be reused, it is important to configure the new volume(s) that will contain the backup before utilising the NFS/Fuse sharing. The actions you take vary depending on the strategy you decide on. Here, we simply cover the simpler and more dependable option.
Storage for external objects
A specific Carbonio bucket has to be constructed prior to using an ObjectStorage.
In spite of having a similar concept, the Carbonio Backup and Carbonio Advanced buckets are incompatible. It is not feasible to keep Backup data in the same bucket as Carbonio Advanced data, and vice versa.
You may utilise the combination of bucket credentials and the prefix of a Carbonio bucket to uniquely identify and store numerous Carbonio buckets inside of a single ObjectStorage bucket since each Carbonio bucket is identified by a prefix.
In other words, several Carbonio Buckets might be defined on the same S3 Bucket and used for both backup and HSM
Backup of objects from storage in a multi-server scenario
Multiple buckets are not required in multi-server environments: When configuring remote backup on the first server, you just enter the bucket configuration data. The data from additional servers may then be kept in a different directory on the same storage by using the prefix and bucket_configuration_id options.
Make Backups Available on External Storage
It is required to permit Carbonio to utilise the external storage after it has been configured. Depending on whether the new storage must be accessible from a freshly installed server or whether current local backups must be moved to the external storage, the process is slightly different.
The answer to this dilemma is, however, rather straightforward, and there are really two options:
You lack a second ObjectStorage option: use the instructionRepairing Backups on Damaged ObjectStorage
Unfortunate situations exist where a distant ObjectStorage storing a Backup totally disappears, for instance due to a hardware issue.
- What transpires in this case is regrettable in a number of ways:
- Data stored on the Bucket has already been completely gone.
- The command carbonio core list still displays the remote bucket.All buckets
- Still attempting to utilise that bucket is the backup
- There is no way to get rid of the flawed bucket.
Because the remote Bucket is unavailable and inaccessible, using the command migrateBackupVolume to reroute the backup to a new volume is unsuccessful.
- But there is a very easy way out of this situation, and there are really two options:
- employ the command
- The default, local path will now be used by the backup.
- Another ObjectStorage is already accessible to you: Use the following command to establish a new Backup Volume (we’ll use a new S3 bucket as an example).
Now that the volume is no longer effective in any scenario, you may delete it.