Close menu

Backup on External Storage

External Storage Backup
Carbonio Backup, as stated in the section Architecture of Carbonio Backup, is made up of metadata and blobs (compressed and deduplicated), which are saved by default to the same folder — or mounted drive — indicated in the Backup Path. The Backup Path must be quick enough to avoid queuing processes and/or data loss during real-time backup.
 
However, S3 buckets, NFS shares, and other storage mounted using Fuse can be very sluggish and may not be suitable for backup path storage.
 
Because the metadata is the most significant element of backups, the idea behind Backup on External Storage is to employ two distinct storage systems: one local (and often fast) for metadata and cache and one external (local network or cloud) for blobs and a copy of metadata.
 
Multiple updates will be combined and transmitted together if the external storage is remote, however if it is local, bigger but slower and cheaper storage can be used.
 
Metadata is preserved locally in the Backup Path, and BLOBs are cached on the local disc for a short time before being sent to remote storage as quickly as feasible.
 
SmartScan refreshes the information for accounts that have been updated since the previous scan and archives it on remote storage. 
 

Remote metadata archiving may also be started manually by performing any of the following commands with the remote_metadata_upload true parameter:

  • carbonio backup doSmartScan

  • carbonio backup doAccountScan

  • carbonio backup doBackupServerCustomizations

  • carbonio backup doBackupLDAP

  • carbonio backup doBackupCluster

By separating the I/O heavy metadata folder from the BLOBs folder, it is also assured that the backup will run even if the remote storage is briefly unavailable (for example, due to network difficulties or ongoing maintenance chores), resulting in improved dependability and backup resilience.

Goals and Advantages
It is worthwhile to mention the two primary benefits of backup on external storage:
 
  • Fast IOPS storage is only required for metadata that accounts for less than 10% of the overall backup size.
  • Backups are often kept outside of the local infrastructure and therefore available from disaster recovery sites.

Warning

When activating the Backup on External Storage, it is not possible to modify the Backup Path from the UI. Indeed, the corresponding input text area will only be shown, but can not be edited. Moreover, the following warning will be shown:

“The backup path cannot be managed using this UI since the Backup On External Storage is enabled. Please use the backup CLI commands”

To deactivate External Storage, use the carbonio backup setBackupVolume Default command.

Data Located in External Storage
Data is saved in external storage using a structure that is fairly similar to the Backup Path:
|-- accounts
|-- items
|-- server
`-- backupstat
The external disc is only utilised to store the things in $BACKUP_PATH/items, while the metadata (in $BACKUP_PATH/accounts) will continue to use the local volume as a working directory to store the modified metadata.
 
For example, this operation downloads the most recent metadata from remote storage to the Backup Path.
zextras$ carbonio backup retrieveMetadataFromArchive S3 *destination*
External Storage Types
External volumes that are supported, i.e. shared volumes mounted at the OS level or object storage wholly handled by Carbonio, are of two types: NFS external volumes and Fuse external volumes, which are explained further in this section.
 
External NFS/Fuse Storage
Because no existing volume may be reused, it is important to configure the new volume(s) that will contain the backup before utilising the NFS/Fuse sharing. The actions to take varies depending on the strategy you pick. Only the simplest and most dependable method is described here.

Single-Server installation

When NFS shares are used, you need to make them visible and accessible to both the Operating System and Carbonio, a task that only requires to add a row in file /etc/fstab with the necessary information to mount the volume, for example, to mount volume /media/mailserver/backup/ from a NAS located at 192.168.72.16 you can add to the bottom of /etc/fstab a line similar to:

192.168.72.16:/media/mailserver/backup/  /media/external/ nfs rw,hard,intr, 0,0

You will now be able to mount the external storage by simply using mount /media/external/ on the server.

Multi-Server installation

In the case of a Multi-Server installation, the admin must ensure that each server writes on its own directory, and the destination volume must be readable and writable by the zextras user.

In a Multi-Server installation, consider a scenario in which the same NAS located on 192.168.72.16 is involved, which exposes via NFS the share as /media/externalStorage. We want to store our multiservers backups on this NAS.

To do so, on each server you need to add one entry similar to the following to the /etc/fstab file:

192.168.72.16:/externalStorage/SRV1 /mnt/backup nfs rw,hard,intr 0 0

In our sample Six Nodes Scenario, on each node you need to add the entry above using SRV1, …, SRV6 on the corresponding node, while on the NAS there will be six directories, one for each node.

Storage of external objects
A dedicated Carbonio bucket must be constructed before utilising an ObjectStorage.
 
Carbonio Backup and Carbonio Storages buckets are not compatible with each other, despite their similarity in principle. If Carbonio Storages data is stored in a bucket, Backup data cannot be stored in the same bucket, and vice versa.

How to check a bucket’s usage.

Use the following command to report the bucket usage.

zextras$  `carbonio core listBuckets`

The output will look similar to:

bucketName                                                  hsm
protocol                                                    HTTPS
storeType                                                   S3
accessKey                                                   xxxxx
region                                                      EU_WEST_1
uuid                                                        58fa4ca2-31dd-4209-aa23-48b33b116090
usage in powerstore volumes
                server: srv1                                      volume: centralized-s3
                server: srv2                                      volume: centralized-s3
usage in external backup                                    unused

bucketName                                                  backup
protocol                                                    HTTPS
storeType                                                   S3
accessKey                                                   xxxxxxx
region                                                      EU_WEST_1
destinationPath                                             server2
uuid                                                        5d32b50d-79fc-4591-86da-35bedca95de7
usage in powerstore volumes                                 unused
usage in external backup
                server: srv2
Because each Carbonio bucket is distinguished by a prefix, you may uniquely identify and store numerous Carbonio buckets within a single ObjectStorage bucket by combining bucket credentials and Carbonio bucket prefix.
 
In other words, you may specify numerous Carbonio Buckets on the same S3 Bucket to be used for both HSM and Backup.
Backup of ObjectStorage in a Multi-Server Environment
It is not essential to construct numerous buckets in Multi-Server environments: When you enable remote backup on the first server, you just enter the bucket setup information. The bucket_configuration_id and prefix parameters are then used to store data from other servers in a different directory on the same storage.
 
Enable Backup on External Storage
Once the external storage has been configured, Carbonio must be granted access to it. The technique differs somewhat depending on whether the new storage must be accessible from a freshly installed server or whether existing local backups must be moved to the external storage.

Note

External Storage is a CLI-only feature.

Configure on newly installed / uninitialized server

If there the backup has not been initialized on the server, an Administrator can configure the external storage by running

zextras$ carbonio backup setBackupVolume S3 bucket_configuration_id VALUE [param VALUE[,VALUE]].

Once the backup will be initialized, it will use the external storage.

Therefore, check for any missing blobs with doCheckBlobs in the mounted volumes to avoid integrity errors.

Migrate existing backups

Before actually carrying out the migration, please perform the following important maintenance task. This procedure will minimise the risk of errors:

  1. Double-check the permissions on the active backup path

  2. Make sure that the Carbonio cache folder is accessible by the zextras user (typically under /opt/zextras/cache)

  3. Check for table errors in the myslow.log and in the MariaDb integrity check report. If any error is found, consider running the mysqlcheck command to verify the database integrity.

  4. Check for any missing blobs in the mounted Carbonio volumes with carbonio powerstore doCheckBlobs

  5. Check for any missing digest in the backup with doSmartScan deep=true

  6. Check for any orphaned digest or metadata in the Backup with carbonio backup doCoherencyCheck

  7. Optionally run a carbonio backup doPurge to remove expired data from the Backup

You can now proceed to migrate the existing backup using the appropriate carbonio backup migrateBackupVolume [[ Default | Local | S3 ]] command.

Finally, once the migration has been completed you can run this final task:

  • Manually remove the old backup data. Indeed, the migration only copies the files of the backup to the new external storage and leaves them in the place.

Configure on newly installed / uninitialized server

If there the backup has not been initialized on the server, an Administrator can configure the external storage by running

zextras$ carbonio backup setBackupVolume S3 bucket_configuration_id VALUE [param VALUE[,VALUE]].

Once the backup will be initialized, it will use the external storage.

Therefore, check for any missing blobs with doCheckBlobs in the mounted volumes to avoid integrity errors.

Backup Troubleshooting on Defective Object Storage
There are times when a distant ObjectStorage containing a Backup becomes entirely unavailable, for example, due to a hardware failure.
 
What occurs in this case is unpleasant in several ways:
 
  • All of the data saved in the Bucket has already been lost.
  • When executing the command carbonio core list, the remote bucket still appears carbonio core listBuckets all
  • The Backup is still attempting to use that bucket.
  • The faulty Bucket cannot be removed.
  • Using the command migrateBackupVolume to redirect the backup to a different volume is futile because the remote Bucket is unavailable and inaccessible.
The answer to this deadlock, however, is pretty straightforward, and there are two options:
 
  1. You don’t have another ObjectStorage: use the command
    zextras$ carbonio backup setBackupVolume Default start
  2. The Backup will now utilise the local path by default.
  3. Another ObjectStorage is already accessible to you: Use the following command to establish a new Backup Volume (we’ll use a new S3 bucket as an example)
    zextras$ carbonio backup setBackupVolume S3 bucket_configuration_id 58fa4ca2-31dd-4209-aa23-48b33b116090 volume_prefix new_backup
In all circumstances, you can proceed to delete the volume that is no longer functioning at this point.