Have a Question?
Each Zextras Suite installation includes one major volume and a configurable number of supplementary volumes. The Powerstore module’s function is to manage secondary volumes and transport stuff between them.
Using Hierarchical Storage Management (Hierarchical Storage Management), a policy-based system, items can be moved as follows: One of the most beneficial is to reserve the fastest storage for intense I/O operations and data with frequent access, while the slower storage will manage older data.
The rest of this part goes on volume management, policies, HSM, and many advanced approaches.
The Basics of Zimbra Stores: Store Types and Their Applications
Zimbra supports two distinct types of stores:
Index Keeping Place
Apache Lucene uses a store that includes information about your data to enable indexing and search features.
Data Storage Facility
A repository containing all of your Zimbra data in a MySql database.
Although you can have numerous stores of each kind, only one Index Store, one Primary Data Store, and one Secondary Data Store can be designated as Current (indicating that it is presently being utilised by Zimbra).
Data Stores, Primary and Secondary
In Zimbra, a data store can be either a Primary Data Store or a Secondary Data Store.
A stated policy is used to migrate data between the current Primary Data Store and the current Secondary Data Store.
Volumes
Zimbra defines three sorts of volumes:
The Primary Current
A volume into which data is written as soon as it arrives.
Alternate Current
A volume into which data is written once an HSM policy has been applied.
Volumes that are not marked as Current and on which data is written only by certain manual processes.
Items are put on the destination server’s Primary Current volume by default.
The Centralised Storage feature enables the use of an S3 bucket to host data from multiple servers at the same time that share the same directory structure, as opposed to “independent” volumes that are self-contained and whose directory structure is strictly related to the server and volume itself.
This improves data management in big multistore settings and significantly increases mailbox transfer speed.
The following are two critical elements of centralised storage that should be considered before deloying:
- Item deduplication is no longer possible.
- S3 buckets are the only ones that may be utilised for centralised storage.
Creating Centralised Storage
A few steps are required to set up a bucket for centralised storage: Make a bucket, connect to it, create volumes on each mailstore, and set the volume to current.
The recommended technique is as follows, and it necessitates the usage of CLI commands.
- Using the ZxCore command doCreateBucket, create an S3 bucket:
We use the following values in this example:
- S3 as the bucket type
- BucketName is the name of the bucket; it must match the name on the remote provider else the command will fail.
- as the remote username X58Y54E5687R543
- as remote password abCderT577eDfjhf
- The URL for the connection should be https://example_bucket_provider.com. You can also specify the provider’s IP address instead of the URL.
- For additional information and alternatives, see the doCreateBucket S3 complete reference.
- When the command is successful, it returns a string containing the unique bucket ID, such as 28m6u4KBwSUnYaPp86XG. Take note of it because it will be needed later in the operation.
- Use the bucket ID obtained in the previous step (28m6u4KBwSUnYaPp86XG) to test the connection:
Proceed to the next stage if the command is successful.Assign the bucket to the first mailstore’s volumes.
powerstore by zxsuite doCreateVolume S3 _Store name_ _primary|secondary_ [param VALUE[,VALUE]]
In this example, the following values are used:
- S3: the bucket type
- VolumeName: the name of the volume as defined on the server where the command is run.
- secondary: the volume’s kind
- 28m6u4KBwSUnYaPp86XG: the bucket ID* obtained in step 1.
- volume_prefix myVolPrefix: a volume ID used for rapid searches (e.g., myVolPrefix).
- The volume is created as centralised true
- For additional information and alternatives, see the doCreateVolume S3 complete reference.
- Set the volume to current to allow it to receive data right away:
- In this example, the following values are used:
- S3: the bucket type
- VolumeName: the name of the volume as defined on the server where the command is run.
- secondary: the volume’s kind
- For additional information and alternatives, see the doUpdateVolume S3 complete reference.
After you’ve built the Centralised Volume, you must duplicate its settings from the first server to all mailbox servers and add it to the volume list. To accomplish so, perform the following instructions on all other mailbox servers:
The second command that must be executed is the one mentioned in the previous step:
Storage Structure Centralised Storage Structure The main directory of a Centralised Volume comprises a single empty directory for each server connected to the volume, as well as a directory for each mailbox stored in it at the same level.
Servers 3aa2d376-1c59-4b5a-94f6-101602fa69c6 and 595a4409-6aa1-413f-9f45-3ef0f1e560f5 are both linked to the same Centralised volume, which contains three mailboxes. As you can see, the storage is unaffected by the effective server where the mails are located.
Volume Management Both primary and secondary volumes may be built on either local storage or third-party storage solutions that are supported.
Volumes for Zimbra
A volume is a unique item (path) on a filesystem that has all of the relevant attributes for Zimbra Blobs.
Volume Characteristics
The following properties characterise all Zimbra volumes:
- Name: The volume’s unique identification.
- Path: The location where the data will be saved.
- File compression for the volume can be enabled or disabled.
- Compression Threshold: The smallest file size that will cause compression to occur. ‘File sizes less than this will never be compressed, even if compression is enabled.’
- Current: A Current volume is one where data is written upon arrival (Primary Current) or where HSM policy is applied (Secondary Current).
Volumes at a Local Level
Local Volumes (FileBlobs) can be hosted on any mountpoint on the system, independent of its destination, and are characterised by the following properties:
- Name: The volume’s unique identification.
- Path: The location where the data will be saved. This route requires r/w access for the zimbra user.
- File compression can be enabled or disabled for the volume.
- Compression Threshold: the smallest file size that will cause compression to occur.
Currently Available Volumes
A Current Volume is a volume into which data will be written upon arrival (Primary Current) or where an HSM Policy Application will be written (Secondary Current). Except for specialised manual procedures such as the Volume-to-Volume transfer, volumes that are not set as Current will not be written on.
Volume Control using Zextras Powerstore
Hierarchical Storage Management is a storage management technique.
HSM is a data storage mechanism that transfers data between multiple repositories based on a set of rules.
The HSM approach is most commonly used to migrate older data from a faster-but-expensive storage device to a slower-but-cheaper one based on the following assumptions:
- Fast storage is more expensive.
- Slow storage is less expensive.
- Data from the past will be retrieved far less frequently than data from the future.
The benefits of the HSM approach are obvious: lower overall storage costs because just a tiny portion of your data requires expensive storage, and improved overall user experience.
Policies, Stores, and Volumes
Using HSM necessitates a thorough grasp of the following terms:
- Primary Store: The quick-but-cheap store where all of your data is originally stored.
- Secondary Store: A slow-but-cheap storage location where older data will be relocated.
Changing Items Between Stores
The ability to apply specific HSM rules is the key feature of the Zextras Powerstore module.
There are three ways to initiate the move:
- In the Administration Zimlet, click the Apply Policy button.
- Begin the doMoveBlobs action with the CLI.
- In the Administration Zimlet, enable Policy Application Scheduling and wait for it to start automatically.
Following the start of the relocation, the following procedures are carried out:
- Zextras Powerstore scans the Primary Store to determine which goods adhere to the stated policy.
- All of the Blobs from the first step are transferred to the Secondary Store.
- The database records for the copied objects are modified to reflect the relocation.
- If the second and third phases are successfully performed (and only in this case), the old Blobs are removed from the Primary Store.
- Because the Move operation is stateful (each step is done only if the previous step was successfully completed), there is no danger of data loss during the Move process.
- doMoveBlobs Zextras Powerstore’s doMoveBlobs Operation
Zextras Powerstore’s heart is the doMoveBlobs.
It transfers objects between the Current Primary Store and the Current Secondary Store in accordance with HSM rules.
A transactional algorithm executes the move. If an error occurs during one of the operation’s steps, a rollback occurs and no changes are made to the data.
Following the identification of the objects to be relocated by Zextras Powerstore, the following actions are taken.
- The Blob gets copied to the Current Secondary Store.
- The Zimbra Database is updated to reflect the item’s new location.
- The original Blob is removed from the Primary Store.
What is transferred? Every object that complies with the HSM policy defined gets transferred.
Policy Directive
All policy criteria are carried out in the precise order that they are given. Zextras Powerstore will cycle over all items in the Current Primary Store, applying each distinct condition before moving on to the next.
This indicates that the following rules, when implemented daily on an example server that sends/receives 1000 emails each day, 100 of which contain one or more attachments, will provide the same results. The execution time of the second policy, on the other hand, will most likely be slightly longer (or considerably longer, depending on the volume and size of emails on the server).
This is because the first condition (message,document:before:-20day) in the first policy will loop on all items and shift many of them to the Current Secondary Store, leaving fewer items for the second condition to loop on.
Similarly, using message:before:-10day has:attachment as the first condition would result in more entries for the second condition to loop on.
This is simply an example and does not apply in all instances, but it illustrates the need of properly planning your HSM policy.
Performing the doMoveBlobs Operation (also known as applying the HSM Policy)
Applying a policy entails performing the doMoveBlobs operation to transfer objects between the Primary and Secondary stores in accordance with the policy.
Zextras Powerstore provides three distinct options:
- Zimlet through Administration
- Using the CLI
- Through Scheduling doMoveBlobs Stats and Info Information on disc space savings, operation performance, and more is available by selecting the Stats button in the Secondary Volumes list of the Administration Zimlet’s Zextras Powerstore page.
Policy Management What exactly is a policy?
An HSM policy is a collection of rules that specify which objects are moved from the Primary Store to the Secondary Store when the doMoveBlobs operation of Zextras Powerstore is initiated, either manually or automatically.
A policy might be composed of a single rule that applies to all item types (simple policy) or many rules that apply to one or more item types (complex policy). Additionally, using Zimbra’s search syntax, an extra sub-rule may be established.
Examples of Policies
Here are some instances of policies. See below for instructions on how to build policies in the Zextras Powerstore module.
All articles older than 30 days should be relocated.
Move emails older than 15 days and other stuff older than 30 days.
Move calendar entries that are more than 15 days old, Drive items that are more than 20 days old, and all emails in the “Archive” folder.
Creating a Policy
Policies may be defined using both the Administration Zimlet’s Zextras Powerstore tab and the CLI. In both circumstances, you may define a Zimbra Search.
Powerstore and S3 buckets from Zextras
Zextras Powerstore Primary and Secondary volumes may be hosted on S3 buckets, thereby shifting the majority of your data to safe and long-term cloud storage.
Services that are compatible with S3
While any storage provider that supports the Amazon S3 API should work out of the box with Zextras Powerstore, the only officially supported platforms are mentioned below:
- FileBlob (normal local volume)
- S3 from Amazon
- EMC
- OpenIO
- Swift
- S3 scalability
- Cloudian
- Custom S3 (any non-supported S3-compliant solution), with V2 and V4 authentication
The “Incoming” directory and the Primary Volumes
A local “Incoming” directory on the mailbox server must exist in order to construct a remote Primary Store. The default directory is /opt/zimbra/incoming; use the following commands to verify or change the current value:
Cache on a local level
Storing a volume on third-party remote storage solutions necessitates the usage of a local directory for item caching that is accessible and written by the zimbra user.
The local directory must be manually established, and its path must be entered in the Zimbra Administration Console’s Zextras Powerstore part of the Administration Zimlet.
You will be unable to build any secondary volume on an S3-compatible device or service if the Local Cache directory is not configured.
Zextras Powerstore does not require any dedicated S3 settings or configuration, thus creating a bucket for your volumes is simple. Although building a specific user bucket and access policy is not essential, it is strongly recommended because it makes management much easier.
To begin storing secondary volumes on S3, all you need to do is:
An S3 container. To utilise the bucket, you must first know its name and area.
The Access Key and Secret of a user.
A policy that provides the user complete access to your bucket.
Bucket Management The Zimbra Administration Console has a centralised Bucket Management UI. This allows you to save bucket information to be reused when building a new volume on an S3-compatible storage rather than inputting it each time.
To access the Bucket Management UI, follow these steps:
- Navigate to the Zimbra Administration Console.
- Choose “Configure” from the left menu.
- Choose the “Global Settings” option.
- Choose the S3 Buckets option.
When building a new volume of the following types, any bucket uploaded to the system will be available: Amazon S3, Ceph, Cloudian, EMC, Scality S3, Custom S3, Yandex, Alibaba.
The zxsuite-core-doCreateBucket commands may also be used to create new buckets using the CLI: [Alibaba Ceph Cloudian CustomS3 EMC S3 ScalityS3 Yandex Ceph Cloudian CustomS3 ScalityS3 Yandex
Paths and names for buckets
Files are kept in a bucket according to a well-defined path, which may be customised at any time to make the contents of your bucket easier to comprehend, even in multi-server situations with various secondary volumes:
The Bucket Name and Destination route are not related to the volume itself, and you may have as many volumes as you like under the same destination route.
The Volume Prefix, on the other hand, is unique to each volume and serves as a rapid method of distinguishing and recognising distinct volumes inside the bucket.
Using Zextras Powerstore to Create Volumes
To add a new volume to the Zimbra Administration Console, follow these steps:
In the Zimbra Administration Console, navigate to the HSM Section of the Zextras Administration Zimlet.
- Select Add from the list of Primary or Secondary Volumes.
- Choose the Volume Type from the available storage options.
- Enter the volume information that is necessary.
Volume Editing using Zextras Powerstore
To modify a volume using Zextras Powerstore via the Zimbra Administration Console, follow these steps:
In the Zimbra Administration Console, navigate to the HSM Section of the Zextras Administration Zimlet.
- Choose a volume
- Select Edit.
- When finished, click Save.
Volume Removal Using Zextras Powerstore
To remove a volume using Zextras Powerstore via the Zimbra Administration Console, follow these steps:
In the Zimbra Administration Console, navigate to the HSM Section of the Zextras Administration Zimlet.
Choose a volume
Click the Delete Amazon S3 Tips Bucket button.
There are no required bucket requirements for storing your secondary Zimbra volumes on Amazon S3, but we recommend that you build a dedicated bucket and deactivate Static Website Hosting for simpler administration.
User
A Programmatic Access user is required to receive an Access Key and the associated Secret. For better management, we recommend that you create a dedicated user in Amazon’s IAM Service.
Management of Rights
You may configure access policies for your users in Amazon’s IAM. The user of your Access Key and Secret must have adequate access privileges to both the bucket and its contents. We advocate offering complete permissions for easy administration, as seen in the following example.
Change the Action section to: if you simply want to provide the most basic permissions.
The bucket’s ARN is formatted in the following way: arn:partition:service:region:account-id:resource. Please visit Amazon’s documentation for additional information on this subject.
Paths in Buckets and Naming
Files are kept in a bucket according to a well-defined path, which may be customised at any time to make the contents of your bucket more understandable (even in multi-server situations with various secondary volumes):
The Bucket Name and Destination route are not related to the volume itself, and you may have as many volumes as you like under the same destination route.
The Volume Prefix, on the other hand, is unique to each volume and serves as a rapid method of distinguishing and recognising distinct volumes inside the bucket.
Zextras Powerstore is compatible with the Amazon S3 Standard – Infrequent Access storage class and will assign any file bigger than the Infrequent Access Threshold value to this storage class as long as the option is enabled on the volume.
Zextras Powerstore is compatible with the Amazon S3 – Intelligent Tiering storage class and will set the relevant Intelligent Tiering flag on all files as long as the option is enabled on the volume.
Item Deduplication explains what it is.
Item deduplication is a method of saving disc space by storing a single duplicate of an item and referring it numerous times rather than storing several copies of the same item and accessing each copy only once.
This may appear to be a tiny improvement. However, it makes a huge difference in practise.
Item Deduplication in Zimbra Zimbra performs item deduplication when a new item is stored in the Current Primary Volume.
When a new item is produced, its message ID is checked against a list of previously cached items. If a match is found, a hard link to the cached message’s BLOB is formed rather than a new BLOB for the message.
Zimbra manages the dedupe cache using the following configuration parameters.
(Older versions of Zimbra may utilise other qualities or lack some of them.)
Zextras Powerstore and Item Deduplication
A doDeduplicate action in the Zextras Powerstore parses a target volume to detect and deduplicate any duplicated items.
You will save much more disc space because, unlike Zimbra’s automated deduplication, Zextras Powerstore’s deduplication will locate and handle numerous copies of the same email independent of cache or time.
Running the doDeduplicate action after a migration or a large data import is also strongly recommended to optimise your storage consumption.
Performing Volume Deduplication
The getAllVolumes command may be used to list all accessible volumes.
doDeduplicate Statistics
The doDeduplicate operation is a valid monitor command target, which means you may view the command’s statistics while it’s executing using the zxsuite powerstore monitor [operationID] command. The following is an example of output:
- Current Pass (Digest Prefix): The doDeduplicate command groups BLOBS by the first character of their digest (name).
- Mailboxes Checked: The number of mailboxes examined during the current pass.
- Blobs Deduplicated/Duplicated: The number of BLOBS deduplicated by the current operation divided by the total number of duplicated items on the volume.
- Already Deduplicated Blobs: The number of deduplicated blobs on the volume (duplicated blobs that have previously been deduplicated).
- Skipped Blobs: BLOBs that were not analysed, typically due to a read error or a missing file.
- Invalid Digests: BLOBs having a faulty digest (name different from the file’s real digest).
- Total Space Saved: The amount of disc space saved as a result of the doDeduplicate operation.
- We can observe from the example output above that:
- The procedure is now performing the second-to-last pass on the final mailbox.
- 137089 duplicated BLOBs were discovered, 71178 of which had already been deduplicated.
- The current operation deduplicated 64868 BLOBs, saving a total of 21.88GB of disc space.
- High Volume Operations
- Zextras Powerstore: There’s More to It Than Meets the Eye
- At first glance, Zextras Powerstore appears to be only focused to HSM. However, it also includes several really helpful volume-related tools that are unrelated to HSM.
- Because of the inherent dangers in volume control, these tools are only available via the CLI.
At a Glance Volume Operations
Volume operations include the following:
doCheckBlobs: Check the coherency of BLOBs on one or more volumes.
startItemDeduplicate: Begin Item Deduplication on a volume.
moveAllItemsFromOneVolumeToAnother: Move all items from one volume to another.
getVolumeStats: Returns information about the size of a volume and the number of items/blobs it contains.
The options show_volume_size and show_blob_num will add the following information to the output:
Transferring Mailboxes Between Mailstores
You may use the doMailboxMove command to move a single mailbox or all accounts from a particular domain from one mailbox server to another inside the same Zimbra infrastructure.
When the command is performed, it will do the following tasks:
- When migrating a domain, each account on the current server is listed and migrated successively.
- Only at the ‘account’ stage is the mailbox put to maintenance mode.
- If there are 5% or more write errors on the objects being transferred, the move will be halted.
- When moving numerous mailboxes in the same operation, the error count is global rather than per-mailbox.
- If the destination server does not have adequate space to host the mailbox, the move will be aborted.
- When many mailboxes are moved in a single operation, the space check is conducted before transferring each mailbox.
- Except for the mailbox ID, all data is transported at a low level and will not be modified.
- The process is divided into three stages: blobs backup account. For each mailbox, write:
- Blobs: The source server copies all blobs to the destination server.
- backup: From the source server to the destination server, all backup items are transferred.
- account: All database entries are migrated exactly as they are, and LDAP entries are modified, essentially relocating the mailbox.
- data can be used as a shorthand for blobs, accounts, and so on.
- Because all phases must be completed consecutively, executing blobs after backup or account is not feasible. For example, blobs, account (but not vice versa!) is an acceptable sequence. Blobs will give an error when using the order account.
- If the reindex stage is completed, a new HSM operation is sent to the destination server unless otherwise specified.
- All compression options for volumes are used.
- If and only if no other actions are occurring on the source server, the MailboxMove operation can be done.
- If the destination server does not have adequate capacity or if the user only belongs to the destination host, the move will be aborted.
- Items are put on the destination server’s Current Primary volume by default.
- After a mailbox is successfully migrated, the hsm true option can be used to apply the destination server’s HSM rules.
- If the migration is interrupted for any reason, the original account will remain operational and the proper notification will be sent.
- If the mailboxd crashes during the transfer, the Operation Interrupted notice is sent, as it is for other operations, informing users of the interrupted operation.
- Because index information is migrated during the account stage, no manual reindexing is required, nor will one be triggered automatically.
- When transferring accounts from the source to the destination server, HSM is performed only on the transferred accounts immediately after they have been successfully relocated.
- The administrator can, however, opt to postpone the HSM.
- If the second stage fails for whatever reason, HSM is not run
Powerstore CLI by Zextras
The index of all zxsuite powerstore instructions may be found in this section. The whole reference may be found in the section ZxPowerstore CLI Commands.
Indexing content-extraction-tool add Indexing content-extraction-tool list testS3Connection Remove the indexing content extraction tool doCheckBlobs doCreateVolume Alibaba doCreateVolume Centralised doCreateVolume Ceph doCreateVolume Cloudian doCreateVolume CustomS3 EMC doCreateVolume FileBlob doCreateVolume OpenIO doCreateVolume S3 doCreateVolume doCreateVolume ScalityS3 Swift doReplicate doDrivePreviews are deleted.DeleteVolume doMailboxMove doMailboxMoveBlobs doPurgeMailboxesdoRemoveOrphanedBlobs doRestartService doStartService doStopAllOperations removeHsmPolicy doRemoveOrphanedBlobs doRestartService doStartService doStopAllO doStopOperation doStopService doUpdateVolume Alibaba doUpdateVolume Ceph doUpdateVolume Cloudian doUpdateVolume Ceph doUpdateVolume Ceph doUpdateVolume Ceph doUpdateVolume Cep S3 doUpdateVolume ScalityS3 doUpdateVolume Swift doVolumeToVolume CustomS3 doUpdateVolume EMC doUpdateVolume FileBlob doUpdateVolume OpenIO doUpdateVolume S3 doUpdateVolumeMove getAllOperations getAllVolumes getHsmPolicy getMovedMailboxes getNonLocalMailboxes obtainProperty obtainServices monitor getVolumeStats runBulkSetHSMPolicy should be removed. setHsmPolicy +setProperty