Introduction to Zextras Powerstore

By adminUncategorized

One major volume and an unknown number of subordinate volumes make up each Zextras Suite installation. The Powerstore module’s job is to transport objects between the secondary volumes and manage them.

Using Hierarchical Storage Management (Hierarchical Storage Management), a policy-based method, objects may be relocated in accordance with: One of the most practical is, for instance, to set aside the fastest storage for intense I/O operations and for often accessed data, while managing older data with the slower storage.

The remainder of this part discusses policies, HSM, and different advanced procedures as well as volumes and how they are managed.

The Fundamentals of Zimbra Stores: Store Types and Uses
Zimbra supports two distinct storage types:
 
Catalogue Store
a repository that houses your data’s metadata and is utilised by Apache Lucene to do indexing and search.
 
Data Centre
a database called MySql that houses all of your Zimbra data in one place.
 
However, only one Index Store, one Primary Data Store, and one Secondary Data Store may be designated as Current (i.e., presently being utilised by Zimbra), even if you can have numerous stores of each kind.
Secondary and Primary Data Stores
Zimbra data stores come in two flavours: primary data stores and secondary data stores.
 
A stated protocol is followed while moving data between the current Primary Data Store and the current Secondary Data Store.
 
Volumes
Zimbra specifies the following three categories of volumes:
 
Principal Current
a storage device where new data is immediately written.
 
Primary Current
a volume to which data are written once an HSM policy has been applied.
 
Not Current Volumes are those that are not marked as Current and where data may only be written via particular manual procedures.
 
Items are usually put in the destination server’s Primary Current volume by default.
Instead of using “independent” volumes, which are isolated and whose directory structure is only related to the server and volume itself, the Centralised Storage feature enables the use of an S3 bucket to host data from multiple servers simultaneously sharing the same directory structure.
 
This dramatically increases mailbox move speed and enables better data management in big multistore systems.
 
Before using centralised storage, the following two crucial factors need to be considered.
  1. Deduplication of the item fails.
  2. The only storage type that may be utilised centrally is S3 buckets.
Configuring Centralised Storage
Several procedures must be followed in order to set up a bucket for centralised storage: construct the volumes on each mailstore, test the connection to the bucket, construct the bucket, and set the volume to current.
 
The suggested process is as follows in specifics and calls for the usage of CLI commands.
 
Using the ZxCore command doCreateBucket, create an S3 bucket:
We employ the following values in this example:
  • S3 as the bucket type 
  • The bucket name, BucketName, must match the name on the remote provider exactly for the command to succeed.
  • The remote username is X58Y54E5687R543.
  • The remote password is abCderT577eDfjhf. 
  • Choose https://example_bucket_provider.com as the connection’s URL. Instead of using the URL, you may alternatively provide the provider’s IP address.
For additional information and choices, see the whole documentation for doCreateBucket S3.
  • When the command is successful, a string, for instance 28m6u4KBwSUnYaPp86XG, is produced that contains the distinct bucket ID. Make a note of it because the rest of the operation will require it.
  • Utilising the bucket ID obtained in the previous stage (28m6u4KBwSUnYaPp86XG), test the connection:
If the command is effective, go on to the following action.
 
Connect the bucket to the first mailstore’s volumes.
powerstore zxsuite doCreateVolume S3 [param VALUE[,VALUE]] _Name of the Zimbra store_ _primary|secondary
 
For instance:
These values are utilised in this illustration:
  • The type of bucket, S3
  • VolumeName: The volume name as it is listed on the server where the command is run.
  • secondary: the volume’s kind
  • The bucket ID* acquired in step 1 is 28m6u4KBwSUnYaPp86XG 
  • volume_prefix a volume’s allocated ID, such as “myVolPrefix,” is utilised for speedy searches.
  • The volume is created with centralised true.
  • For further information and choices, see the whole doCreateVolume S3 documentation.
Turn on the current volume so that it may accept data right away:
These values are utilised in this illustration:
  • The type of bucket, S3
  • VolumeName: The volume name as it is listed on the server where the command is run.
  • secondary: the volume’s kind
For further information and choices, see the whole doUpdateVolume S3 documentation.
  • After the Centralised Volume has been built, it must be added to the volume list and its configuration copied from the first server to all mailbox servers. Run the following commands on every other mailbox server to do this:
The command reported in the preceding step is the second one that must be executed:
Storage Structure – Centralised Storage Structure A centralised volume simply stores data by having a directory for each mailbox that is saved in it at the same level and a single empty directory for each server that is connected to the volume.
 
The servers 595a4409-6aa1-413f-9f45-3ef0f1e560f5 and 3aa2d376-1c59-4b5a-94f6-101602fa69c6 are both linked to the same central volume, which houses three mails, in the example below. As you can see, the storage is unrelated to the actual server where the mails are housed.
Volume management Both primary and secondary volumes may be created on supported third-party storage systems as well as on local storage.
Volumes for Zimbra
Zimbra Blobs are stored on a volume, which is a unique object (path) on a filesystem with all of the related features.
 
Volume Characteristics
The following attributes characterise all Zimbra volumes:
 
  • Name: The volume’s special identification.
  • The data’s intended saving location is designated by a route.
  • Compression: Select whether the volume should enable or disable file compression.
  • Compression Threshold: The smallest file size at which compression begins. Even if compression is enabled, files smaller than this size will never be compressed.
  • Current: A current volume is one that will have HSM policy application (secondary current) or data written to it immediately upon arrival (primary current).
Regional Volumes
No matter where a mountpoint is located on the system, Local Volumes (i.e., FileBlob types) can be hosted on any mountpoint and are specified by the following properties:
  • Name: The volume’s special identification.
  • The data’s intended saving location is designated by a route. This route requires r/w access for the zimbra user
  • File compression can be enabled or disabled for the volume.
  • Compression Threshold: the smallest file size at which compression begins.
Actual Volumes
A current volume is one that will have data written to it immediately (primary current) or through the secondary current of an HSM policy application. Except for specialised manual actions like the Volume-to-Volume transfer, volumes that are not set as Current will not be written onto.
The Hierarchical Storage Management Technique is known as hierarchical storage management.
Data is transferred between stores using the HSM data storage technology in accordance with a predetermined policy.
 
The HSM approach is most frequently used to transfer older data from a faster but more costly storage device to a slower but less expensive one based on the following assumptions:
  • More is spent on quick storage.
  • Slow storage has lower expenses.
  • In comparison to new data, old data will be accessed significantly less frequently.
The HSM approach has two distinct benefits: it lowers total storage costs because just a tiny portion of your data has to be on expensive storage, and it enhances user experience in general.
Volumes, Stores, and Policies
It’s important to grasp the following terms in order to use HSM:v
  • Primary storage: Your data is originally deposited in this quick-but-expensive storage.
  • Secondary Store: The expensive yet sluggish storage location where older data will be sent.
Changing Products Between Stores
The ability to implement predefined HSM rules is the Zextras Powerstore module’s key feature.
There are three ways to start the move:
  • In the Administration Zimlet, choose the Apply Policy button.
  • Utilise the CLI to begin the doMoveBlobs procedure.
  • Activate the Administration Zimlet’s Policy Application Scheduling option, then wait for it to launch automatically.
Following the move’s launch, the following actions are taken:
Zextras Powerstore searches the Primary Store to determine which products adhere to the established policy.
  • The Secondary Store receives a copy of every Blob of the products discovered in the first phase.
  • The duplicated objects’ database records are changed to reflect the relocation.
  • The old Blobs are removed from the Primary Store if (and only if) the second and third phases are successfully performed.
  • Since each step of the Move operation is stateful and is only carried out if the one before it has been properly completed, there is zero chance that any data will be lost.
DoMoveBlobs Zextras Powerstore’s DoMoveBlobs Operation
The doMoveBlobs serve as Zextras Powerstore’s beating heart.
 
According to the appropriate HSM policy, it transfers objects between the Current Primary Store and the Current Secondary Store.
 
An algorithm for transactions executes the move. If there is a problem with one of the phases of the procedure, a rollback occurs, and the data is not changed.
 
The following actions are taken when Zextras Powerstore has determined which objects need to be moved:
  • The Blob is copied to the current secondary store.
  • To inform Zimbra of the item’s new location, the Zimbra Database is updated.
  • The primary store’s current copy of the original Blob is removed.
Everything that complies with the designated HSM policy gets migrated.
Policy Order: A policy’s requirements are fulfilled in the precise order that they are listed. Zextras Powerstore will execute a loop on all items in the current primary store, putting each condition to use before moving on to the next.
 
This indicates that the following rules will produce the same outcome if they are used every day on an example server that sends and receives a total of 1000 emails each day, 100 of which have one or more attachments. However, depending on the quantity and size of the emails stored on the server, the second policy’s execution time would presumably be significantly longer.
As a result, there are fewer items for the second condition to cycle on in the first policy’s first condition (message,document:before:-20day), which loops on all things and moves many of them to the Current Secondary Store.
 
The second condition will have more items to loop on if the first condition is message:before:-10day has:attachment.
 
This is only an illustration and does not apply in all circumstances, but it illustrates the necessity for meticulous planning in your HSM strategy.
 
Applying the HSM Policy, also known as carrying out the doMoveBlobs Operation
Running the doMoveBlobs function in order to move objects between the Primary and Secondary stores in accordance with the specified policy is referred to as applying a policy.
 
Three alternatives are available to you through Zextras Powerstore:
  • through the administrative Zimlet
  • via CLI
  • By using Scheduling doMoveBlobs Stats and Info, you may learn more about how much disc space is saved, how well operations execute, and other things by selecting the Stats button next to the Secondary Volumes list on the Zextras Powerstore tab of the Administration Zimlet.
What is a policy in terms of policy management?
When the doMoveBlobs function of Zextras Powerstore is performed, either manually or by scheduling, a set of rules known as an HSM policy determines which objects will be transferred from the Primary Store to the Secondary Store.
 
Single rules that apply to all item kinds make up a simple policy, whereas many rules that apply to one or more item categories make up a composite policy. Additionally, a further sub-rule may be created by utilising the search syntax in Zimbra.
Policy Case Studies
Here are some instances of policies. See the information below to learn how to establish the policies in the Zextras Powerstore module.
  • Move everything that is older than 30 days.
  • Move emails that are more than 15 days old and all other items that are more than 30 days old.
  • Place all emails in the “Archive” folder, calendar entries older than 15 days, and drive objects older than 20 days.
Establishing a Policy
Both the Administration Zimlet’s Zextras Powerstore tab and the CLI allow for the definition of policies. In both situations, a Zimbra Search can be specified.
Powerstore Zextras and S3 buckets
Zextras Powerstore’s primary and secondary volumes may be housed on S3 buckets, thereby shifting the majority of your data to reliable and secure cloud storage.
Services compatible with S3
Zextras Powerstore should function with any storage solution compatible with the Amazon S3 API right out of the box, however the following are the only ones that are officially supported:
  • (Standard Local Volume) FileBlob
  • S3 Amazon
  • EMC
  • OpenIO
  • Swift
  • S3 Scality
  • Cloudian
  • Any unsupported S3-compliant system that is customised utilising V2 and V4 authentication
The “Incoming” directory and primary volumes
A local “Incoming” directory must exist on a mailbox server before a remote Primary Store may be created there. The default location is /opt/zimbra/incoming; the following commands allow you to view or change the current setting:
Regional Cache
The zimbra user must have read and write access to the local directory that will be used for item caching when storing a volume on third-party remote storage systems.
 
In the Zextras Powerstore part of the Administration Zimlet in the Zimbra Administration Console, the local directory must be manually established and its path must be entered.
 
On an S3-compatible device or service, you won’t be able to create any secondary volumes if the Local Cache directory is not configured.
Setup of a bucket for your volumes is simple because Zextras Powerstore requires no special S3 configuration or settings. Although setting up a separate user bucket and access policy is not essential, it is highly advised because it makes management much simpler.
  • To begin storing your secondary discs on S3, you only need to:
  • an S3 container. To utilise the bucket, you must be aware of its name and location.
  • Access Key and Secret of a user.
a rule that gives the user complete access to your bucket.
In the Zimbra Administration Console, a centralised Bucket Management UI is accessible. Instead of inputting the information each time, this makes it easier to save bucket information to be reused when building a new volume on an S3-compatible storage.
 
The Bucket Management UI is accessed by
  • Use the Zimbra Administration Console to get there
  • Choose “Configure” from the left menu by clicking it.
  • Select “Global Settings” from the menu.
  • S3 Buckets should be chosen.
When creating a new volume of the following types: Amazon S3, Ceph, Cloudian, EMC, Scality S3, Custom S3, Yandex, Alibaba, any bucket added to the system will be accessible.
 
The zxsuite-core-doCreateBucket commands may also be used to create new buckets using the CLI: Amazon Ceph Cloudian CustomS3 EMC S3 ScalityS3 Yandex
Paths and naming for buckets
In order to make your bucket’s contents more understandable even in multi-server situations with several secondary volumes, files are stored in buckets using a well-defined path that may be changed at will:
  • There can be as many volumes under the same destination route as you like because the Bucket Name and Destination route are not linked to the volume itself.
  • On the other hand, the Volume Prefix is unique to each volume and serves as a rapid means of differentiating and identifying various volumes inside the bucket.
Volume Creation Using Zextras Powerstore
From the Zimbra Administration Console, follow these steps to establish a new volume using Zextras Powerstore:
  • In the Zimbra Administration Console, access the HSM Section of the Zextras Administration Zimlet.
  • Either in the Primary Volumes or Secondary Volumes list, click Add.
  • From the storage options available, pick the Volume Type.
  • Type in the necessary volume information.
Utilising Zextras Powerstore to Edit Volumes
From the Zimbra Administration Console, use Zextras Powerstore to modify a volume as follows:
  • In the Zimbra Administration Console, access the HSM Section of the Zextras Administration Zimlet.
  • Decide on a volume
  • Select Edit.
Click Save when finished.
 
Volumes may be deleted using Zextras Powerstore.
In the Zimbra Administration Console, remove the volume that uses Zextras Powerstore as follows:
 
In the Zimbra Administration Console, access the HSM Section of the Zextras Administration Zimlet.
 
Decide on a volume
 
Select Delete.
Amazon S3 Bucket of Tips
Although there are no strict bucket requirements for storing your backup Zimbra volumes on Amazon S3, we do advise that you do so and turn off Static Website Hosting to make maintenance simpler.
 
User
A Programmatic Access user is required in order to receive an Access Key and the associated Secret. For simpler management, we advise you to create a dedicated user in Amazon’s IAM Service.
 
Rights Administration
You may define access policies for your users in Amazon’s IAM. A set of suitable privileges must be granted to the user of your Access Key and Secret for both the bucket itself and its contents. We advise assigning complete permissions like in the example below for simpler administration.
Change the Action section to: if you simply want to grant the bare minimum of permissions.
 
According to Amazon’s standard naming convention, the bucket’s ARN is written as arn:partition:service:region:account-id:resource. Please refer to Amazon’s documentation for further details on this subject.
 
Paths and Naming for Buckets
In order to make your bucket’s contents more understandable (even in multi-server situations with several secondary volumes), files are stored in buckets using a well-defined path that may be changed at will:
There can be as many volumes under the same destination route as you like because the Bucket Name and Destination route are not linked to the volume itself.
 
On the other hand, the Volume Prefix is unique to each volume and serves as a rapid means of differentiating and identifying various volumes inside the bucket.
 
Infrequent Access Storage Class As long as the option has been enabled on the volume, Zextras Powerstore is compatible with the Amazon S3 Standard – Infrequent access storage class and will set any file bigger than the Infrequent Access Threshold setting to this storage class.
As long as the option has been activated on the volume, Zextras Powerstore is compatible with the Amazon S3 – Intelligent Tiering storage class and will set the necessary Intelligent Tiering flag on all files.
The definition of item deduplication
By keeping only one duplicate of an item and referring it numerous times rather than storing several copies of the same item and accessing each copy only once, you can conserve disc space using the item deduplication approach.
 
This could appear to be a modest advancement. However, it actually makes a big impact in real life.
 
When a new item is stored in the Current Primary Volume by Zimbra, item deduplication is carried out by Zimbra.
The message ID of a newly produced item is compared to a list of previously cached objects. Instead of creating a brand-new BLOB for the message if there is a match, a hard link to the BLOB of the cached message is generated.
 
The following configuration attributes in Zimbra are used to control the dedupe cache.
Zextras Powerstore and Item Deduplication
The doDeduplicate operation of the Zextras Powerstore parses a target volume to identify and deduplicate any duplicated items.

You will save even more disc space by doing this because, unlike Zimbra’s automated deduplication, which is restricted to a small cache, Zextras Powerstore’s deduplication will locate and handle duplicate copies of an email independent of cache or timing.
 
In order to optimise your storage consumption, running the doDeduplicate procedure following a migration or a significant data import is also strongly advised.
Volume Deduplication being used
The getAllVolumes command may be used to display a list of all accessible volumes.
 
duplicate statistics
By using the zxsuite powerstore monitor [operationID] command, you may view the statistics of the doDeduplicate operation while it executes as it is a valid target for the monitor command. Typical Output is:
  • Current Pass (Prefix for Digest): Based on the first character of their digests (name), the BLOBS will be analysed by the doDeduplicate command in groups.
  • Number of mailboxes that were checked during the current pass 
  • Deduplicated/duplicated Blobs: Total number of duplicated items on the volume / Number of BLOBS deduplicated by the current operation.
  • Number of already deduplicated blobs (duplicated blobs that have already undergone a prior run) on the volume.
  • Blobs that have not been examined, typically as a result of a read error or a missing file, are known as skipped blobs.
  • Invalid Digests: BLOBs having incorrect digests (names that differ from the file’s real digest).
  • Amount of disc space saved overall with the doDeduplicate process.
  • The example output shown above reveals the following 
  • On the last mailbox, the procedure is now on the second-to-last pass.
  • 137089 duplicated BLOBs have been discovered, of which 71178 have already undergone deduplication.
  • 64868 BLOBs were deduplicated in the current operation, saving a total of 21.88GB in disc space.
Operations using Advanced Volume
More Than Meets the Eye at Zextras Powerstore
Zextras Powerstore appears to be solely focused on HSM at first glance. However, it also includes a number of really helpful volume-related utilities that are unrelated to HSM.
 
These tools are only accessible through the CLI because to the associated dangers in volume management.
Operations by Volume at a Glance
There are accessible volume procedures such as:
 
perform BLOB coherency tests on one or more volumes with the doCheckBlobs command.
 
doDeduplicate: Start a volume’s item deduplication.
 
Move all things from one volume to another using the doVolumeToVolumeMove command.
getVolumeStats: Show statistics on the size and quantity of things or blobs that a volume contains.
The following information will be included in the output if show_volume_size and show_blob_number are selected:
Moving Postboxes Between Post Offices
You can move a single mailbox or all of the accounts from a particular domain from one mailbox server to another inside the same Zimbra infrastructure by using the doMailboxMove command.
The command’s execution will accomplish a variety of tasks, including:
  1. Each account on the current server is listed and migrated progressively when changing a domain.
  2. When the ‘account’ stage is active, the mailbox is merely put to maintenance mode 
  3. If there are 5% or more write errors on the objects being transferred, the move will be terminated.
  4. When moving many mailboxes concurrently, the error count is global rather than per-mailbox.
  5. When moving many mailboxes concurrently, the error count is global rather than per-mailbox.
  6. If the destination server does not have adequate room to accommodate the mailbox, moves will not begin.
  7. Before relocating each mailbox when several are being moved in a single operation, the space check will be carried out.
  8. Except for the mailbox ID, all data is sent at a low level.
Three phases make up the operation: blobs backup account. Every mailbox:
  • Blobs: Each and every blob is sent from the source server to the destination server.
  • Copying all backup data from the source server to the destination server is a backup.
  • account: The mailbox is effectively relocated by updating the LDAP entries and migrating the database records as-is.
  • Data can be a shorthand for blobs and accounts.
  1. Blobs cannot be run after a backup or account because each stage must be completed in order. Blobs, account, for instance, is an acceptable sequence (but not vice versa!). Blobs will give an error when used with the order account.
  2. If not otherwise provided, a new HSM operation is sent to the destination server upon the conclusion of the reindex step
  3. The choices for volume compression are all used.
  4. If and only if no other actions are happening on the source server, the MailboxMove operation can be carried out.
  5. If the destination server is out of space or if the user only has access to the destination host, a transfer will not begin.
  6. Items are typically put on the destination server’s Current Primary volume by default.
  7. After a mailbox has been successfully migrated, the hsm true option can be used to implement the HSM rules of the destination server.
  8. The original account will remain active and the proper notification will be sent if, for any reason, the move is stopped before it is finished.
  9. The Operation Interrupted notice is sent out as it is for all operations in the event that the mailbox crashes during the transfer, alerting users to the interrupted operation.
  10. No manual reindexing is required, and none will be initiated automatically because index information is migrated during the account step
  11. By default, HSM is only run on the migrated accounts after they have been successfully relocated when migrating accounts from source to destination servers.
  12. However, the admin has the option to postpone the HSM for a later date.
  13. HSM is not carried out if the second step fails for whatever reason.
CLI for Zextras Powerstore
The index of all zxsuite powerstore instructions may be found in this section. ZxPowerstore CLI Commands has a section dedicated to it called Full Reference.
Indexing content-extraction-tool add testS3Connection Indexing content-extraction-tool list  remove indexing content extraction tool  Cloudian doCheckBlobs doCreateVolume Ceph doCreateVolume Alibaba doCreateVolume Centralised doCreateVolume S3 doCreateVolume, CustomS3 doCreateVolume, EMC doCreateVolume, FileBlob doCreateVolume S3 doCreateVolume Scality doDeduplicate in SwiftIn fact, DeleteDrivePreviewsdelete volume, move mailboxes, move blobs, and purge mailboxesStopAllOperations doRemoveOrphanedBlobs doRestartService doStartService doRemoveHsmPolicy  doStopOperation doStopService doUpdateVolume Ceph, Alibaba, Cloudian, and doUpdateVolume Swift doVolumeToVolume CustomS3 doUpdateVolume EMC doUpdateVolume FileBlob doUpdateVolume OpenIO doUpdateVolume S3 doUpdateVolumeMove  getAllOperations  getAllVolumes  getHsmPolicy  getMovedMailboxes  getNonLocalMailboxes  getServices, getProperty  monitor getVolumeStats  runBulkSubstitute setHSMPolicy  setHSMPolicy + setProperty
 

Leave a Reply

Your email address will not be published. Required fields are marked *