Close menu

Volume Management

Volume Management

Volume management
Both primary and secondary volumes may be created on supported third-party storage systems as well as on local storage.
 
Volumes by Carbonio
On a filesystem, a volume is a unique entity (path) with all the corresponding attributes that contains Carbonio Blobs.
 
Volume Characteristics
The following characteristics describe all Carbonio volumes:
 
  • Name: The volume’s special identification number
  • The data’s intended saving location is designated by a route. On this route, the zextras user has to have r/w rights.
  • File compression can be turned on or off for the volume. 
  • Compression Threshold: The smallest file size at which compression begins. Even if compression is enabled, files that are smaller than this size will never be compressed.
  • Current: A current volume is one that will have HSM policy application (secondary current) or data written to it immediately upon arrival (primary current).
Regional Volumes

Regardless of where the mountpoint is located, Local Volumes (i.e., FileBlob type) can be hosted on any mountpoint on the system and are determined by the following properties:

  • Name: The volume’s special identification number
  • The data’s intended saving location is designated by a route. This location requires r/w permissions for the zextras user.
  • File compression can be turned on or off for the volume.
  • Compression Threshold: the smallest file size at which compression begins. Even if compression is enabled, files that are smaller than this size will never be compressed.
Actual Volumes
A current volume is one that will have data written to it immediately (primary current) or through the secondary current of an HSM policy application. Except for specialised manual actions like the Volume-to-Volume transfer, volumes that are not set as Current will not be written onto.
 
Managing Volumes using Carbonio Storages

There are essentially three volume management commands: carbonio powerstore doCreateVolume [storeType] | zextras$ doUpdateVolume [storeType] | doDeleteVolume [name]

Volume deletion simply needs the volume name, however the other two actions require the  storeType parameter, which is always placed first and can take any value corresponding to an S3-Compatible Service. The command’s subsequent parameters are now dependent on the storeType that was chosen.
 
Depending on the [kind] of volume that has to be defined, which is one of the following, the parameters required by these commands may change.
 
  • (Local) FileBlob 
  • Alibaba 
  • Ceph
  • OpenIO
  • Swift
  • Cloudian (object storage that is S3 compliant) 
  • Amazon and any other S3-compatible solution are not expressly supported by S3.
  • Scalability (object storage S3 compatible)
  • EMC (object storage that is S3 compatible)
 
Hierarchical Storage Management
The Hierarchical Storage Management Technique
Data is transferred between stores using the HSM data storage technology in accordance with a predetermined policy.
 
The HSM approach is most frequently used to transfer older data from a faster but more costly storage device to a slower but less expensive one based on the following assumptions:
  • More is spent on quick storage.
  • Slow storage has lower expenses.
  • In comparison to new data, old data will be accessed significantly less frequently.
The HSM approach has two distinct benefits: it lowers total storage costs because just a tiny portion of your data has to be on expensive storage, and it enhances user experience in general.
 
Volumes, Stores, and Policies
It’s important to grasp the following terms in order to use HSM:
 
  • Primary storage: Your data is originally deposited in this quick-but-expensive storage.
  • Secondary Store: The expensive yet sluggish storage location where older data will be sent.
Changing Products Between Stores
The ability to implement predetermined HSM rules is the Carbonio Storages module’s key feature.
 
Start the doMoveBlobs action using the CLI to initiate the move.
 
Following the move’s launch, the following actions are taken:
 
In order to determine whether things adhere to the established policy, Carbonio Storages searches through the Primary Store.
 
  • The Secondary Store receives a copy of every Blob of the products discovered in the first phase.
  • The duplicated objects’ database records are changed to reflect the relocation.
  • The old Blobs are removed from the Primary Store if (and only if) the second and third phases are successfully performed.
  • Since each step of the Move operation is stateful and is only carried out if the one before it has been properly completed, there is zero chance that any data will be lost.
 
DoMoveBlobs
Carbonio Storages’ DoMoveBlobs Operation
The doMoveBlobs serve as Carbonio Storages’ beating heart.
 
According to the appropriate HSM policy, it transfers objects between the Current Primary Store and the Current Secondary Store.
 
An algorithm for transactions executes the move. If there is a problem with one of the phases of the procedure, a rollback occurs, and the data is not changed.
 
The following actions are taken when Carbonio Storages has determined which things need to be moved:
  • The Blob is copied to the current secondary store.
  •  To inform Carbonio of the item’s new location, the Carbonio Database is updated.
  • The primary store’s current copy of the original Blob is removed.
What is Moved?

Everything that complies with the designated HSM policy gets migrated.

Example

The following policy:

message,document:before:-20day
message:before:-10day has:attachment

will move all emails and documents older than 20 days along with all emails older than 10 days that contain an attachment.

Warning

By default, results from the Trash folder do not appear in any search–and this includes the HSM Policy. In order to ensure that all items are moved, add “is:anywhere” to your policy.

Policy Order: A policy’s requirements are fulfilled in the precise order that they are listed. Before beginning the subsequent condition, Carbonio Storages will loop over all of the objects in the Current Primary Store and apply each individual condition.

message,document:before:-20day
message:before:-10day has:attachment

message:before:-10day has:attachment
message,document:before:-20day
This implies that the subsequent laws the same outcome will be obtained if everyday on a sample server that sends/receives a total of 1000 emails each day, 100 of which contain one or more attachments. However, depending on the quantity and size of the emails stored on the server, the second policy’s execution time would presumably be significantly longer.
 
As a result, there are fewer items for the second condition to cycle on in the first policy’s first condition (message,document:before:-20day), which loops on all things and moves many of them to the Current Secondary Store.
 
The second condition will have more items to loop on if the first condition is message:before:-10day has:attachment
This is only an illustration and does not apply in all circumstances, but it illustrates the necessity for meticulous planning in your HSM strategy.
 
Executing the doMoveBlobs Operation
Applying the HSM Policy, also known as carrying out the doMoveBlobs Operation
Running the doMoveBlobs function in order to move objects between the Primary and Secondary stores in accordance with the specified policy is referred to as applying a policy.
 
There are two choices available to you from Carbonio Storages:
  • via CLI
  • By use of scheduling

Warning

Items in Trash or dumpster folders are not moved to the secondary store by the HSM module. Currently, there is no option to define a policy for Trash and dumpster.

Run the following command as the zextras user to apply the HSM Policy through the CLI.

zextras$ carbonio powerstore doMoveBlobs
What is a policy in terms of policy management?
When the doMoveBlobs function of Carbonio Storages is performed, either manually or by scheduling, a set of rules known as an HSM policy specifies which objects will be transferred from the Primary Store to the Secondary Store.
 
Single rules that apply to all item kinds make up a simple policy, whereas many rules that apply to one or more item categories make up a composite policy.
 
Policy Examples
Here are some instances of policies. See the information below to learn how to establish the policies in the Carbonio Storages module.
 
  • Move everything that is older than 30 days.
  • Move emails that are more than 15 days old and all other items that are more than 30 days old.
  • Put all emails in the Archive folder, Carbonio Files items older than 20 days, and calendar items older than 15 days
 
Establishing a Policy
By using one of the two accessible policy management commands from the CLI, policies may be defined.
zextras$ carbonio powerstore setHSMPolicy hsm_policy


zextras$ carbonio powerstore +setHsmPolicy hsm_policy
The difference between both commands is that  setHSMPolicy adds policies to existing ones while +setHSMPolicy  generates new policies and replaces old ones.