Streaming of Events

Have a Question?

This Role is necessary to enable the Carbonio Active Replica functionality, which is the backbone of Carbonio’s High Availability. It is built on Apache Kafka and ZooKeeper, both of which must be installed on the same Node. It is strongly advised to install both services on a dedicated Node for optimal performance; otherwise, you may directly use a Kafka + ZooKeeper SaaS solution.

Multiple Event Streaming Roles can be installed in a Carbonio architecture; however, you must ensure that the ZooKeeper configuration is duplicated on each Event Streaming Node.


Apache ZooKeeper deployment

Instructions are now available for Ubuntu; installation for RHEL should be nearly same, but official instructions will be available soon.

Preliminary Tasks and Requirements

Because all installation processes are performed via CLI, you must be familiar with it.

We also assume you are using ZooKeeper version 3.8.3. Other 3.8.x versions are available; verify their availability on the Apache ZooKeeper official page, and remember to update the version in the instructions below.

  • In addition, to get ZooKeeper up and running, you’ll need:
  • A dedicated user, carbonio-queue, shared a ZooKeeper data directory with Kafka: /var/lib/queue/zookeeper/.
Configuration
Update the data by copying the example configuration into /opt/zookeeper/conf/zoo.cfg.Dir:
Create a ZooKeeper ID.
Each Node in a Carbonio architecture must have a unique ZooKeeper ID in order to function properly.

Add a value to the file for the Node (we use 10) on which ZooKeper is installed.
If you install a second or even third Event Streaming Role, you must append one item for each zookeeper node to the end of the configuration file /opt/zookeeper/conf/zoo.cfg using the format server.[ZOOKEEPER_ID]=[NODE_IP]:2888:3888

Assume you wish to set up a three-node Event Streaming system. You’ve already allocated ZOOKEEPER_ID=1O to node IP 10.0.10.11, and you’ve added

  • ZOOKEEPER_ID=2O to 10.0.10.12 node IP
  • ZOOKEEPER_ID=3O to 10.0.10.13 node IP
You must ensure that three entries are added to the file /opt/zookeeper/conf/zoo.cfg on each machine.
Make a System Unit file.
Copy this snippet into the file /lib/systemd/system/carbonio-zookeeper.service to establish a simple zookeeper setup.
Apache Kafka Installation Requirements and Preliminary Tasks
We also assume you are running Kafka 3.1.2; however, you can use other 3.1.x versions: Check the Apache Kafka official website for availability, and remember to change the version in the instructions below.
  • In addition, to get ZooKeeper up and running, you’ll need:
  • a dedicated user, carbonio-queue, which ZooKeeper also uses
Data directory for Kafka: /var/lib/queue/kafka/logs/
Configuration
In files /opt/kafka/config/server.properties and /opt/kafka/config/producer.properties, the default Kafka configuration must be adjusted to reflect the right settings for broker_id, which must be unique within the infrastructure, log.dirs, and topic.
Make a System Unit File
Copy this snippet into the file /lib/systemd/system/carbonio-kafka.service to establish a simple zookeeper setup.