Setup Single Node Single/Multi Broker Configuration (Kafka and Zookeeper) : Kafka Operations Sample Commands

Install/Configure Zookeeper: Install and Verify Zookeeper environment and create data directory to store Zookeeper data.

1. Download zookeeper and unzip it. Rename directory as zookeeper & place it at some location like (/usr/local/zookeeper)
2. Set environment variables. The directory /usr/local/zookeeper will be referred as $ZK_HOME. Open bash profile & add the environment variables by running the below commands
   

   $ vi /etc/profile
   

   Add below two lines in profile page which make $ZK_HOME environment variable referring to /usr/local/zookeeper.
   export ZK_HOME=/usr/local/zookeeper
   export PATH=$ZK_HOME/bin:$PATH
3. Create zookeeper directory(/var/lib/zookeeper) and provide access to <centos> user. Here centos is my privileged user for accessing all services.

   $ sudo mkdir /var/lib/zookeeper
   $ sudo chown -R centos:centos /var/lib/zookeeper
   

4. Create a file $ZK_HOME/conf/zoo.cfg and add following lines in it. Refer sample file  "zoo_sample.cfg"

   tickTime=2000
   dataDir=/var/lib/zookeeper
   clientPort=2181
   initLimit=20
   syncLimit=5
   

5. Verify zookeeper installation. Start Zookeeper by running the command below and verify that zookeeper is started.

   [centos@host01 conf]$ zkServer.sh start
   ZooKeeper JMX enabled by default
   Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
   Starting zookeeper ... STARTED

 For more details related to zookeeper refer Zookeeper documentation.

Install/Configure Kafka : Install and Verify Kafka environment and create data directory to store Kafka data.

1. Download Apache Kafka and unzip it. Rename directory as zookeeper & place it at some location like (/usr/local/kafka)
2. Set environment variables. The directory /usr/local/zookeeper will be referred as $KAFKA_HOME. Open bash profile & add the environment variables by running the below commands
   

   $ vi /etc/profile
   

   Add below two lines in profile page which make $KAFKA_HOME environment variable referring to /usr/local/zookeeper.
   export KAFKA_HOME =/usr/local/kafka
   export PATH=$KAFKA_HOME/bin:$PATH
3. Create kafka directory(/var/lib/kafka) and provide access to <centos> user. Here centos is my privileged user for accessing all services.

   $ sudo mkdir /var/lib/kafka
   $ sudo chown -R centos:centos /var/lib/kafka

Setup Single Broker Kafka (Single Node-Single Broker)

Below steps setup and configure Kafka as a Single Node-Single Broker.

1. Go to $KAFKA_HOME/config directory and Make a copy of server.properties.

   [centos@host01 conf]$ cd $KAFKA_HOME/config
   [centos@host01 config]$ cp server.properties server-1.properties
   

2. Open server-1.properties and make the following changes.
   Modify broker.id to 101, uncomment listeners entry and configure it to bind on localhost:9091 and modify log.dirs to /data/kafka-logs-1.

   broker.id=101
   listeners=PLAINTEXT://localhost:9091
   log.dirs=/var/lib/kafka/kafka-logs-1

3. Go to $KAFKA_HOME and Start Kafka broker by running the commands below. Verify that Kafka server is started.

   [centos@host01 ~]$ cd $KAFKA_HOME
   [centos@host01 kafka]$ bin/kafka-server-start.sh config/server-1.properties
   

4. Run jps command and verify that QuorumPeerMain and Kafka java processes are running.

   [centos@host01 ~]$ jps
   3072 QuorumPeerMain
   4116 Kafka
   4475 Jps

Basic Kafka Operations : (Single Node - Single Broker)

1. Create topic: Execute below command to create a topic named as topic-devinline-1

   [centos@host01 ~]$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic-devinline-1
   Created topic "topic-devinline-1".
   

2. List topics in broker: Execute below command to display all topics and validate that topic-devinline-1 exist.

   [centos@host01 ~]$ kafka-topics.sh --list --zookeeper localhost:2181
   topic-devinline-1
   

3. Describe topic details: Using --describe 

   [centos@host01 ~]$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic-devinline-1
   Topic:topic-devinline-1 PartitionCount:1 ReplicationFactor:1 Configs:
    Topic: topic-devinline-1 Partition: 0 Leader: 101 Replicas:
   

4. Start Producer: Run below command to start Producer and produce messages

   [centos@host01 ~]$ kafka-console-producer.sh --broker-list localhost:9091 --topic topic-devinline-1
   >Hello,Message-1    
   >Hello,Message-2
   >Hello,Message-3
   >
   

5. Start Consumer : Run below command to start consumer to consume messages from beginning produced by producer and Verify all the messages produced are consumed.

   [centos@host01 ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9091 --topic  topic-devinline-1 --from-beginning
   Hello,Message-1
   Hello,Message-2
   Hello,Message-3
   

6. Producer and Consumer in Sync: Produces message in producer terminal, at the same time it is consumed in consumer terminal.

   

   Producer - producing message and Consumer consuming in sync

Setup Multi Broker Kafka (Single Node - Multiple Broker)

Earlier we setup one topic in a broker (Single node). Add two more Kafka brokers to the existing configuration and make it Single Node – Multiple Brokers configuration. Execute following commands to setup Multiple Brokers configuration.

1. Go to kafka/config directory and Make two copies of server.properties.

   [centos@host01 config]$ cp server-1.properties server-2.properties
   [centos@host01 config]$ cp server-1.properties server-3.properties
   

2. Update server-2.properties with following details.

   broker.id=102
   listeners=PLAINTEXT://localhost:9092
   log.dirs=/var/lib/kafka/kafka-logs-2
   

3. Update server-3.properties with following details.

   broker.id=103
   listeners=PLAINTEXT://localhost:9093
   log.dirs=/var/lib/kafka/kafka-logs-3
   

4. Verify all three Kafka broker is running along with zookeeper.Verify that all three brokers are up and running

   [centos@host01 ~]$ jps
   3072 QuorumPeerMain
   8821 Kafka
   9514 Kafka
   8411 Kafka
   9867 Jps
   

Basic Kafka Operations : (Single Node - Multiple Broker)

Now we have three brokers up and running. We can perform Kafka broker operations in multi broker environment.

1. Create Topic : Create a new topic with replication factor 3 as there are 3 brokers.

   [centos@host01 ~]$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 -partitions 1 --topic Multibroker-Devinline
   Created topic "Multibroker-Devinline".
   

2. Describe topic details : Using switch --describe we can display topic details.    

   [centos@host01 ~]$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic Multibroker-Devinline
   Topic:Multibroker-Devinline PartitionCount:1 ReplicationFactor:3 Configs:
    Topic: Multibroker-Devinline Partition: 0 Leader: 102 Replicas: 102,101,103 Isr: 102,101,103
   

3. Start the producer and produce messages

   [centos@host01 ~]$ kafka-console-producer.sh --broker-list localhost:9093 --topic Multibroker-Devinline
   >Hello, Multibroker-1        
   >Hello, Multibroker-2
   >Hello, Multibroker-3
   >Hello, Multibroker-4
   

4. Start the consumer and consume messages

   [centos@host01 ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic Multibroker-Devinline --from-beginning
   Hello, Multibroker-1
   Hello, Multibroker-1
   Hello, Multibroker-2
   Hello, Multibroker-3
   Hello, Multibroker-4
   

5. List the topics in the environment: --list switch is used to display all topics in environment. 

   [centos@host01 ~]$ kafka-topics.sh --list --zookeeper localhost:2181
   Multibroker-Devinline
   __consumer_offsets
   topic-devinline-1
   

6. Modify topic config - Increase partition value from 1 to 2 for topic topic-devinline-1

   [centos@host01 ~]$ kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic-devinline-1 --partitions 2
   WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
   Adding partitions succeeded!
   

7. Delete topic : Using --delete topic can be deleted. By default, topic is not immediately deleted it is marked for deletion.

   [centos@host01 ~]$ kafka-topics.sh --zookeeper localhost:2181 --delete --topic topic-devinline-1
   Topic topic-devinline-1 is marked for deletion.
   Note: This will have no impact if delete.topic.enable is not set to true.