Install/Configure Zookeeper: Install and Verify Zookeeper environment and create data directory to store Zookeeper data.
Install/Configure Kafka : Install and Verify Kafka environment and create data directory to store Kafka data.
- Download zookeeper and unzip it. Rename directory as zookeeper & place it at some location like (/usr/local/zookeeper)
- Set environment variables. The directory /usr/local/zookeeper will be referred as $ZK_HOME. Open bash profile & add the environment variables by running the below commands
$ vi /etc/profile
export ZK_HOME=/usr/local/zookeeper
export PATH=$ZK_HOME/bin:$PATH - Create zookeeper directory(/var/lib/zookeeper) and provide access to <centos> user. Here centos is my privileged user for accessing all services.
$ sudo mkdir /var/lib/zookeeper $ sudo chown -R centos:centos /var/lib/zookeeper
- Create a file $ZK_HOME/conf/zoo.cfg and add following lines in it. Refer sample file "zoo_sample.cfg"
tickTime=2000 dataDir=/var/lib/zookeeper clientPort=2181 initLimit=20 syncLimit=5
- Verify zookeeper installation. Start Zookeeper by running the command below and verify that zookeeper is started.
[centos@host01 conf]$ zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED
Install/Configure Kafka : Install and Verify Kafka environment and create data directory to store Kafka data.
- Download Apache Kafka and unzip it. Rename directory as zookeeper & place it at some location like (/usr/local/kafka)
- Set environment variables. The directory /usr/local/zookeeper will be referred as $KAFKA_HOME. Open bash profile & add the environment variables by running the below commands
$ vi /etc/profile
export KAFKA_HOME =/usr/local/kafka
export PATH=$KAFKA_HOME/bin:$PATH - Create kafka directory(/var/lib/kafka) and provide access to <centos> user. Here centos is my privileged user for accessing all services.
$ sudo mkdir /var/lib/kafka $ sudo chown -R centos:centos /var/lib/kafka
Setup Single Broker Kafka (Single Node-Single Broker)
Below steps setup and configure Kafka as a Single Node-Single Broker.- Go to $KAFKA_HOME/config directory and Make a copy of server.properties.
[centos@host01 conf]$ cd $KAFKA_HOME/config [centos@host01 config]$ cp server.properties server-1.properties
- Open server-1.properties and make the following changes.
Modify broker.id to 101, uncomment listeners entry and configure it to bind on localhost:9091 and modify log.dirs to /data/kafka-logs-1.broker.id=101 listeners=PLAINTEXT://localhost:9091 log.dirs=/var/lib/kafka/kafka-logs-1
- Go to $KAFKA_HOME and Start Kafka broker by running the commands below. Verify that Kafka server is started.
[centos@host01 ~]$ cd $KAFKA_HOME [centos@host01 kafka]$ bin/kafka-server-start.sh config/server-1.properties
- Run jps command and verify that QuorumPeerMain and Kafka java processes are running.
[centos@host01 ~]$ jps 3072 QuorumPeerMain 4116 Kafka 4475 Jps
Basic Kafka Operations : (Single Node - Single Broker)
- Create topic: Execute below command to create a topic named as topic-devinline-1
[centos@host01 ~]$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic-devinline-1 Created topic "topic-devinline-1".
- List topics in broker: Execute below command to display all topics and validate that topic-devinline-1 exist.
[centos@host01 ~]$ kafka-topics.sh --list --zookeeper localhost:2181 topic-devinline-1
- Describe topic details: Using --describe
[centos@host01 ~]$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic-devinline-1 Topic:topic-devinline-1 PartitionCount:1 ReplicationFactor:1 Configs: Topic: topic-devinline-1 Partition: 0 Leader: 101 Replicas:
- Start Producer: Run below command to start Producer and produce messages
[centos@host01 ~]$ kafka-console-producer.sh --broker-list localhost:9091 --topic topic-devinline-1 >Hello,Message-1 >Hello,Message-2 >Hello,Message-3 >
- Start Consumer : Run below command to start consumer to consume messages from beginning produced by producer and Verify all the messages produced are consumed.
[centos@host01 ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9091 --topic topic-devinline-1 --from-beginning Hello,Message-1 Hello,Message-2 Hello,Message-3
- Producer and Consumer in Sync: Produces message in producer terminal, at the same time it is consumed in consumer terminal.
Producer - producing message and Consumer consuming in sync
Setup Multi Broker Kafka (Single Node - Multiple Broker)
Earlier we setup one topic in a broker (Single node). Add two more Kafka brokers to the existing configuration and make it Single Node – Multiple Brokers configuration. Execute following commands to setup Multiple Brokers configuration.- Go to kafka/config directory and Make two copies of server.properties.
[centos@host01 config]$ cp server-1.properties server-2.properties [centos@host01 config]$ cp server-1.properties server-3.properties
- Update server-2.properties with following details.
broker.id=102 listeners=PLAINTEXT://localhost:9092 log.dirs=/var/lib/kafka/kafka-logs-2
- Update server-3.properties with following details.
broker.id=103 listeners=PLAINTEXT://localhost:9093 log.dirs=/var/lib/kafka/kafka-logs-3
- Verify all three Kafka broker is running along with zookeeper.Verify that all three brokers are up and running
[centos@host01 ~]$ jps 3072 QuorumPeerMain 8821 Kafka 9514 Kafka 8411 Kafka 9867 Jps
Basic Kafka Operations : (Single Node - Multiple Broker)
Now we have three brokers up and running. We can perform Kafka broker operations in multi broker environment.- Create Topic : Create a new topic with replication factor 3 as there are 3 brokers.
[centos@host01 ~]$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 -partitions 1 --topic Multibroker-Devinline Created topic "Multibroker-Devinline".
- Describe topic details : Using switch --describe we can display topic details.
[centos@host01 ~]$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic Multibroker-Devinline Topic:Multibroker-Devinline PartitionCount:1 ReplicationFactor:3 Configs: Topic: Multibroker-Devinline Partition: 0 Leader: 102 Replicas: 102,101,103 Isr: 102,101,103
- Start the producer and produce messages
[centos@host01 ~]$ kafka-console-producer.sh --broker-list localhost:9093 --topic Multibroker-Devinline >Hello, Multibroker-1 >Hello, Multibroker-2 >Hello, Multibroker-3 >Hello, Multibroker-4
- Start the consumer and consume messages
[centos@host01 ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic Multibroker-Devinline --from-beginning Hello, Multibroker-1 Hello, Multibroker-1 Hello, Multibroker-2 Hello, Multibroker-3 Hello, Multibroker-4
- List the topics in the environment: --list switch is used to display all topics in environment.
[centos@host01 ~]$ kafka-topics.sh --list --zookeeper localhost:2181 Multibroker-Devinline __consumer_offsets topic-devinline-1
- Modify topic config - Increase partition value from 1 to 2 for topic topic-devinline-1
[centos@host01 ~]$ kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic-devinline-1 --partitions 2 WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected Adding partitions succeeded!
- Delete topic : Using --delete topic can be deleted. By default, topic is not immediately deleted it is marked for deletion.
[centos@host01 ~]$ kafka-topics.sh --zookeeper localhost:2181 --delete --topic topic-devinline-1 Topic topic-devinline-1 is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true.
Amazing information, thank you for your ideas. After a long time, I have studied an interesting article.
ReplyDeleteMobile Testing Training in Chennai
Manual Testing Training in Chennai
testing courses in chennai
Manual Testing Training
Manual Testing Course
Drupal Training in Chennai
Photoshop Classes in Chennai
Great Article android based projects
DeleteJava Training in Chennai
Project Center in Chennai
Java Training in Chennai
projects for cse
The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training
Awesome blog with great piece of information. Very well written blog with crisp and neat content. Keep sharing more such blogs.
ReplyDeleteCloud Computing Training in Chennai
Cloud Training in Chennai
Data Science Course in Chennai
Azure courses in Chennai
VMware course
R Programming Training in Chennai
Cloud Certification in Chennai
The way you have presented the blog is really good.... thanks for sharing with us...
ReplyDeleteAndroid Training in Chennai
android classes in chennai
android development course in chennai
Android Course in Chennai
Android Training in Tambaram
Android training in Guindy
Python Training in Chennai
Big data training in chennai
SEO training in chennai
JAVA Training in Chennai
Thanks for sharing an informative blog keep rocking bring more details.I like the helpful info you provide in your articles. I’ll bookmark your weblog and check again here regularly. I am quite sure I will learn much new stuff right here! Good luck for the next!
ReplyDeleteWeb Designing Training Institute in Chennai | web design training class in chennai | web designing course in chennai with placement
Mobile Application Development Courses in chennai
Data Science Training in Chennai | Data Science courses in Chennai
Professional packers and movers in chennai | PDY Packers | Household Goods Shifting
Web Designing Training Institute in Chennai | Web Designing courses in Chennai
Google ads services | Google Ads Management agency
Web Designing Course in Chennai | Web Designing Training in Chennai
Here is the colleges details to study in Bangalore. You can select the best college details from the below mentioned courses that you love to study. The list of colleges are only in Bangalore, so, if you are looking to study in Bangalore, just click the below mentioned links.
ReplyDeleteBSc Medical Imaging Technology Colleges in Bangalore | Medical Imaging Technology Colleges in Bangalore | BSc Optometry Colleges in Bangalore | Optometry Colleges in Bangalore |BSc Renal Dialysis Colleges in Bangalore | Renal Dialysis Technology Colleges in Bangalore |BSc Respiratory Care Technology Colleges in Bangalore | Respiratory Care Colleges in Bangalore |BSc Cardiac Care Technology Colleges in Bangalore | Cardiac Care Colleges in Bangalore |BSc Perfusion Technology Colleges in Bangalore | Perfusion Technology Colleges in Bangalore |
Taldeen is one of the best plastic manufacturing company in Saudi Arabia. They are manufacturing Handling Solutions Plastic products like Plastic Pallets and plastic crates. Here is the link of the product
ReplyDeleteHandling Solutions
Plastic Pallets
GrueBleen is one of the Branding and Marketing agency Based in Riyadh- Saudi Arabia. The main functions of GrueBleen is Advertising, Branding, Marketing, Office Branding, Exhibition Management and Digital Marketing. Visit the below link to know more about GrueBleen Creative Club.
Branding Agency Riyadh
Marketing Agency Riyadh
Agriculture Solutions – Taldeen is a plastic manufacturing company in Saudi Arabia. They are manufacturing agricultural plastic products like greenhouse cover and hay cover. Visit the below link to know more details
Agriculture Solutions
Greenhouse Cover
GrueBleen – One of the best social media marketing agency in Riyadh- Saudi Arabia. Visit here for the all service details of GrueBleen.
Social Media Marketing Agency | Social Media Agency In Saudi Arabia | Social Media Agency In Riyadh | Social Media Agency in Jeddah |
Pretty blog, so many ideas in a single site, thanks for the informative article, keep updating more article.
ReplyDeleteDigital Marketing Course in Chennai
Digital Marketing Training in Chennai
Digital Marketing Training Institute in Chennai
Digital Marketing Institute in Chennai
Best Digital Marketing Course in Chennai
Pretty blog, so many ideas in a single site, thanks for the informative article, keep updating more article.
ReplyDeleteDigital Marketing Courses in Bangalore
Digital Marketing Training in Bangalore
digital marketing training in marathahalli
Digital Marketing Course in Coimbatore
Digital Marketing Course in Madurai
digital marketing training in btm
Nice article. I liked very much. All the informations given by you are really helpful for my research. keep on posting your views.
ReplyDeleteccna course in bangalore
ccna course in marathahalli
ccna training institutes in btm
ccna course in Coimbatore
ccna course in Madurai
ccna training in madurai
ccna training in coimbatore
Nice article. I liked very much. All the informations given by you are really helpful for my research. keep on posting your views.
ReplyDeleteccna course in Chennai
ccna Training in Chennai
ccna Training institute in Chennai
ccna institute in Chennai
Best CCNA Training Institute in Chennai
Liên hệ Aivivu, đặt vé máy bay tham khảo
ReplyDeletevé máy bay đi Mỹ hạng thương gia
vé máy bay hà nội hồ chà minh vietjet
vé máy bay đà lạt hà nội
vé máy bay sà i gòn đà lạt pacific airlines
vé máy bay vinh quy nhơn
xe taxi sân bay nội bà i
combo hà nội đà lạt 3 ngà y 2 đêm
https://salsabeelahmedandco.com/
ReplyDeleteVery Informative and useful... Keep it up the great work. I really appreciate your post.
https://salsabeelahmedandco.com/company-registration-in-bangalore/
https://salsabeelahmedandco.com/ca-firms-in-bangalore/
https://salsabeelahmedandco.com/accounting-services-in-bangalore/
https://salsabeelahmedandco.com/gst-consultants-in-bangalore/
https://salsabeelahmedandco.com/public-limited-company/
https://salsabeelahmedandco.com/private-limited-company-in-india/
https://salsabeelahmedandco.com/one-person-company-in-india/
https://salsabeelahmedandco.com/partnership-firm-in-bangalore/
https://salsabeelahmedandco.com/limited-liability-partnership-llp/
Thanks for sharing a valuable information.
ReplyDeleteBest Interior Designers in Chennai
Best Interior Decorators in Chennai
chennai renovation
flat renovation contractor services in chennai
modular kitchen in chennai
false ceiling contractor services in chennai
carpenter services in chennai
Thank you for this wonderful post. It is very informative and useful. Ca for Tax Filing in Bangalore
ReplyDelete