Apache Kafka Qiuckstart

Introduction

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system. Kafka maintains feeds of messages in topics. Producers write data to topics and consumers read from the topics. One of the use case of Kafka is messaging system. Kafka offers queuing and publish-subscribe model.

Kafka command line tool

• Download Apache Kafka distribution form here 

   

• Extract and goto Kafka home  

    

• Start the Zookeeper   

         bin/zookeeper-server-start.sh config/zookeeper.properties       

• Start the Kafka server   

          bin/kafka-server-start.sh config/server.properties

• Create a topic,

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

• Run a producer and type the message.

bin/kafka-console-producer.sh --broker-list localhost:9092 

-- topic test 

         This is a message
         This is another message 

• Run the consumer,the messages appear in the consumer

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test  --from-beginning

This is a message

This is another message

Kafka Multi Broker Cluster Setup

We will create 3 Kafka brokers (broker0, broker1 and broker2) whose configurations are based on the default.

First we make a configuration file for each of the brokers:

The default broker0 server properties file is.

config/server-1.properties:
    broker.id=0
    port=9092
    log.dir=/tmp/kafka-logs-0

copy the server.properties file for the broker1 and broker2.

> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties

Edit these new files and set the following properties:

config/server-1.properties:
    broker.id=1
    port=9093
    log.dir=/tmp/kafka-logs-1

config/server-2.properties:
    broker.id=2
    port=9094
    log.dir=/tmp/kafka-logs-2

Now we have created 3 Kafka  broker cluster. Start the Kafka server with the  appropriate server properties file.

Broker0

> bin/kafka-server-start.sh config/server.properties

Broker1

> bin/kafka-server-start.sh config/server1.properties

Broker2

> bin/kafka-server-start.sh config/server2.properties