Sunday, September 20, 2015

Apache Kafka Qiuckstart

Introduction

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system. Kafka maintains feeds of messages in topics. Producers write data to topics and consumers read from the topics. One of the use case of Kafka is messaging system. Kafka offers queuing and publish-subscribe model.

Kafka command line tool

  • Download Apache Kafka distribution form here 
   
  • Extract and goto Kafka home  
    
  • Start the Zookeeper   
         bin/zookeeper-server-start.sh config/zookeeper.properties       

  • Start the Kafka server   
          bin/kafka-server-start.sh config/server.properties

  • Create a topic,
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

  • Run a producer and type the message.
bin/kafka-console-producer.sh --broker-list localhost:9092 
-- topic test 
         This is a message
         This is another message 

  • Run the consumer,the messages appear in the consumer
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test  --from-beginning
This is a message
This is another message


Kafka Multi Broker Cluster Setup

We will create 3 Kafka brokers (broker0, broker1 and broker2) whose configurations are based on the default.

First we make a configuration file for each of the brokers:

The default broker0 server properties file is.

config/server-1.properties:
    broker.id=0
    port=9092
    log.dir=/tmp/kafka-logs-0

copy the server.properties file for the broker1 and broker2.

> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties


Edit these new files and set the following properties:

config/server-1.properties:
    broker.id=1
    port=9093
    log.dir=/tmp/kafka-logs-1

config/server-2.properties:
    broker.id=2
    port=9094
    log.dir=/tmp/kafka-logs-2

Now we have created 3 Kafka  broker cluster. Start the Kafka server with the  appropriate server properties file.

Broker0
> bin/kafka-server-start.sh config/server.properties

Broker1
> bin/kafka-server-start.sh config/server1.properties

Broker2
> bin/kafka-server-start.sh config/server2.properties

No comments:

Post a Comment

Simple JSP Ajax Tutorial

In this blog post, a simple JSP Ajax sample for adding two numbers.  Ajax is a method for returning a result without reloading the HTML pag...