A Tool to manage Apache Kafka -Yahoo Kafka Manager

A Tool to manage Apache Kafka

It supports(as given in its github page) :

  • Managing Multiple Clusters
  • Easy Inspection of Cluster States (Topics, Consumers, Offsets, Brokers, Replica Distribution, Partition Distribution)
  • Replica Election
  • Generate Partition assignments with option to select Brokers to use
  • Re Assignment of partition (based on generated assignments)
  • Topic can be created with optional configs (0.8.1.1 has different configs than 0.8.2+)
  • Topic can be deleted (only supported on 0.8.2+ ,set delete.topic.enable=true in broker config)
  • List Topic indicates Topics marked for deletion (only supported on 0.8.2+)
  • Partition Assignments for Multiple topics with an option to select Brokers
  • Re Assignment of Partition for multiple topics
  • Add partitions to existing Topic
  • Update Config for existing Topic
  • Optionally enable JMX polling for Broker level and Topic level metrics.
  • Optionally filter out Consumers that do not have ids/ owners/ & offsets/ directories in Zookeeper.

1. Prerequisites of Yahoo Kafka Manager

Apache Zookeeper
Apache Kafka
Java 8 as root user
SBT 13 +

1.1 Environment to set

export JAVA_HOME=/home/datadotz/jdk1.8.0_45

2. Apache Zookeeper – Single Node Installation (Optional)

2.1 Download Apache Zookeeper from Apache Zookeeper site. Please Check for Apache Zookeeper Website if the link is broken. Please check for the current version in the Apache Zookeeper
http://www.us.apache.org/dist/zookeeper/zookeeper-3.3.6/

a. Untar zookeeper-3.3.6.tar.gz
b. Change the directory to conf
c. Create a new file zoo.cfg

And add the below content

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/home/dd/zookeeper/data/
# the port at which the clients will connect
clientPort=2181
dataLogDir=/home/dd/zookeeper/logs/

Start Zookeeper bin/zkServer.sh start

3. Apache Kafka – Installation

Kindly refer chennaihug.org  for Kafka installation
http://chennaihug.org/knowledgebase/kafka-single-node-cluster/

Start Kafka

4. Installation of  Yahoo Kafka Manager

Please follow the commands as given below
wget https://github.com/yahoo/kafka-manager/archive/master.zip
unzip master.zip
mv kafka-manager-master/ kafka-manager
cd kafka-manager
sbt clean dist
cd  target/universal/
unzip  kafka-manager-1.1.zip
cd kafka-manager-1.1

Start the Service using below command
sudo bin/kafka-manager -Dkafka-manager.zkhosts=”localhost:2181″

Web UI

Once started , check for the web UI in the location “http://localhost:9000

Create a New Cluster
1

Cluster added successfully
2

List of Brokers
3

Click Broker ID and check for the details
4

Cluster Information
5

List of Clusters
6

Create a New Topic  as ‘Test’
7

List of Topics
9

Information about Topic – Test 
10

Conclusion:
It would be useful in Enterprise Level, where system Engineers don’t have to always go through the CLI to complete their task, where Engineers complete 5,000 tasks a day and its just one of them. It enables you to digest the Information pretty quickly.

————————-

Article written by DataDotz Team

DataDotz is a Chennai based BigData Team primarily focussed on consulting and training on technologies such as Apache Hadoop, Apache Spark , NoSQL(HBase, Cassandra, MongoDB), Search and Cloud Computing.

Note: DataDotz also provides classroom based Apache Kafka training in Chennai. The Course includes Cassandra , MongoDB, Scala and Apache Spark Training. For more details related to Apache Spark training in Chennai, please visit http://datadotz.com/training/