Installing Flume

Hi Flume Learners,

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. This Article will provide you some basic information about installing of Flume in your machine. Follow the steps and get Flume installed on your machine and start transferring data.

1. Download Flume

 Use below link to download Flume or download from its official Site.

2. Untar Flume

Untar flume using below command:
tar -zxvf apache-flume-1.4.0-bin.tar.gz

3. Path to set in .bashrc file

export JAVA_HOME=/home/bigdata/jdk1.6.0_45
export PATH=$HOME/bin:$JAVA_HOME/bin:$PATH
export HADOOP_HOME=/home/bigdata/hadoop-1.2.1

(execute .bashrc file after editing it)

4. Flume check
Lets check whether flume has installed correctly or not. Now enter in to flume folder using terminal and give the below command in it and give enter.
$bin/flume-ng agent -n $agent_name -c conf -f conf/

If this command is running perfectly with out errors, then monitor the terminal it will show us that flume is reading the hadoop jars.

5. Write configuration for flume
We need to write configuration for starting flume. We have attached a sample flume configuration, just download it and then use it to check the flume process.

Just click this to download flume configuration file. Click_here_to_download

After downloading, put this file inside the flume folder.
eg : /home/bigdata/apache-flume-1.4.0-bin/netcat_flume.conf_.txt

6. Start Flume
Before starting flume. First open two terminals in your screen. And make those two terminal to be viewed in a parallel way. You should be able to monitor those two terminal at the same time.

In Terminal-1
Enter in to flume directory. And give the following command.
$bin/flume-ng agent –conf-file netcat_flume.conf_.txt –name a1

This will start the flume server with the configuration file that you have inserted in flume folder(ie: The Downloaded configuration file from this material)
Now you can find flume server is getting started. This terminal will be ending with the blinking cursor.

In Terminal-2
In this terminal we need to start telnet with port number 44444. Give the below command in this terminal to start telnet
$telnet localhost 44444

Telnet will get started in this terminal now. It will be ending with the blinking icon.

Test how flume work
Go to Terminal-2 (ie: telnet terminal) type some words in that terminal and give enter
Go to Terminal-1 (ie: Flume running terminal) monitor this terminal, you can find the word or message what you have entered in Terminal-2 will be received in Terminal-1, with some additional information. Now, what ever you type in Terminal-2 will be send to Terminal-1. But there is a limit, according to this configuration Terminal-1 can show you only the first 16 characters that you have entered in Terminal-2.

Flume is working perfectly in your machine. You can change the configuration file to ask your flume to track of some other file or folder etc. You can provide different source and different destination for your flume.


Article written by DataDotz Team

DataDotz is a Chennai based BigData Team primarily focussed on consulting and training on technologies such as Apache Hadoop, Apache Spark , NoSQL(HBase, Cassandra, MongoDB), Search and Cloud Computing.

Note: DataDotz also provides classroom based Apache Kafka training in Chennai. The Course includes Cassandra , MongoDB, Scala and Apache Spark Training. For more details related to Apache Spark training in Chennai, please visit