Flume Hello World tutotiral

I am using flume for some time now and really like it. This is simple HelloWorld tutorial that i thought would be helpful if you want to get started with Flume. This tutorial will walk you through steps for setting up Flume that listens to messages on port 44444, once it gets message it just prints it out on console, Follow these steps

1. First create sampleflume.properties file on your machine like this

   
   # example.conf: A single-node Flume configuration
   
   # Name the components on this agent
   agent1.sources = netcat1
   agent1.sinks = logger1
   agent1.channels = memory1
   
   # Describe/configure the source
   agent1.sources.netcat1.type = netcat
   agent1.sources.netcat1.bind = localhost
   agent1.sources.netcat1.port = 44444
   
   # Describe the sink
   agent1.sinks.logger1.type = logger
   
   
   # Use a channel which buffers events in memory
   agent1.channels.memory1.type = memory
   agent1.channels.memory1.capacity = 1000
   agent1.channels.memory1.transactionCapacity = 100
   
   # Bind the source and sink to the channel
   agent1.sources.netcat1.channels = memory1
   agent1.sinks.logger1.channel = memory1
   

   Your flume configuration file must have at least 3 elements a source, channel and sink
   • netcat1: netcat1 source defines how flume is listening to messages. In this case type of netcat means it will listen on port that you can connect to using either netcat or telnet
   • memory: memory channel defines how flume stores messages that it has received before they are consumed by sink. In this case i am saying keep the messages in memory
   • logger1: Logger sink is for testing, it just prints the messages on console
2. Once your configuration file is ready you can start a flume agent by executing following command

   
   flume-ng agent --conf conf --conf-file sampleflume.properties  --name agent1 -Dflume.root.logger=DEBUG,console
   

   YOu will see flume printing messages on the console while it is starting like this
3. Once server is started you can connect to it using nc or telnet and send messages to it like this. Whatever messages you send will be printed to console
4. Once you send messages using nc command look at the server console and you should see the messages that you sent