Flume Hello World tutotiral

I am using flume for some time now and really like it. This is simple HelloWorld tutorial that i thought would be helpful if you want to get started with Flume. This tutorial will walk you through steps for setting up Flume that listens to messages on port 44444, once it gets message it just prints it out on console, Follow these steps
  1. First create sampleflume.properties file on your machine like this 
    
# example.conf: A single-node Flume configuration

# Name the components on this agent
agent1.sources = netcat1
agent1.sinks = logger1
agent1.channels = memory1

# Describe/configure the source
agent1.sources.netcat1.type = netcat
agent1.sources.netcat1.bind = localhost
agent1.sources.netcat1.port = 44444

# Describe the sink
agent1.sinks.logger1.type = logger


# Use a channel which buffers events in memory
agent1.channels.memory1.type = memory
agent1.channels.memory1.capacity = 1000
agent1.channels.memory1.transactionCapacity = 100

# Bind the source and sink to the channel
agent1.sources.netcat1.channels = memory1
agent1.sinks.logger1.channel = memory1
    Your flume configuration file must have at least 3 elements a source, channel and sink
    • netcat1: netcat1 source defines how flume is listening to messages. In this case type of netcat means it will listen on port that you can connect to using either netcat or telnet
    • memory: memory channel defines how flume stores messages that it has received before they are consumed by sink. In this case i am saying keep the messages in memory
    • logger1: Logger sink is for testing, it just prints the messages on console
  2. Once your configuration file is ready you can start a flume agent by executing following command 
    
flume-ng agent --conf conf --conf-file sampleflume.properties  --name agent1 -Dflume.root.logger=DEBUG,console
    YOu will see flume printing messages on the console while it is starting like this
  3. Once server is started you can connect to it using nc or telnet and send messages to it like this. Whatever messages you send will be printed to console
  4. Once you send messages using nc command look at the server console and you should see the messages that you sent

2 comments: