Flume主要有以下集中監控方式:
1.JMX監控
配置 {$FLUME_HOME}/flume-env.sh
cd $FLUME_HOME
vi flume-env.sh
JAVA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.port=54321 \
-Dcom.sun.management.jmxremote.rmi.port=54322
-Djava.rmi.server.hostname=192.168.16.214"
之后啟動flume
bin/flume-ng agent -c . -f conf/exec-tail.conf -n a1 -Dflume.root.logger=INFO,console
在圖形界面的系統(windows、mac、linux圖形)和jdk的環境下啟動jconsole
jconsole
可以看到flume應用所使用的內存、線程、類、CPU等使用情況。
2.HTTP監控
Flume可以通過HTTP以JSON形式報告metrics,啟用HTTP監控,Flume需要配置一個端口。
配置一個簡單的打印conf文件
vi conf/exec-tail.conf
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /tmp/test
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
啟動flume并指定http監控形式和端口
bin/flume-ng agent -c . -f conf/exec-tail.conf -n a1 -Dflume.root.logger=INFO,console -Dflume.monitoring.type=http -Dflume.monitoring.port=1234
查看metric監控
用瀏覽器或者GET方式打開
http://ip:1234/metrics
獲得數據如下:
CURL方式:
http://localhost:1234/metrics 2>/dev/null|sed -e 's/([,])\s*/\1\n/g' -e 's/[{}]/\n/g' -e 's/[“,]//g'
獲得數據如下:
"CHANNEL.c1":
"ChannelCapacity":"1000"
"ChannelFillPercentage":"0.0"
"Type":"CHANNEL"
"EventTakeSuccessCount":"10"
"ChannelSize":"0"
"EventTakeAttemptCount":"12"
"StartTime":"1476166839656"
"EventPutAttemptCount":"10"
"EventPutSuccessCount":"10"
"StopTime":"0"
"SOURCE.r1":
"EventReceivedCount":"10"
"AppendBatchAcceptedCount":"0"
"Type":"SOURCE"
"AppendReceivedCount":"0"
"EventAcceptedCount":"10"
"StartTime":"1476166840159"
"AppendAcceptedCount":"0"
"OpenConnectionCount":"0"
"AppendBatchReceivedCount":"0"
"StopTime":"0"
3.Ganglia監控
Flume也可以報告metrics到Ganglia 3或者是Ganglia 3.1的metanodes。要將metrics報告到Ganglia,必須在啟動的時候就支持Flume Agent。這個Flume Agent使用flume.monitoring作為前綴,通過下面的參數啟動。當然也可以在flume-env.sh中設置:
如果要支持Ganglia,可以通過如下命令啟動。
bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=com.example:1234,com.example2:5455
4.自定義監控
自定義的監控需要實現org.apache.flume.instrumentation.MonitorService
接口。例如有一個HTTP的監控類叫HttpReporting,我可以通過如下方式啟動這個監控。
bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=com.example.reporting.HTTPReporting -Dflume.monitoring.node=com.example:332
報告metrics我們也可以自定義組件,不過一定要繼承org.apache.flume.instrumentation.MonitoredCounterGroup
虛擬類。Flume已經實現的類,如下圖:
根據上面的規范就可以開發自定義的監控組件了。