实时写入Flume"/>
使用Log4j将日志实时写入Flume
需求:使用Flume实时收集WebServer或者其它服务器上通过log4j产生的日志
Agent选型:Log4jAppender的使用
创建一个maven项目,pom.xml文件中引入依赖
<dependency><groupId>log4j</groupId><artifactId>log4j</artifactId><version>1.2.16</version>
</dependency><dependency><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId><version>1.7.21</version><scope>test</scope>
</dependency>
在src/main/resource目录下新建log4j.properties,配置如下
log4j.rootLogger=INFO,stdout
log4j.appender.stdout = org.apache.log4j.ConsoleAppender
log4j.appender.stdout.target = System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n
将resource目录加载到classpath下
选择左侧Modules,找到resource目录,点击上面的Resources
新建LoggerGenerate类,用于产生日志(模仿项目中的日志输出)
import org.apache.log4j.Logger;/*** 循环打印值,模仿日志输出*/
public class LoggerGenerate {private static Logger logger = Logger.getLogger(LoggerGenerate.class.getName());public static void main(String[] args) throws InterruptedException {int index = 0;while (true) {Thread.sleep(1000);logger.info("value is:" + index++);}}
}
运行测试(部分结果):
2017-9-25 09:44:45,238 [main] [LoggerGenerate] [INFO] - value is:0
2017-9-25 09:44:46,241 [main] [LoggerGenerate] [INFO] - value is:1
2017-9-25 09:44:47,241 [main] [LoggerGenerate] [INFO] - value is:2
2017-9-25 09:44:48,241 [main] [LoggerGenerate] [INFO] - value is:3
2017-9-25 09:44:49,255 [main] [LoggerGenerate] [INFO] - value is:4
2017-9-25 09:44:50,258 [main] [LoggerGenerate] [INFO] - value is:5
目前这些日志信息都打印在了控制台,接下来我们想要把这些日志收集到flume,首先先配置Agent
# Name the components on this agent
flume-log-agent.sources = arvo-source
flume-log-agent.sinks = logger-sink
flume-log-agent.channels = memory-channel# Describe/configure the source
flume-log-agent.sources.arvo-source.type = avro
flume-log-agent.sources.arvo-source.bind = Master
flume-log-agent.sources.arvo-source.port = 44444# Describe the sink
flume-log-agent.sinks.logger-sink.type = logger# Use a channel which buffers events in memory
flume-log-agent.channels.memory-channel.type = memory# Bind the source and sink to the channel
flume-log-agent.sources.arvo-source.channels = memory-channel
flume-log-agent.sinks.logger-sink.channel = memory-channel
启动Agent
flume-ng agent \
--name flume-log-agent \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/config/flume-log-agent.conf \
-Dflume.root.logger=INFO,console
想要将控制台的日志收集到flume,需要修改log4j.properties中的配置
参考官网:.html
log4j.properties中新增内容如下配置
log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = Master
log4j.appender.flume.Port = 44444
log4j.appender.flume.UnsafeMode = true
在 log4j.rootLogger=INFO,stdout 后面加上flume
log4j.rootLogger=INFO,stdout,flume
运行测试,笔者的项目是在windows下创建的(你也可以将项目部署在linux服务器中),需要在windows的hosts文件中配置映射,否者在log4j中是不可以写主机名的,只能写ip地址,参照下图配置即可
在hosts文件中添加一行内容(格式:IP 主机名)
192.168.242.150 Master
在运行期间,报了一个错误,如下
java.lang.ClassNotFoundException: org.apache.flume.clients.log4jappender.Log4jAppender
报错的原因是类找不到,很明显是项目中没有添加依赖,在pom.xml文件中添加flume-ng-log4jappender的依赖
<dependency><groupId>org.apache.flume.flume-ng-clients</groupId><artifactId>flume-ng-log4jappender</artifactId><version>1.6.0</version>
</dependency>
再次运行项目,查看Agent控制台中打印的信息
2017-09-25 10:16:12,862 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926512143, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 30 value is...0 }
2017-09-25 10:16:12,863 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926513197, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 31 value is...1 }
2017-09-25 10:16:12,864 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926514203, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 32 value is...2 }
2017-09-25 10:16:13,082 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926515210, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 33 value is...3 }
2017-09-25 10:16:14,092 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926516221, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 34 value is...4 }
2017-09-25 10:16:15,103 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926517228, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 35 value is...5 }
总结:在Agent的配置文件编写过程中,常用的参数设置,无需死记硬背(参照官网即可)
更多推荐
使用Log4j将日志实时写入Flume
发布评论