ELK 收集 Java 后台日志

01 Java 日志样式

Java日志的特点在于输出信息非常多，通常需要将多行日志信息拼成一个事件，所以需要多行匹配模式。由于Elasticsearch本身就是使用Java开发的，所以Java日志收集实例就直接收集ES的日志。

如下所示是Elasticsearch的几条日志目录，可以看到这些日志条目通过第一个中括号中的时间戳进行区分，第二个日志条目中有多行Java日志，这多行日志组成了一个事件，怎么使用Filebeat采集这种多行日志呢？

[2021-08-02T07:14:18,201][INFO ][o.e.x.s.c.f.PersistentCache] [master] persistent cache index loaded
[2021-08-02T07:14:28,351][ERROR][o.e.b.Bootstrap          ] [master] Exception
.elasticsearch.transport.BindTransportException: Failed to bind to 172.16.255.13:[9300-9400]at .elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:406) ~[elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:370) ~[elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:120) ~[?:?]at .elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.doStart(SecurityNetty4Transport.java:85) ~[?:?]at .elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport.doStart(SecurityNetty4ServerTransport.java:47) ~[?:?]at .elasticsearch.mon.ponent.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:48) ~[elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.transport.TransportService.doStart(TransportService.java:263) ~[elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.mon.ponent.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:48) ~[elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.node.Node.start(Node.java:865) ~[elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:311) ~[elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:406) [elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.13.2.jar:7.13.2]at .elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.13.2.jar:7.13.2]at .elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.13.2.jar:7.13.2]at .elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.13.2.jar:7.13.2]
Caused by: java.net.BindException: Cannot assign requested addressat sun.nio.ch.Net.bind0(Native Method) ~[?:?]at sun.nio.ch.Net.bind(Net.java:552) ~[?:?]at sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:336) ~[?:?]at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294) ~[?:?]at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134) ~[?:?]at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550) ~[?:?]at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) ~[?:?]at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) ~[?:?]at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) ~[?:?]at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) ~[?:?]at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248) ~[?:?]at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) ~[?:?]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[?:?]at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[?:?]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[?:?]at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]at java.lang.Thread.run(Thread.java:831) ~[?:?]
[2021-08-02T07:14:28,357][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [master] uncaught exception in thread [main]

02 配置 Filbeat 多行匹配收集多行日志

多行匹配配置参考官方文档：多行日志收集配置

参考多行日志配置指导，配置Filebeat采集Java日志的输入如下

# ------------------------------Elasticsearch-Java----------------------------------
- type: logenabled: truepaths:# - /var/log/tomcat8/localhost_aess_log.2021-08-02.log- /var/log/elasticsearch/elasticsearch.logtags: ["es-java"]# 多行日志配置一下四行内容multiline.type: patternmultiline.pattern: '^\['multiline.negate: truemultiline.match: after

03 测试 Filbeat 收集多行日志

先启动Filebeat让其一直收集ES中的Java日志，然后修改ES的配置文件使其产生多行错误日志，最后修复ES配置文件并查看日志采集结果

# 修改配置文件并重启Filebeat
root@master:/etc/filebeat$ vim /etc/filebeat/filebeat.yml
root@master:/etc/filebeat$ systemctl restart filebeat
# 修改ES的配置文件（可以通过修改IP地址制作错误），使其启动失败产生多行输出的错误日志
root@master:/etc/filebeat$ vim /etc/elasticsearch/elasticsearch.yml 
root@master:/etc/filebeat$ systemctl restart elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
# 修复ES的配置文件，并重新启动查看多行错误日志是否被正确收集
root@master:/etc/filebeat$ vim /etc/elasticsearch/elasticsearch.yml 
root@master:/etc/filebeat$ systemctl restart elasticsearch

查看ES-head，是否成功采集生成对应索引

使用Kibana查看是否正确收集多行Java日志