ELK 日志收集中使用 Nginx 代理优化 Redis 缓存

使用Redis 缓存优化 ELK 日志收集请参考文章 使用 Redis 缓存优化 ELK 日志收集

01 ELK 使用 Redis 作为缓存的高可用架构

ELK使用Redis作为缓存的一个主要缺陷是：被收集的数据只能够缓存在单台Redis中。这降低了整体架构的可靠性，为了解决这一问题，使用Nginx实现负载均衡，使用keepalived实现主从热备，使用nginx+keepalived的方式代理多台Redis，达到整体架构高性能高可靠的目的。

如图所示架构主要分为四个部分：

日志收集：Filebeat采集Nginx日志负载均衡：Nginx通过负载均衡算法将收集的日志数据均匀的分配到不同Redis节点中存储高可用：keepalived将Virtual IP映射到真实Redis节点IP地址， 同时也可以设置Redis节点的主从热备关系；另外，keepalived还要检测本机Nginx状态日志聚合存储与展示：Logstash消费Redis日志数据传送给ES存储，最后使用Kibana展示

02 部署安装相关组件

2.1 部署安装keepalived

keepalived高可用详细内容参考：keepalived高可用

选择两个节点安装Nginx和keepalived，作为高可用节点

HOSTNAME             IP                               说明
master          172.16.255.131    Keepalived主服务器(Nginx主负载均衡器)
node1           172.16.255.132    Keepalived备服务器(Nginx备负载均衡器)

安装keepalived

sudo apt-get install keepalived

配置keepalived

keepalived软件的配置文件默认路径及配置文件名 /etc/keepalived/keepalived.conf

这里的具备高可用功能keepalived.conf配置文件包含了两个重要区块:

全局变量(Global Definitions)部分：这部分主要用来设置keepalived的故障通知机制和Router ID标识VRRP实例定义区域(VRRP instance(s))部分：这部分主要用来定义具体服务实例配置，包括Keepalived主备状态,接口,优先级,认证方式和IP信息等

代理Redis服务中，主要是通过keepalived配置Virtual IP，master节点keepalived配置如下

root@master:/home/wang$ vim /etc/keepalived/keepalived.conf
root@master:/home/wang$ cat /etc/keepalived/keepalived.conf 
global_defs {router_id lb01
}
vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 50priority 150advert_int 1authentication {auth_type PASSauth_pass 1111}# 配置虚拟IPvirtual_ipaddress {172.16.255.1}
}

node1节点keepalived配置如下

root@node1:/home/wang$ vim /etc/keepalived/keepalived.conf
root@node1:/home/wang$ cat /etc/keepalived/keepalived.conf
global_defs {router_id lb02
}
vrrp_instance VI_1 {state BACKUPinterface eth0virtual_router_id 50priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.16.255.1}
}

启动keepalived

在两个节点都是用如下命令启动keepalived

systemctl start keepalived

在剩下的第三个节点中使用ping命令，测试虚拟地址是否配置成功

root@node2:/home/wang$ ping 172.16.255.1
PING 172.16.255.1 (172.16.255.1) 56(84) bytes of data.
64 bytes from 172.16.255.1: icmp_seq=1 ttl=64 time=0.144 ms

2.2 部署安装 Redis

选择两个节点安装Redis，作为数据缓存节点

Redis安装参考：Redis安装部署

安装部署完成之后在两个节点上都是用Redis运维脚本启动Redis

root@master:/home/wang$ sh redis_shell.sh start 6379
root        947      1  0 02:17 ?        00:00:39 /usr/local/bin/redis-server 127.0.0.1:6379
root       3625   2552  0 08:20 pts/0    00:00:00 sh redis_shell.sh start 6379
root       3633      1  0 08:20 ?        00:00:00 redis-server 172.16.255.131:6379
root       3635   3625  0 08:20 pts/0    00:00:00 grep redis

03 配置相关组件

3.1 Nginx反向代理配置

在添加反向代理配置stream模块时，主要要在nginx.conf里的文件末尾添加，而不是在其中添加子配置，添加内容如下

stream {# 代理的Redis服务节点upstream redis{server 172.16.255.131:6379 max_fails=2 fail_timeout=10s; server 172.16.255.132:6379 max_fails=2 fail_timeout=10s backup; # 添加backup选项可以指定该redis服务节点为备份}server {listen     6379;proxy_connect_timeout      1s;proxy_timeout      3s;proxy_pass      redis;}
}

完整Nginx配置文件如下

root@master:/home/wang# cat /etc/nginx/nginx.conf 
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;events {worker_connections 768;# multi_aept on;
}http {### Basic Settings##sendfile on;tcp_nopush on;tcp_nodelay on;keepalive_timeout 65;types_hash_max_size 2048;# server_tokens off;# server_names_hash_bucket_size 64;# server_name_in_redirect off;include /etc/nginx/mime.types;default_type application/octet-stream;### SSL Settings##ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLEssl_prefer_server_ciphers on;### Logging Settings### log_json format added by wanghaihua on 2021.07.30log_format log_json '{ "@time_local": "$time_local", ''"remote_addr": "$remote_addr", ''"referer": "$http_referer", ''"request": "$request", ''"status": $status, ''"bytes": $body_bytes_sent, ''"agent": "$http_user_agent", ''"x_forwarded": "$http_x_forwarded_for", ''"up_addr": "$upstream_addr",''"up_host": "$upstream_http_host",''"up_resp_time": "$upstream_response_time",''"request_time": "$request_time"'' }';# aess_log /var/log/nginx/aess.log;aess_log /var/log/nginx/aess.log log_json;error_log /var/log/nginx/error.log;### Gzip Settings##gzip on;# gzip_vary on;# gzip_proxied any;# gzip_p_level 6;# gzip_buffers 16 8k;# gzip_http_version 1.1;# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;### Virtual Host Configs##include /etc/nginx/conf.d/*.conf;include /etc/nginx/sites-enabled/*;
}#mail {
#	# See sample authentication script at:
#	# wiki.nginx./ImapAuthenticateWithApachePhpScript
# 
#	# auth_http localhost/auth.php;
#	# pop3_capabilities "TOP" "USER";
#	# imap_capabilities "IMAP4rev1" "UIDPLUS";
# 
#	server {
#		listen     localhost:110;
#		protocol   pop3;
#		proxy      on;
#	}
# 
#	server {
#		listen     localhost:143;
#		protocol   imap;
#		proxy      on;
#	}
#}stream {upstream redis{server 172.16.255.131:6379 max_fails=2 fail_timeout=10s;server 172.16.255.132:6379 max_fails=2 fail_timeout=10s;}server {listen     6379;proxy_connect_timeout      1s;proxy_timeout      3s;proxy_pass      redis;}
}

3.2 Filebeat输出配置

Nginx使用虚拟地址反向代理了Redis缓存的服务节点，所以Filebeat使用一个虚拟IP就可以将采集的日志数据缓存到多台Redis节点中

修改Filebeat输出配置对应的Redis地址为虚拟地址如下

# ------------------------------ Redis Output -------------------------------
output.redis:hosts: ["172.16.255.1"] # 修改为虚拟地址keys:- key: "nginx_aess"   # send to info_list if `message` field contains INFOwhen.contains:tags: "aess"- key: "nginx_error"   # send to info_list if `message` field contains INFOwhen.contains:tags: "error"

Filebeat 完整配置如下

root@master:/home/wang$ cat /etc/filebeat/filebeat.yml 
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# -----------------------------Nginx-----------------------------------
- type: logenabled: truepaths:- /var/log/nginx/aess.logjson.keys_under_root: truejson.overwrite_keys: truetags: ["aess"]- type: logenabled: truepaths:- /var/log/nginx/error.logtags: ["error"]# ======================= Elasticsearch template setting =======================
setup.template.settings:index.number_of_shards: 1
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true# =================================== Kibana ==============================
setup.kibana:host: "172.16.255.131:5601"# ------------------------------ Redis Output -------------------------------
output.redis:hosts: ["172.16.255.1"]keys:- key: "nginx_aess"   # send to info_list if `message` field contains INFOwhen.contains:tags: "aess"- key: "nginx_error"   # send to info_list if `message` field contains INFOwhen.contains:tags: "error"

3.3 Logstash输入配置

Nginx使用虚拟地址反向代理了Redis缓存的服务节点，所以Logstash使用一个虚拟IP就可以消费缓存在多台Redis节点中日志数据

修改Logstash输入配置对应的Redis地址为虚拟地址如下

input {redis {host => "172.16.255.1" # 修改为虚拟地址port => 6379db => "0"key => "nginx_aess"data_type => "list"type  => "redis"}
}

完整Logstash配置如下

root@master:/home/wang$ cat /etc/logstash/conf.d/redis.conf
input {redis {host => "172.16.255.1"port => 6379db => "0"key => "nginx_aess"data_type => "list"type  => "redis"codec => plain{charset=>"UTF-8"}}redis {host => "172.16.255.1"port => 6379db => "0"key => "nginx_error"data_type => "list"type  => "redis"codec => plain{charset=>"UTF-8"}}
}filter {mutate {convert => ["upstream_time","float"]convert => ["request_time","float"]}json {source => "message"}
}

3.4 配置完成之后重新启动Filebeat和Logstash

systemctl restart filebeat
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf