使用Redis 缓存优化 ELK 日志收集请参考文章 使用 Redis 缓存优化 ELK 日志收集
01 ELK 使用 Redis 作为缓存的高可用架构
ELK使用Redis作为缓存的一个主要缺陷是:被收集的数据只能够缓存在单台Redis中。这降低了整体架构的可靠性,为了解决这一问题,使用Nginx实现负载均衡,使用keepalived实现主从热备,使用nginx+keepalived的方式代理多台Redis,达到整体架构高性能高可靠的目的。
如图所示架构主要分为四个部分:
日志收集:Filebeat采集Nginx日志负载均衡:Nginx通过负载均衡算法将收集的日志数据均匀的分配到不同Redis节点中存储高可用:keepalived将Virtual IP映射到真实Redis节点IP地址, 同时也可以设置Redis节点的主从热备关系;另外,keepalived还要检测本机Nginx状态日志聚合存储与展示:Logstash消费Redis日志数据传送给ES存储,最后使用Kibana展示02 部署安装相关组件
2.1 部署安装keepalived
keepalived高可用详细内容参考:keepalived高可用
选择两个节点安装Nginx和keepalived,作为高可用节点
HOSTNAME IP 说明
master 172.16.255.131 Keepalived主服务器(Nginx主负载均衡器)
node1 172.16.255.132 Keepalived备服务器(Nginx备负载均衡器)
安装keepalived
sudo apt-get install keepalived
配置keepalived
keepalived软件的配置文件默认路径及配置文件名 /etc/keepalived/keepalived.conf
这里的具备高可用功能keepalived.conf配置文件包含了两个重要区块:
全局变量(Global Definitions)部分:这部分主要用来设置keepalived的故障通知机制和Router ID标识VRRP实例定义区域(VRRP instance(s))部分:这部分主要用来定义具体服务实例配置,包括Keepalived主备状态,接口,优先级,认证方式和IP信息等代理Redis服务中,主要是通过keepalived配置Virtual IP,master节点keepalived配置如下
root@master:/home/wang$ vim /etc/keepalived/keepalived.conf
root@master:/home/wang$ cat /etc/keepalived/keepalived.conf
global_defs {router_id lb01
}
vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 50priority 150advert_int 1authentication {auth_type PASSauth_pass 1111}# 配置虚拟IPvirtual_ipaddress {172.16.255.1}
}
node1节点keepalived配置如下
root@node1:/home/wang$ vim /etc/keepalived/keepalived.conf
root@node1:/home/wang$ cat /etc/keepalived/keepalived.conf
global_defs {router_id lb02
}
vrrp_instance VI_1 {state BACKUPinterface eth0virtual_router_id 50priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.16.255.1}
}
启动keepalived
在两个节点都是用如下命令启动keepalived
systemctl start keepalived
在剩下的第三个节点中使用ping
命令,测试虚拟地址是否配置成功
root@node2:/home/wang$ ping 172.16.255.1
PING 172.16.255.1 (172.16.255.1) 56(84) bytes of data.
64 bytes from 172.16.255.1: icmp_seq=1 ttl=64 time=0.144 ms
2.2 部署安装 Redis
选择两个节点安装Redis,作为数据缓存节点
Redis安装参考:Redis安装部署
安装部署完成之后在两个节点上都是用Redis运维脚本启动Redis
root@master:/home/wang$ sh redis_shell.sh start 6379
root 947 1 0 02:17 ? 00:00:39 /usr/local/bin/redis-server 127.0.0.1:6379
root 3625 2552 0 08:20 pts/0 00:00:00 sh redis_shell.sh start 6379
root 3633 1 0 08:20 ? 00:00:00 redis-server 172.16.255.131:6379
root 3635 3625 0 08:20 pts/0 00:00:00 grep redis
03 配置相关组件
3.1 Nginx反向代理配置
在添加反向代理配置stream模块时,主要要在nginx.conf
里的文件末尾添加,而不是在其中添加子配置,添加内容如下
stream {# 代理的Redis服务节点upstream redis{server 172.16.255.131:6379 max_fails=2 fail_timeout=10s; server 172.16.255.132:6379 max_fails=2 fail_timeout=10s backup; # 添加backup选项可以指定该redis服务节点为备份}server {listen 6379;proxy_connect_timeout 1s;proxy_timeout 3s;proxy_pass redis;}
}
完整Nginx配置文件如下
root@master:/home/wang# cat /etc/nginx/nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;events {worker_connections 768;# multi_aept on;
}http {### Basic Settings##sendfile on;tcp_nopush on;tcp_nodelay on;keepalive_timeout 65;types_hash_max_size 2048;# server_tokens off;# server_names_hash_bucket_size 64;# server_name_in_redirect off;include /etc/nginx/mime.types;default_type application/octet-stream;### SSL Settings##ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLEssl_prefer_server_ciphers on;### Logging Settings### log_json format added by wanghaihua on 2021.07.30log_format log_json '{ "@time_local": "$time_local", ''"remote_addr": "$remote_addr", ''"referer": "$http_referer", ''"request": "$request", ''"status": $status, ''"bytes": $body_bytes_sent, ''"agent": "$http_user_agent", ''"x_forwarded": "$http_x_forwarded_for", ''"up_addr": "$upstream_addr",''"up_host": "$upstream_http_host",''"up_resp_time": "$upstream_response_time",''"request_time": "$request_time"'' }';# aess_log /var/log/nginx/aess.log;aess_log /var/log/nginx/aess.log log_json;error_log /var/log/nginx/error.log;### Gzip Settings##gzip on;# gzip_vary on;# gzip_proxied any;# gzip_p_level 6;# gzip_buffers 16 8k;# gzip_http_version 1.1;# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;### Virtual Host Configs##include /etc/nginx/conf.d/*.conf;include /etc/nginx/sites-enabled/*;
}#mail {
# # See sample authentication script at:
# # wiki.nginx./ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}stream {upstream redis{server 172.16.255.131:6379 max_fails=2 fail_timeout=10s;server 172.16.255.132:6379 max_fails=2 fail_timeout=10s;}server {listen 6379;proxy_connect_timeout 1s;proxy_timeout 3s;proxy_pass redis;}
}
3.2 Filebeat输出配置
Nginx使用虚拟地址反向代理了Redis缓存的服务节点,所以Filebeat使用一个虚拟IP就可以将采集的日志数据缓存到多台Redis节点中
修改Filebeat输出配置对应的Redis地址为虚拟地址如下
# ------------------------------ Redis Output -------------------------------
output.redis:hosts: ["172.16.255.1"] # 修改为虚拟地址keys:- key: "nginx_aess" # send to info_list if `message` field contains INFOwhen.contains:tags: "aess"- key: "nginx_error" # send to info_list if `message` field contains INFOwhen.contains:tags: "error"
Filebeat 完整配置如下
root@master:/home/wang$ cat /etc/filebeat/filebeat.yml
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# -----------------------------Nginx-----------------------------------
- type: logenabled: truepaths:- /var/log/nginx/aess.logjson.keys_under_root: truejson.overwrite_keys: truetags: ["aess"]- type: logenabled: truepaths:- /var/log/nginx/error.logtags: ["error"]# ======================= Elasticsearch template setting =======================
setup.template.settings:index.number_of_shards: 1
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true# =================================== Kibana ==============================
setup.kibana:host: "172.16.255.131:5601"# ------------------------------ Redis Output -------------------------------
output.redis:hosts: ["172.16.255.1"]keys:- key: "nginx_aess" # send to info_list if `message` field contains INFOwhen.contains:tags: "aess"- key: "nginx_error" # send to info_list if `message` field contains INFOwhen.contains:tags: "error"
3.3 Logstash输入配置
Nginx使用虚拟地址反向代理了Redis缓存的服务节点,所以Logstash使用一个虚拟IP就可以消费缓存在多台Redis节点中日志数据
修改Logstash输入配置对应的Redis地址为虚拟地址如下
input {redis {host => "172.16.255.1" # 修改为虚拟地址port => 6379db => "0"key => "nginx_aess"data_type => "list"type => "redis"}
}
完整Logstash配置如下
root@master:/home/wang$ cat /etc/logstash/conf.d/redis.conf
input {redis {host => "172.16.255.1"port => 6379db => "0"key => "nginx_aess"data_type => "list"type => "redis"codec => plain{charset=>"UTF-8"}}redis {host => "172.16.255.1"port => 6379db => "0"key => "nginx_error"data_type => "list"type => "redis"codec => plain{charset=>"UTF-8"}}
}filter {mutate {convert => ["upstream_time","float"]convert => ["request_time","float"]}json {source => "message"}
}
3.4 配置完成之后重新启动Filebeat和Logstash
systemctl restart filebeat
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf
更多推荐
缓存,日志,ELK,Redis,Nginx
发布评论