prometheus使用邮件和企业微信告警

邮件和企业微信告警"/>

prometheus使用邮件和企业微信告警

1：prometheus使用邮件和微信告警

参考文档：
官方：
githup：.yaml

注意：没开启POP3/SMTP服务的要到163官网开启并获取授权码

编写alertmanager-secret.yaml文件

metadata:name: alertmanager-mainnamespace: monitoring
stringData:alertmanager.yaml: |-"global":      #以下添加"resolve_timeout": "5m"smtp_from: "lsj13006818773@163"smtp_smarthost: "smtp.163:465"smtp_hello: "163"smtp_auth_username: "lsj13006818773@163"smtp_auth_password: "SNUDIRHSJYBAPJDD"  #第三方登陆邮箱授权码smtp_require_tls: false"inhibit_rules":- "equal":- "namespace"- "alertname""source_match":"severity": "critical""target_match_re":"severity": "warning|info"- "equal":- "namespace"- "alertname""source_match":"severity": "warning""target_match_re":"severity": "info""receivers":- "name": "Default"    #以下添加"email_configs":- to: "lsj13006818773@163"send_resolved: true- "name": "Watchdog""email_configs":- to: "lsj13006818773@163"send_resolved: true- "name": "Critical""email_configs":- to: "lsj13006818773@163"send_resolved: true"route":"group_by":- "namespace""group_interval": "5m""group_wait": "30s""receiver": "Default""repeat_interval": "12h""routes":- "match":"alertname": "Watchdog""receiver": "Watchdog"- "match":"severity": "critical""receiver": "Critical"
type: Opaque

参数参考：

# global块配置下的配置选项在本配置文件内的所有配置项下可见
global:# 在Alertmanager内管理的每一条告警均有两种状态: "resolved"或者"firing". 在altermanager首次发送告警通知后, 该告警会一直处于firing状态,设置resolve_timeout可以指定处于firing状态的告警间隔多长时间会被设置为resolved状态, 在设置为resolved状态的告警后,altermanager不会再发送firing的告警通知.resolve_timeout: 1h# 邮件告警配置smtp_smarthost: 'smtp.exmail.qq:25'smtp_from: 'dukuan@xxx'smtp_auth_username: 'dukuan@xxx'smtp_auth_password: 'DKxxx'# HipChat告警配置# hipchat_auth_token: '123456789'# hipchat_auth_url: '/'# wechatwechat_api_url: '/'wechat_api_secret: 'JJ'wechat_api_corp_id: 'ww'# 告警通知模板
templates:
- '/etc/alertmanager/config/*.tmpl'# route: 根路由,该模块用于该根路由下的节点及子路由routes的定义. 子树节点如果不对相关配置进行配置，则默认会从父路由树继承该配置选项。每一条告警都要进入route，即要求配置选项group_by的值能够匹配到每一条告警的至少一个labelkey(即通过POST请求向altermanager服务接口所发送告警的labels项所携带的<labelname>)，告警进入到route后，将会根据子路由routes节点中的配置项match_re或者match来确定能进入该子路由节点的告警(由在match_re或者match下配置的labelkey: labelvalue是否为告警labels的子集决定，是的话则会进入该子路由节点，否则不能接收进入该子路由节点).
route:# 例如所有labelkey:labelvalue含cluster=A及altertname=LatencyHigh labelkey的告警都会被归入单一组中group_by: ['job', 'altername', 'cluster', 'service','severity']# 若一组新的告警产生，则会等group_wait后再发送通知，该功能主要用于当告警在很短时间内接连产生时，在group_wait内合并为单一的告警后再发送group_wait: 30s# 再次告警时间间隔group_interval: 5m# 如果一条告警通知已成功发送，且在间隔repeat_interval后，该告警仍然未被设置为resolved，则会再次发送该告警通知repeat_interval: 12h# 默认告警通知接收者，凡未被匹配进入各子路由节点的告警均被发送到此接收者receiver: 'wechat'# 上述route的配置会被传递给子路由节点，子路由节点进行重新配置才会被覆盖# 子路由树routes:# 该配置选项使用正则表达式来匹配告警的labels，以确定能否进入该子路由树# match_re和match均用于匹配labelkey为service,labelvalue分别为指定值的告警，被匹配到的告警会将通知发送到对应的receiver- match_re:service: ^(foo1|foo2|baz)$receiver: 'wechat'# 在带有service标签的告警同时有severity标签时，他可以有自己的子路由，同时具有severity != critical的告警则被发送给接收者team-ops-mails,对severity == critical的告警则被发送到对应的接收者即team-ops-pagerroutes:- match:severity: criticalreceiver: 'wechat'# 比如关于数据库服务的告警，如果子路由没有匹配到相应的owner标签，则都默认由team-DB-pager接收- match:service: databasereceiver: 'wechat'# 我们也可以先根据标签service:database将数据库服务告警过滤出来，然后进一步将所有同时带labelkey为database- match:severity: criticalreceiver: 'wechat'
# 抑制规则，当出现critical告警时 忽略warning
inhibit_rules:
- source_match:severity: 'critical'target_match:severity: 'warning'# Apply inhibition if the alertname is the same.#   equal: ['alertname', 'cluster', 'service']#
# 收件人配置
receivers:
- name: 'team-ops-mails'email_configs:- to: 'dukuan@xxx'
- name: 'wechat'wechat_configs:- send_resolved: truecorp_id: 'ww'api_secret: 'JJ'to_tag: '1'agent_id: '1000002'api_url: '/'message: '{{ template "wechat.default.message" . }}'
#- name: 'team-X-pager'
#  email_configs:
#  - to: 'team-X+alerts-critical@example'
#  pagerduty_configs:
#  - service_key: <team-X-key>
#
#- name: 'team-Y-mails'
#  email_configs:
#  - to: 'team-Y+alerts@example'
#
#- name: 'team-Y-pager'
#  pagerduty_configs:
#  - service_key: <team-Y-key>
#
#- name: 'team-DB-pager'
#  pagerduty_configs:
#  - service_key: <team-DB-key>
#  
#- name: 'team-X-hipchat'
#  hipchat_configs:
#  - auth_token: <auth_token>
#    room_id: 85
#    message_format: html
#    notify: true 

更新secret

kubectl delete -f alertmanager-secret.yaml
kubectl apply -f alertmanager-secret.yaml

验证：查看邮件

2：Alertmanager创建告警屏蔽
备注：有新的项目不需要看告警的，可以使用Alertmanager屏蔽掉
/

创建一个屏蔽项目

2：使用企业告警

1：创建应用选择部门/成员

编写alertmanager-secret.yaml文件

metadata:name: alertmanager-mainnamespace: monitoring
stringData:alertmanager.yaml: |-"global":      #以下添加"resolve_timeout": "5m"smtp_from: "lsj13006818773@163"smtp_smarthost: "smtp.163:465"smtp_hello: "163"smtp_auth_username: "lsj13006818773@163"smtp_auth_password: "SNUDIRHSJYBAPJDD"  #第三方登陆邮箱授权码smtp_require_tls: false#wechatwechat_api_url: '/'wechat_api_secret: 'JJ'   #在应用管理个人信息里wechat_api_corp_id: 'ww'  #企业ID（我的企业）"inhibit_rules":- "equal":- "namespace"- "alertname""source_match":"severity": "critical""target_match_re":"severity": "warning|info"- "equal":- "namespace"- "alertname""source_match":"severity": "warning""target_match_re":"severity": "info""receivers":- "name": "Default"    #以下添加"email_configs":- to: "lsj13006818773@163"send_resolved: true- "name": "Watchdog""email_configs":- to: "lsj13006818773@163"send_resolved: true- "name": "Critical""email_configs":- to: "lsj13006818773@163"send_resolved: true- name: 'wechat'wechat_configs:- send_resolved: trueto_tag: '1'    #部门IDagent_id: '1000002'  #应用ID"route":"group_by":- "namespace""group_interval": "5m""group_wait": "30s""receiver": "Default""repeat_interval": "12h""routes":- "match":"alertname": "Watchdog""receiver": "wechat"    #修改为wechat告警- "match":"severity": "critical""receiver": "Critical"
type: Opaque