多端口服务的Ingress IP"/>
多端口服务的Ingress IP
背景
业务反馈使用Ingress的ip-hash, 同一个服务开启了http和websocket分别是两个端口, 但是配置ip-hash后, 同一个client的请求http和websocket不在同一个后端.
探究
根据业务Ingress配置,配置如下实例:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/cors-allow-origin: '*'nginx.ingress.kubernetes.io/enable-cors: "true"nginx.ingress.kubernetes.io/proxy-body-size: 200mnginx.ingress.kubernetes.io/proxy-read-timeout: "300"nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addrname: hellogo
spec:rules:- host: hellogo.d.xiaomihttp:paths:- backend:serviceName: hellogo #http1, 8080servicePort: 8080path: /8080- backend:serviceName: hellogo #http2, 9090servicePort: 9090path: /9090- backend:serviceName: hellogo #websocket, 8081servicePort: 8081path: /ws
创建多个副本
$ kubectl get po -l app=hellogo
NAME READY STATUS RESTARTS AGE
hellogo-699f997454-b5vs4 1/1 Running 0 66m
hellogo-699f997454-hm924 1/1 Running 0 66m
hellogo-699f997454-mfbqv 1/1 Running 0 66m
hellogo-699f997454-qdrwn 1/1 Running 0 66m
hellogo-699f997454-srh9b 1/1 Running 0 66m
hellogo-699f997454-wlwfh 1/1 Running 0 66m
测试http 8080端口, 请求到pod hellogo-699f997454-qdrwn
$ curl
hello 8080!
host hellogo.d.xiaomi
remoteaddr 10.46.23.1:15340
realip 10.232.41.102
hostname hellogo-699f997454-qdrwn $ curl
hello 8080!
host hellogo.d.xiaomi
remoteaddr 10.46.23.1:15866
realip 10.232.41.102
hostname hellogo-699f997454-qdrwn
测试http 8080端口, 请求到pod hellogo-699f997454-b5vs4
$ curl
hello 9090!
host hellogo.d.xiaomi
remoteaddr 10.38.200.195:23706
realip 10.232.41.102
hostname hellogo-699f997454-b5vs4$ curl
hello 9090!
host hellogo.d.xiaomi
remoteaddr 10.38.200.195:23706
realip 10.232.41.102
hostname hellogo-699f997454-b5vs4
猜想是由于获取的nginx server列表顺序不一致导致的, 但是看源码ip list是直接从endpoint获取的, 进入nginx-ingress查看
$ kubectl exec -it -n kube-system nginx-ingress-controller-m496n sh
# dbg工具查看nginx后端列表
/etc/nginx $ /dbg backends list | grep hellogo
default-hellogo-8080
default-hellogo-8081
default-hellogo-9090
# 8080端口的列表
/etc/nginx $ /dbg backends get default-hellogo-8080
{"endpoints": [{"address": "10.46.12.107","port": "8080"},{"address": "10.46.12.108","port": "8080"},{"address": "10.46.12.109","port": "8080"},{"address": "10.46.23.23","port": "8080"},{"address": "10.46.23.25","port": "8080"},{"address": "10.46.23.29","port": "8080"}],"name": "default-hellogo-8080","noServer": false,"port": 8080,...
}
# 9090端口的列表
/etc/nginx $ /dbg backends get default-hellogo-9090
{"endpoints": [{"address": "10.46.12.107","port": "9090"},{"address": "10.46.12.108","port": "9090"},{"address": "10.46.12.109","port": "9090"},{"address": "10.46.23.23","port": "9090"},{"address": "10.46.23.25","port": "9090"},{"address": "10.46.23.29","port": "9090"}],"name": "default-hellogo-9090","noServer": false,"port": 9090,...
}
对比发现两个端口的列表是一样的,只能看看代码.
ip-hash代码在.lua
function _M.new(self, backend)local nodes = util.get_nodes(backend.endpoints)local o = {instance = self.factory:new(nodes), --获取后端pod ip列表hash_by = backend["upstreamHashByConfig"]["upstream-hash-by"],traffic_shaping_policy = backend.trafficShapingPolicy,alternative_backends = backend.alternativeBackends,}setmetatable(o, self)self.__index = selfreturn o
endfunction _M.balance(self)local key = util.lua_ngx_var(self.hash_by) --获取需要hash的变量return self.instance:find(key) --计算hash值
endreturn _M
关键是在get_nodes
函数,位于.lua
function _M.get_nodes(endpoints)local nodes = {}local weight = 1 --所有后端weight相同都为1for _, endpoint in pairs(endpoints) dolocal endpoint_string = endpoint.address .. ":" .. endpoint.port --endpoint为ip+portnodes[endpoint_string] = weightendreturn nodes
end
通过代码可以看到在ingress-nginx
中,实际的后端(upstream)是包含端口的,通过hash计算得到的值也不一样。
解决建议
首先确认系统的架构是不是合理,不同的端口提供不同的服务,一般是相互独立的。
如果确实有类似需求:
- 通过同一个端口提供服务,使用path来区分不同功能
- 修改代码,也比较简单
更多文章见 Qinng’s Blog
更多推荐
多端口服务的Ingress IP
发布评论