多端口服务的Ingress IP

编程入门行业动态更新时间:2024-10-26 23:38:27

多端口服务的Ingress IP

背景

业务反馈使用Ingress的ip-hash, 同一个服务开启了http和websocket分别是两个端口, 但是配置ip-hash后, 同一个client的请求http和websocket不在同一个后端.

探究

根据业务Ingress配置,配置如下实例:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/cors-allow-origin: '*'nginx.ingress.kubernetes.io/enable-cors: "true"nginx.ingress.kubernetes.io/proxy-body-size: 200mnginx.ingress.kubernetes.io/proxy-read-timeout: "300"nginx.ingress.kubernetes.io/upstream-hash-by: $binary_remote_addrname: hellogo
spec:rules:- host: hellogo.d.xiaomihttp:paths:- backend:serviceName: hellogo #http1, 8080servicePort: 8080path: /8080- backend:serviceName: hellogo #http2, 9090servicePort: 9090path: /9090- backend:serviceName: hellogo #websocket, 8081servicePort: 8081path: /ws

创建多个副本

$ kubectl get po -l app=hellogo
NAME                       READY   STATUS    RESTARTS   AGE
hellogo-699f997454-b5vs4   1/1     Running   0          66m
hellogo-699f997454-hm924   1/1     Running   0          66m
hellogo-699f997454-mfbqv   1/1     Running   0          66m
hellogo-699f997454-qdrwn   1/1     Running   0          66m
hellogo-699f997454-srh9b   1/1     Running   0          66m
hellogo-699f997454-wlwfh   1/1     Running   0          66m

测试http 8080端口, 请求到pod hellogo-699f997454-qdrwn

$ curl 
hello 8080!
host hellogo.d.xiaomi
remoteaddr 10.46.23.1:15340
realip 10.232.41.102
hostname hellogo-699f997454-qdrwn $ curl 
hello 8080!
host hellogo.d.xiaomi
remoteaddr 10.46.23.1:15866
realip 10.232.41.102
hostname hellogo-699f997454-qdrwn

测试http 8080端口, 请求到pod hellogo-699f997454-b5vs4

$ curl 
hello 9090!
host hellogo.d.xiaomi
remoteaddr 10.38.200.195:23706
realip 10.232.41.102
hostname hellogo-699f997454-b5vs4$ curl 
hello 9090!
host hellogo.d.xiaomi
remoteaddr 10.38.200.195:23706
realip 10.232.41.102
hostname hellogo-699f997454-b5vs4

猜想是由于获取的nginx server列表顺序不一致导致的, 但是看源码ip list是直接从endpoint获取的, 进入nginx-ingress查看

$ kubectl exec -it -n kube-system nginx-ingress-controller-m496n sh
# dbg工具查看nginx后端列表
/etc/nginx $ /dbg backends list | grep hellogo
default-hellogo-8080
default-hellogo-8081
default-hellogo-9090
# 8080端口的列表
/etc/nginx $ /dbg backends get default-hellogo-8080
{"endpoints": [{"address": "10.46.12.107","port": "8080"},{"address": "10.46.12.108","port": "8080"},{"address": "10.46.12.109","port": "8080"},{"address": "10.46.23.23","port": "8080"},{"address": "10.46.23.25","port": "8080"},{"address": "10.46.23.29","port": "8080"}],"name": "default-hellogo-8080","noServer": false,"port": 8080,...
}
# 9090端口的列表
/etc/nginx $ /dbg backends get default-hellogo-9090
{"endpoints": [{"address": "10.46.12.107","port": "9090"},{"address": "10.46.12.108","port": "9090"},{"address": "10.46.12.109","port": "9090"},{"address": "10.46.23.23","port": "9090"},{"address": "10.46.23.25","port": "9090"},{"address": "10.46.23.29","port": "9090"}],"name": "default-hellogo-9090","noServer": false,"port": 9090,...
}

对比发现两个端口的列表是一样的,只能看看代码.

ip-hash代码在.lua

function _M.new(self, backend)local nodes = util.get_nodes(backend.endpoints)local o = {instance = self.factory:new(nodes),  --获取后端pod ip列表hash_by = backend["upstreamHashByConfig"]["upstream-hash-by"],traffic_shaping_policy = backend.trafficShapingPolicy,alternative_backends = backend.alternativeBackends,}setmetatable(o, self)self.__index = selfreturn o
endfunction _M.balance(self)local key = util.lua_ngx_var(self.hash_by) --获取需要hash的变量return self.instance:find(key)  --计算hash值
endreturn _M

关键是在get_nodes函数,位于.lua

function _M.get_nodes(endpoints)local nodes = {}local weight = 1 --所有后端weight相同都为1for _, endpoint in pairs(endpoints) dolocal endpoint_string = endpoint.address .. ":" .. endpoint.port --endpoint为ip+portnodes[endpoint_string] = weightendreturn nodes
end

通过代码可以看到在ingress-nginx中,实际的后端(upstream)是包含端口的,通过hash计算得到的值也不一样。