kubernetes安装详细步骤

步骤"/>

kubernetes安装详细步骤

一：简介

二：基础环境安装

1.系统环境

os	Role	ip	Memory
Centos 7	master01	192.168.25.30	4G
Centos 7	node01	192.168.25.31	4G
Centos 7	node02	192.168.25.31	4G

2.关闭selinux

 

sed -i "s/SELINUX\=.*/SELINUX=disabled/g" /etc/selinux/config

3.关闭防火墙

 

systemctl disable firewalld && systemctl stop firewalld 

4.修改主机名

 

hostnamectl set-hostname Role_name

5.添加hosts解析

 

echo -e "192.168.25.30 master01\n192.168.25.31 node01\n192.168.25.32 node02" >> /etc/hosts

6.设置k8s内核参数

设置内核参数

 

cat << EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF

加载内核模块

 

modprobe br_netfilter
echo "modprobe br_netfilter" >> /etc/rc.local

使内核参数生效

 

sysctl -p /etc/sysctl.d/k8s.conf

7.关闭系统swap

 

 swapoff -a

修改fstab文件，关闭swap的自动挂载。

8.修改防火墙策略

 

/sbin/iptables -P FORWARD ACCEPT
echo  "sleep 60 && /sbin/iptables -P FORWARD ACCEPT" >> /etc/rc.local

9.安装依赖包

 

yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget

10.时间同步

 

yum -y install ntpdate
/usr/sbin/ntpdate -u ntpserver1: ntp1.aliyun
/usr/sbin/ntpdate -u ntp1.aliyun

11.安装docker-ce软件

提示：master节点不需要安装

1. 删除自带的docker

   yum remove docker \docker-client \docker-client-latest \docker-common \docker-latest \docker-latest-logrotate \docker-logrotate \docker-selinux \docker-engine-selinux \docker-engine
   

2. 安装依赖包

   yum install -y yum-utils \device-mapper-persistent-data \lvm2
   

3. 安装yum源

   yum-config-manager --add-repo .repo
   

4. 安装docker-ce

   yum -y install docker-ce
   

5. 启动，并设置开机自启[安装设置好flanneld后，再启动docker]

   systemctl start docker && systemctl enable docker
   

11.安装CFSSL

cfssl

 

export CFSSL_URL=".2"
wget "${CFSSL_URL}/cfssl_linux-amd64" -O /usr/local/bin/cfssl
wget "${CFSSL_URL}/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

三：创建CA证书和密钥

​ kubernetes系统各组件需要使用TLS证书对通信进行加密，本本档使用CloudFlare的工具集cfssl来生成Certificate Authority(CA)证书和密钥文件，CA是自签名的证书，用来签名后续创建的其他TLS证书。

​ 以下操作都在master节点上执行，证书只需要创建一次即可，以后新增节点时，只需要将/etc/kubernetes/目录下的证书拷贝到新节点即可。

1.创建CA配置文件

 

mkdir /root/ssl
cd /root/sslcat > ca-config.json << EOF
{"signing": {"default": {"expiry": "8760h"},"profiles": {"kubernetes": {"usages": ["signing","key encipherment","server auth","client auth"],"expiry": "8760h"}}}
}
EOF

• ca-config.json：可以定义多个profiles，分别指定不同的过期时间，使用场景等参数，后续在签名证书时会使用到某个profile；
• signing：表示该证书可用于签名其他证书；生成ca.pem证书中的CA=TRUE；
• server auto：表示client可以用该CA对server提供的证书进行验证；
• client auth：表示server可以用该CA对client提供的证书进行验证

2.创建CA证书签名请求

 

cat > ca-csr.json << EOF
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "System"}]
}
EOF

• "CN"：Common Name，kube-apiserver从证书中提取该字段作为请求的用户名(User name)；浏览器检验该字段验证网站是否合法；
• “O”：Organization，kube-apiserver从证书提取该字段作为请求用户所属的组（Group）；

3.生成CA证书和私钥

 

# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2018/03/29 14:38:31 [INFO] generating a new CA key and certificate from CSR
2018/03/29 14:38:31 [INFO] generate received request
2018/03/29 14:38:31 [INFO] received CSR
2018/03/29 14:38:31 [INFO] generating key: rsa-2048
2018/03/29 14:38:31 [INFO] encoded CSR
2018/03/29 14:38:31 [INFO] signed certificate with serial number 438768005817886692243142700194592359153651905696

4.创建kubernetes证书签名请求文件

 

cat > kubernetes-csr.json << EOF
{"CN": "kubernetes","hosts": ["127.0.0.1","192.168.25.30","192.168.25.31","192.168.25.32","10.254.0.1","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "System"}]
}
EOF

• hosts中的内容可以为空，即是按照上面的配置，向集群中增加新节点也不需要重新生成证书；如果hosts不为空，则需要指定授权使用该证书的IP或域名列表，由于该证书后续被etcd集群和kubernetes master集群使用，所以上面分别指定了etcd集群，kubernetes master集群的主机IP和kuberunetes服务ip。

5.生成kubernetes证书和私钥

 

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
2018/03/29 14:46:12 [INFO] generate received request
2018/03/29 14:46:12 [INFO] received CSR
2018/03/29 14:46:12 [INFO] generating key: rsa-2048
2018/03/29 14:46:12 [INFO] encoded CSR
2018/03/29 14:46:12 [INFO] signed certificate with serial number 6955479006214073693226115919937339031303355422
2018/03/29 14:46:12 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum ();
specifically, section 10.2.3 ("Information Requirements").# ls kubernetes*
kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem

6.创建admin证书签名请求文件

 

cat > admin-csr.json << EOF
{"CN": "admin","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "system:masters","OU": "System"}]
}
EOF

• kube-apiserver使用RBAC对客户端（如Kubelet,kube-proxy,Pod）请求进行授权。
• kube-apiserver预定义了一些RBAC使用的RoleBindings，如cluster-admin将Group System：masters与Role cluster-admin绑定，该Role授予kube-apiserver的所有API的权限；
• OU指定该证书的Group为system:masters，kubelet使用该证书访问kube-apiserver时，由于证书为CA签名，所以认证通过，同时由于证书用户组为经过预授权的system：masters，所以被授予访问所有API的权限

7.生成admin证书和私钥

 

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
2018/03/29 14:57:01 [INFO] generate received request
2018/03/29 14:57:01 [INFO] received CSR
2018/03/29 14:57:01 [INFO] generating key: rsa-2048
2018/03/29 14:57:02 [INFO] encoded CSR
2018/03/29 14:57:02 [INFO] signed certificate with serial number 356467939883849041935828635530693821955945645537
2018/03/29 14:57:02 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum ();
specifically, section 10.2.3 ("Information Requirements")# ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem

8.创建kube-proxy证书签名请求文件

 

cat > kube-proxy-csr.json << EOF
{"CN": "system:kube-proxy","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "System"}]
}
EOF

• CN指定该证书的User为system：kube-proxy；
• kube-apiserver预定义的RoleBinding cluster-admin将User system：kube-proxy与Role System：node-proxies绑定，该Role授予了调用kube-apiserver Proxy相关API的权限；

9.生成kube-proxy客户端证书和私钥

 

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy
2018/03/29 15:09:36 [INFO] generate received request
2018/03/29 15:09:36 [INFO] received CSR
2018/03/29 15:09:36 [INFO] generating key: rsa-2048
2018/03/29 15:09:36 [INFO] encoded CSR
2018/03/29 15:09:36 [INFO] signed certificate with serial number 225974417080991591210780916866547658424323006961
2018/03/29 15:09:36 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum ();
specifically, section 10.2.3 ("Information Requirements").# ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

10.证书分发

将生成的证书和密钥文件（后缀为pem）拷贝到所有机器的/etc/kubernetes/ssl目录下

 

mkdir -p /etc/kubernetes/ssl
cp *.pem /etc/kubernetes/sslssh node01 "mkdir -p /etc/kubernetes/ssl"
scp *.pem node01:/etc/kubernetes/sslssh node02 "mkdir -p /etc/kubernetes/ssl"
scp *.pem node02:/etc/kubernetes/ssl

四：部署Etcd集群

​ 在三个节点都需要安装etcd，下面的操作在每台机器上操作一遍。

1.下载etcd安装包并生成命令

 

wget .2.12/etcd-v3.2.12-linux-amd64.tar.gz
tar -xvf etcd-v3.2.12-linux-amd64.tar.gz
mv etcd-v3.2.12-linux-amd64/etcd* /usr/local/bin# 生成以下两条命令
# etcd
etcd     etcdctl 

2.创建工作目录

 

mkdir -p /var/lib/etcd

3.创建系统服务文件

master01

 

cat > etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \\--name master01 \\--cert-file=/etc/kubernetes/ssl/kubernetes.pem \\--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\--peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \\--peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\--initial-advertise-peer-urls https://192.168.25.30:2380 \\--listen-peer-urls https://192.168.25.30:2380 \\--listen-client-urls https://192.168.25.30:2379,http://127.0.0.1:2379 \\--advertise-client-urls https://192.168.25.30:2379 \\--initial-cluster-token etcd-cluster-0 \\--initial-cluster master01=https://192.168.25.30:2380,node01=https://192.168.25.31:2380,node02=https://192.168.25.32:2380 \\--initial-cluster-state new \\--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF

node01

 

cat > etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \\--name node01 \\--cert-file=/etc/kubernetes/ssl/kubernetes.pem \\--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\--peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \\--peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\--initial-advertise-peer-urls https://192.168.25.31:2380 \\--listen-peer-urls https://192.168.25.31:2380 \\--listen-client-urls https://192.168.25.31:2379,http://127.0.0.1:2379 \\--advertise-client-urls https://192.168.25.31:2379 \\--initial-cluster-token etcd-cluster-0 \\--initial-cluster master01=https://192.168.25.30:2380,node01=https://192.168.25.31:2380,node02=https://192.168.25.32:2380 \\--initial-cluster-state new \\--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF

node02

 

cat > etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \\--name node02 \\--cert-file=/etc/kubernetes/ssl/kubernetes.pem \\--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\--peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \\--peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\--initial-advertise-peer-urls https://192.168.25.32:2380 \\--listen-peer-urls https://192.168.25.32:2380 \\--listen-client-urls https://192.168.25.32:2379,http://127.0.0.1:2379 \\--advertise-client-urls https://192.168.25.32:2379 \\--initial-cluster-token etcd-cluster-0 \\--initial-cluster master01=https://192.168.25.30:2380,node01=https://192.168.25.31:2380,node02=https://192.168.25.32:2380 \\--initial-cluster-state new \\--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF

• 指定etcd的工作目录为/var/lib/etcd，数据目录为/var/lib/etcd，需在启动服务前创建这个目录，否则启动服务的时候会报错“Failed at step CHDIR spawning /usr/bin/etcd: No such file or directory”；
• 为了保证通信安全，需要指定etcd的公私钥(cert-file和key-file),Peers通信的公私钥和CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书（trusted-ca-file）；
• 创建kubernetes.pem证书时使用的kubernestes-csr.json文件的hosts字段包含所有的etcd节点的IP，否则证书校验会出错；
• --initial-cluster-state值为new时，-name的参数值必须位于-initial-cluster列表中；

4.启动etcd服务

 

cp etcd.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

5.验证etcd服务

 

# etcdctl \--ca-file=/etc/kubernetes/ssl/ca.pem \--cert-file=/etc/kubernetes/ssl/kubernetes.pem \--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \cluster-health
member 2ea4d6efe7f32da is healthy: got healthy result from https://192.168.25.32:2379
member 5246473f59267039 is healthy: got healthy result from https://192.168.25.31:2379
member be723b813b44392b is healthy: got healthy result from https://192.168.25.30:2379
cluster is healthy

五：部署Flannel

在node节点上都需要部署安装

1.下载安装Flannel

 

wget .9.1/flannel-v0.9.1-linux-amd64.tar.gz
mkdir flannel
tar -xzvf flannel-v0.9.1-linux-amd64.tar.gz -C flannel
cp flannel/{flanneld,mk-docker-opts.sh} /usr/local/bin

2.向etcd中写入网段信息，只需要在一台执行即可

 

etcdctl --endpoints=https://192.168.25.30:2379,https://192.168.25.31:2379,https://192.168.25.32:2379 \--ca-file=/etc/kubernetes/ssl/ca.pem \--cert-file=/etc/kubernetes/ssl/kubernetes.pem \--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \mkdir /kubernetes/networketcdctl --endpoints=https://192.168.25.30:2379,https://192.168.25.31:2379,https://192.168.25.32:2379 \--ca-file=/etc/kubernetes/ssl/ca.pem \--cert-file=/etc/kubernetes/ssl/kubernetes.pem \--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \mk /kubernetes/network/config '{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}'

3.创建服务启动文件

 

cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service[Service]
Type=notify
ExecStart=/usr/local/bin/flanneld \\-etcd-cafile=/etc/kubernetes/ssl/ca.pem \\-etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \\-etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \\-etcd-endpoints=https://192.168.25.30:2379,https://192.168.25.31:2379,https://192.168.25.32:2379 \\-etcd-prefix=/kubernetes/network
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

• mk-docker-opts.sh：将分配给flanneld的pod子网网段信息写入到/run/flannel/docker文件中，后续docker启动时使用这个文件中参数值设置docker0网桥
• flanneld使用系统缺省路由所在的接口和其他节点通信，对于有多个网络接口的机器（如内网和公网），可使用-iface=enpxx选项值指定通信接口；

4.启动Flanneld服务

 

mv flanneld.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
systemctl status flanneld

5.检查flanneld服务状态

 

# /usr/local/bin/etcdctl \--endpoints=https://192.168.25.30:2379,https://192.168.25.31:2379,https://192.168.25.32:2379 \--ca-file=/etc/kubernetes/ssl/ca.pem \--cert-file=/etc/kubernetes/ssl/kubernetes.pem \--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ls /kubernetes/network/subnets/kubernetes/network/subnets/172.30.82.0-24
/kubernetes/network/subnets/172.30.1.0-24
/kubernetes/network/subnets/172.30.73.0-24

6.配置docker使用flanneld网络

/usr/lib/systemd/system/docker.service

 

[Unit]
Description=Docker Application Container Engine
Documentation=
After=network-online.target firewalld.service
Wants=network-online.target[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
# 修改
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
# 新增
EnvironmentFile=/run/flannel/dockerExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s[Install]
WantedBy=multi-user.target

• flanneld 启动时将网络配置写入到 /run/flannel/docker 文件中的变量 DOCKER_NETWORK_OPTIONS，dockerd 命令行上指定该变量值来设置 docker0 网桥参数；
• 如果指定了多个 EnvironmentFile 选项，则必须将 /run/flannel/docker 放在最后(确保 docker0 使用 flanneld 生成的 bip 参数)；
• 不能关闭默认开启的 –iptables 和 –ip-masq 选项；
• 如果内核版本比较新，建议使用 overlay 存储驱动；
• –exec-opt native.cgroupdriver=systemd参数可以指定为”cgroupfs”或者“systemd”

7.启动docker

 

systemctl daemon-reload && systemctl start docker && systemctl enable docker

六：部署kubectl工具

​ kubectl是kubernetes的集群管理工具，任何节点通过kubetcl都可以管理整个k8s集群。本文档部署在master01这个节点，部署成功后会生成/root/.kube/config文件，kubectl就是通过这个获取kube-apiserver地址，证书，用户名等信息。

1.下载安装包

 

wget .8.6/kubernetes-client-linux-amd64.tar.gz
tar -xzvf kubernetes-client-linux-amd64.tar.gz
sudo cp kubernetes/client/bin/kube* /usr/local/bin/
chmod a+x /usr/local/bin/kube*
export PATH=/root/local/bin:$PATH

2.创建/root/.kube/config文件

 

# 设置集群参数,--server指定Master节点ip
kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/ssl/ca.pem \--embed-certs=true \--server=https://192.168.25.30:6443# 设置客户端认证参数
kubectl config set-credentials admin \--client-certificate=/etc/kubernetes/ssl/admin.pem \--embed-certs=true \--client-key=/etc/kubernetes/ssl/admin-key.pem# 设置上下文参数
kubectl config set-context kubernetes \--cluster=kubernetes \--user=admin# 设置默认上下文
kubectl config use-context kubernetes

• admin.pem：证书O字段值为system:masters，kube-apiserver预定义的RoleBinding cluster-admin将Group system:master与Role cluster-admin绑定，该Role 授予了调用Kube-apiserver相关的API权限

3.创建bootstartp.kubeconfig文件

 

#生成token 变量
export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOFmv token.csv /etc/kubernetes/# 设置集群参数--server为master节点ip
kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/ssl/ca.pem \--embed-certs=true \--server=https://192.168.25.30:6443 \--kubeconfig=bootstrap.kubeconfig# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \--token=${BOOTSTRAP_TOKEN} \--kubeconfig=bootstrap.kubeconfig# 设置上下文参数
kubectl config set-context default \--cluster=kubernetes \--user=kubelet-bootstrap \--kubeconfig=bootstrap.kubeconfig# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfigmv bootstrap.kubeconfig /etc/kubernetes/

4.创建kube-proxy.kubeconfig

 

# 设置集群参数 --server参数为master ip
kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/ssl/ca.pem \--embed-certs=true \--server=https://192.168.25.30:6443 \--kubeconfig=kube-proxy.kubeconfig# 设置客户端认证参数
kubectl config set-credentials kube-proxy \--client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \--client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \--embed-certs=true \--kubeconfig=kube-proxy.kubeconfig# 设置上下文参数
kubectl config set-context default \--cluster=kubernetes \--user=kube-proxy \--kubeconfig=kube-proxy.kubeconfig# 设置默认上下文
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
mv kube-proxy.kubeconfig /etc/kubernetes/

• 设置集群参数和客户端认证参数，--embed-certs都为true，这会将certificate-authority，client-cretificate和client-key指向的证书文件内容写入到生成的kube-proxy.kebuconfig文件中；
• kube-proxy.pem证书中CN为system:kube-proxy,kube-apiserver预定义的RoleBinding cluster-admin将User system:kube-proxy与Role system:node-proxy绑定，该Role授予了调用kube-apiserver Proxy相关的api权限；

5.将生成的配置文件拷贝到其他的节点

 

scp /etc/kubernetes/kube-proxy.kubeconfig node01:/etc/kubernetes/
scp /etc/kubernetes/kube-proxy.kubeconfig node02:/etc/kubernetes/scp /etc/kubernetes/bootstrap.kubeconfig node01:/etc/kubernetes/
scp /etc/kubernetes/bootstrap.kubeconfig node02:/etc/kubernetes/

七：部署master节点

1.下载安装文件

 

wget .8.6/kubernetes-server-linux-amd64.tar.gz
tar -xzvf kubernetes-server-linux-amd64.tar.gz
cp -r kubernetes/server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/

2.部署apiserver服务

配置kube-apiserver服务管理文件

 

cat > kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=
After=network.target
After=etcd.service[Service]
ExecStart=/usr/local/bin/kube-apiserver \\--logtostderr=true \\--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \\--advertise-address=192.168.25.30 \\--bind-address=192.168.25.30 \\--insecure-bind-address=127.0.0.1 \\--authorization-mode=Node,RBAC \\--runtime-config=rbac.authorization.k8s.io/v1alpha1 \\--kubelet-https=true \\--enable-bootstrap-token-auth \\--token-auth-file=/etc/kubernetes/token.csv \\--service-cluster-ip-range=10.254.0.0/16 \\--service-node-port-range=8400-10000 \\--tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem \\--tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \\--client-ca-file=/etc/kubernetes/ssl/ca.pem \\--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \\--etcd-cafile=/etc/kubernetes/ssl/ca.pem \\--etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem \\--etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem \\--etcd-servers=https://192.168.25.30:2379,https://192.168.25.31:2379,https://192.168.25.32:2379 \\--enable-swagger-ui=true \\--allow-privileged=true \\--apiserver-count=3 \\--audit-log-maxage=30 \\--audit-log-maxbackup=3 \\--audit-log-maxsize=100 \\--audit-log-path=/var/lib/audit.log \\--event-ttl=1h \\--v=2
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF

• --authorization-mode=RBAC 指定在安全端口使用RBAC模式，拒绝未通过授权的请求；
• kube-scheduler，kube-controller-manager一般和kube-apiserver部署在同一台机器上，他们使用非安全端口和kube-apiserver通信；
• kubelet，kube-proxy，kubectl部署在其他Node节点，如果通过安全端口访问kube-apiserver，则必须先通过TLS证书认证，再通过RBAC授权；
• kube-proxy，kubectl通过在使用的证书里指定相关的User，Group来达到通过RBAC授权的目的。
• Bootstartp：如果使用了kubelet TLS Bootstartp机制，则不能再指定 –kubelet-certificate-authority、–kubelet-client-certificate 和 –kubelet-client-key 选项，否则后续kube-apiserver校验kubelet证书时出现”x509: certificate signed by unknown authority“ 错误；
• --admission-control值必须包含ServiceAccount，否则部署集群插件时会失败；
• --bind-address不能为127.0.0.1；
• --runtime-config：配置rbac.authorization.k8s.io/v1beta1，表示运行时的apiVersion；
• service-cluster-ip-range：指定Service cluster ip段地址，该地址路由不可达；
• --service-node-port-range：指定NodePort的端口范围

确实情况下，kubernetes对像保存在etcd的/registry路径下，可以通过--etcd-prefix参数进行跳转

启动服务，并设置开启自启

 

cp kube-apiserver.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver

3.部署manager服务

生成服务启动脚本

 

cat > kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=[Service]
ExecStart=/usr/local/bin/kube-controller-manager \\--logtostderr=true  \\--address=127.0.0.1 \\--master=http://127.0.0.1:8080 \\--allocate-node-cidrs=true \\--service-cluster-ip-range=10.254.0.0/16 \\--cluster-cidr=172.30.0.0/16 \\--cluster-name=kubernetes \\--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \\--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \\--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \\--root-ca-file=/etc/kubernetes/ssl/ca.pem \\--leader-elect=true \\--v=2
Restart=on-failure
LimitNOFILE=65536
RestartSec=5[Install]
WantedBy=multi-user.target
EOF

• --address的值必须为127.0.0.1,应为当前kube-apiserver期望scheduler和conntroller-manaager在同一台机器上；
• --master=http://{master_ip}:8080：使用非安全的8080端口与kube-apiserver通信；
• --cluster-cidr指定Cluster中Pod的CIDR范围，该网段在各Node必须路由可达（flanneld保证）
• --service-cluster-ip-range参数指定Cluster中Service的CIDR范围，该网络在各Node间必须路由不可达，必须与kube-apiserver中的参数保持一致；
• --cluster-signing-*指定的证书和私钥文件用来签名TLS BootStrap创建的证书和私钥
• --root-ca-file用来对kube-apiserver证书进行校验，指定该参数后，才会在Pod容器的ServiceAccount中放置该CA证书文件
• --leader-elect=true部署多台master集群时选举产生一直处于工作状态的kube-controller-manager进程；

启动服务

 

cp kube-controller-manager.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl start kube-controller-manager
systemctl status kube-controller-manage

4.部署scheduler服务

配置kube-scheduler

 

cat > kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=[Service]
ExecStart=/usr/local/bin/kube-scheduler \\--logtostderr=true \\--address=127.0.0.1 \\--master=http://127.0.0.1:8080 \\--leader-elect=true \\--v=2
Restart=on-failure
LimitNOFILE=65536
RestartSec=5[Install]
WantedBy=multi-user.target
EOF

• --address必须为127.0.0.1，应为当前kube-apiserver期望scheduler和contorller-manager在同一主机；
• master=http://{MASTER_IP}:8080：使用非安全 8080 端口与 kube-apiserver 通信；
• –leader-elect=true 部署多台机器组成的 master 集群时选举产生一处于工作状态的 kube-controller-manager 进程；

启动kube-scheduler

 

cp kube-scheduler.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable kube-scheduler
systemctl start kube-scheduler
systemctl status kube-scheduler

5.验证master节点

 

# kubectl get componentstatuses
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-2               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
etcd-0               Healthy   {"health": "true"} 

八:部署Node节点

1.部署kubelet服务

​ kubelet在启动时向kube-apiserver发送TLS bootstrapping请求，需要先将bootstrap token文件中的kubelet-bootstrap用户赋予system:node-bootstrapper角色，然后kubelet才有权限创建认证请求。

授权,在master上运行一次即可

 

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

下载和安装kubelet和kube-proxy

 

wget .8.6/kubernetes-server-linux-amd64.tar.gz
tar -xzvf kubernetes-server-linux-amd64.tar.gz
cp -r kubernetes/server/bin/{kube-proxy,kubelet} /usr/local/bin/

创建kubelet工作目录

 

mkdir /var/lib/kubelet

配置kubelt

master01

 

cat > kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=
After=docker.service
Requires=docker.service[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \\--address=192.168.25.30 \\    --hostname-override=192.168.25.30 \\--pod-infra-container-image=registry.access.redhat/rhel7/pod-infrastructure:latest \\--experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \\--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\--require-kubeconfig \\--cert-dir=/etc/kubernetes/ssl \\--container-runtime=docker \\--cluster-dns=10.254.0.2 \\--cluster-domain=cluster.local \\--hairpin-mode promiscuous-bridge \\--allow-privileged=true \\--serialize-image-pulls=false \\--register-node=true \\--logtostderr=true \\--cgroup-driver=cgroupfs  \\--v=2Restart=on-failure
KillMode=process
LimitNOFILE=65536
RestartSec=5[Install]
WantedBy=multi-user.target
EOF

• --address：本机IP，不能设置为127.0.0.1，否则后续Pods访问kubelet的API接口时会失败，因为 Pods 访问的 127.0.0.1 指向自己，而不是 kubelet;
• --hostname-overeide：本机IP；
• --cgroup-driver配置成cgroupfs（保持docker和kubelet中的cgroup driver配置一致即可）；
• --experimental-bootstrap-kubeconfig指向bootstrap kubeconfig文件，kubelet使用该文件中的用户名和token向kube-apiserver发送TLS Bootstrapping请求；
• 管理员通过了CSR请求后，kubelet自动在--cert-dir目录创建证书和私钥文件（kubelet-client.crt和kubelet-client.key）,然后写入--kubeconfig文件（自动创建）
• 建议在--kubeconfig配置文件中指定kube-apiserver地址，如果未指定--api-servers选项，则必须指定--require-kubeconfig选项后才从配置文件中读取kube-apiserver的地址，否则kubelet启动后会找不到kube-apiserver（日志中提示找不到API server），kubectl get nodes不会返回对应的Node信息；
• --cluster-dns指定kubedns的Service ip(可以先分配，后续创建kubedns服务时指定该IP)，--cluster-domain指定域名后缀，这两个参数同时配置才会生效；
• --cluster-domain指定pod启动时/etc/resolve.conf文件中的search domain，起初我们将其配置成了 cluster.local.，这样在解析 service 的 DNS 名称时是正常的，可是在解析 headless service 中的 FQDN pod name 的时候却错误，因此我们将其修改为 cluster.local，去掉嘴后面的 ”点号“ 就可以解决该问题；
• --kubeconfig=/etc/kubernetes/kubelet.kubeconfig中指定的kubelet.kubeconfig文件在第一次启动kubelet之前并不存在，请看下文，当通过CSR请求后，会自动生成，如果你的节点节点上已经生成了~/.kube/config文件，你可以将该文件拷贝到该路径面，并命名为kubelet.kubeconfig文件，所有的节点可以共用同一个config文件，这样新添加节点时就不需要再创建CSR请求就能自动添加到kubernetes集群中，同样，在任意能够访问到kubernetes集群的主机上使用kubectl --kubeconfig命令操作集群，只要使用~/.kube/config文件就能通过权限认证，应为这个文件的认证信息为admin，对集群有所有权限。

启动kubelet服务

 

cp kubelet.service /etc/systemd/system/kubelet.service
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

2.执行TLS证书授权请求

kubelet首次启动时像kube-apiserver发送证书签名求情，必须通过授权后，才会添加到集群。

查询授权请求

 

# kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-450A0zCMYrGxWozsNukv6vh2NdBspA-hr6Rsz-LA9ro   3m        kubelet-bootstrap   Pending
node-csr-5t_AUkaEhT98xX1g7zTpzaNzRB9rXh453i2Fu_yxvvs   3m        kubelet-bootstrap   Pending
node-csr-p9r9gusX2kTGpyYFPlkoaSGyatLQmtDmL8NBee2D_s8   3m        kubelet-bootstrap   Pending

同意授权请求

 

# kubectl certificate approve node-csr-450A0zCMYrGxWozsNukv6vh2NdBspA-hr6Rsz-LA9ro
certificatesigningrequest "node-csr-450A0zCMYrGxWozsNukv6vh2NdBspA-hr6Rsz-LA9ro" approved# kubectl certificate approve node-csr-5t_AUkaEhT98xX1g7zTpzaNzRB9rXh453i2Fu_yxvvs
certificatesigningrequest "node-csr-5t_AUkaEhT98xX1g7zTpzaNzRB9rXh453i2Fu_yxvvs" approved# kubectl certificate approve node-csr-p9r9gusX2kTGpyYFPlkoaSGyatLQmtDmL8NBee2D_s8
certificatesigningrequest "node-csr-p9r9gusX2kTGpyYFPlkoaSGyatLQmtDmL8NBee2D_s8" approved

查看所有集群节点

 

# kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
192.168.25.30   Ready     <none>    15m       v1.8.6
192.168.25.31   Ready     <none>    15m       v1.8.6
192.168.25.32   Ready     <none>    15m       v1.8.6

3.部署kube-proxy服务

创建工作目录

 

mkdir -p /var/lib/kube-proxy

配置kube-proxy服务

 

cat > kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=
After=network.target[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \\--bind-address=192.168.25.30 \\--hostname-override=192.168.25.30 \\--cluster-cidr=10.254.0.0/16 \\--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \\--logtostderr=true \\--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF

• --bind-address：本机ip
• --hostname-override：本机ip，必须需kubelet的值一致，否则kube-proxy启动后找不到node，从而影响任何iptables规则
• --cluster-cidr：必须与kube-apiserver的--service-cluster-ip-range选项值一致，kube-proxy根据--clister-cidr判断集群内部和外部流量，指定--cluster-cidr或--masquerade-all选项后kube-proxy才会对访问service ip的请求做SNAT；
• –kubeconfig 指定的配置文件嵌入了 kube-apiserver 的地址、用户名、证书、秘钥等请求和认证信息；
• 预定义的 RoleBinding cluster-admin 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限；

启动kube-proxy服务

 

cp kube-proxy.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy

九:插件安装

由于默认镜像为谷歌镜像，所以是需要修改的，所以用docker hup做了跳转，修改好的yamk文件下载地址如下：

百度网盘(o3z9)

1.dns插件

 

wget .8.6/kubernetes.tar.gz
tar xzvf kubernetes.tar.gzcd /root/kubernetes/cluster/addons/dns
mv  kubedns-svc.yaml.sed kubedns-svc.yaml
#把文件中$DNS_SERVER_IP替换成10.254.0.2
sed -i 's/$DNS_SERVER_IP/10.254.0.2/g' ./kubedns-svc.yamlmv ./kubedns-controller.yaml.sed ./kubedns-controller.yaml
#把$DNS_DOMAIN替换成cluster.local
sed -i 's/$DNS_DOMAIN/cluster.local/g' ./kubedns-controller.yamlls *.yaml
kubedns-cm.yaml  kubedns-controller.yaml  kubedns-sa.yaml  kubedns-svc.yamlkubectl create -f .

2.dashboard插件

下载部署文件

 

wget .8.1/src/deploy/recommended/kubernetes-dashboard.yaml

修改部署文件

 

kind: Service
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kube-system
spec:# 新增type: NodePortports:- port: 443targetPort: 8443# 新增nodePort: 8510selector:k8s-app: kubernetes-dashboard

创建pod

 

kubectl create -f kubernetes-dashboard.yaml

部署认证服务

 

cat > ./kubernetes-dashboard-admin.rbac.yaml << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:name: dashboard-admin
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: cluster-admin
subjects:
- kind: ServiceAccountname: kubernetes-dashboardnamespace: kube-system
EOFkubectl create -f kubernetes-dashboard-admin.rbac.yaml 

访问地址，目前只能火狐访问

 

https://192.168.25.30:8510

3.heapster插件

下载安装文件

 

wget .5.0.tar.gz
tar xzvf ./v1.5.0.tar.gz
cd ./heapster-1.5.0/kubectl create -f deploy/kube-config/influxdb/
kubectl create -f deploy/kube-config/rbac/heapster-rbac.yaml123456

确认所有pod都正常启动

 

kubectl get pods --all-namespaces

十：常用服务部署

1.nginx

部署文件：

 

apiVersion: v1
kind: Pod
metadata:name: nginxlabels:app: nginx
spec:containers:- name: nginximage: registry-qingdao.aliyuncs/k8/nginx:1.9.0imagePullPolicy: IfNotPresentports:- containerPort: 80restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:name: nginx-service
spec:type: NodePortsessionAffinity: ClientIPselector:app: nginxports:# 将容器的80端口映射到master主机的8888端口- port: 80nodePort: 8888

2.mysql

部署文件：

 

apiVersion: v1
kind: Pod
metadata:name: mysqllabels:app: mysql
spec:containers:- name: mysqlimage: mysql# 环境变量env:- name: MYSQL_ROOT_PASSWORDvalue: "123456"imagePullPolicy: IfNotPresent# 容器暴露端口ports:- containerPort: 3306restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:name: mysql-service
spec:type: NodePortsessionAffinity: ClientIPselector:app: mysqlports:- port: 3306nodePort: 9306

十一：常用命令

1.查看kubelet log

 

journalctl -u kubelet -f

• u：执行用户
• f：动态查看

2.查看pods信息

 

kubectl get pods --all-namespaces

3.查看service信息

 

kubectl get service --all-namespaces

4.查看server详细信息

 

kubeclt describe service zksvc

作者：Goun
链接：
来源：简书
著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。