K8S1.14高可用生产集群部署方案

栏目: 编程工具 · 发布时间: 5年前

内容简介：架构说明：安装好CentOS7的系统，做以下操作：

系统说明

系统组件版本

操作系统: CentOS 7.6
Kernel: 4.4
Kubernetes: v1.14.1
Docker: 18.09 (支持 1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09 )
Etcd: v3.3.12
Flannel: v0.11
cni-plugins: v0.7.5
CoreDNS: 1.4.0

架构示意图

K8S1.14高可用生产集群部署方案

架构说明：

使用六台主机，三台Master节点, 三台node节点
Master节点上部署的Kubernetes组件有kube-apiserver, kube-scheduler, kube-controller-manager,kube-proxy。部署网络组件flannel,数据存储集群Etcd.
Master的两个高可用节点部署Haproxy和keepalived
Node 节点部署的Kubernetes组件有Kubelet,kube-proxy。容器组件Docker,网络组件Flannel
集群IP和主机名信息：

集群角色	主机名	IP
Master	master-1	192.168.20.44
Master	master-2	192.168.20.45
Master	master-3	192.168.20.46
Node	k8s-node-1	192.168.20.47
Node	k8s-node-2	192.168.20.48
Node	k8s-node-3	192.168.20.49

Ceph 需要一个可用的Ceph集群

系统初始化

1. 主机初始化

安装好CentOS7的系统，做以下操作：

yum update

设置内核参数的部分，要确认执行如下操作：

# 高可用Master节点设置内核参数
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.ipv4.ip_nonlocal_bind = 1    
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_port_range = 10000 65000
fs.file-max = 2000000
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF

# 其他Master节点和计算节点设置内核参数
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_port_range = 10000 65000
fs.file-max = 2000000
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF

2. 安装Docker

由于在Kubernetes1.14的版本中，支持的版本有 1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09，所以这里统一使用 Docker 18.09。

下载 docker-ce-18.09 的rpm包和阿里云上对应的 docker-ce.repo 的源，在所有node节点上直接安装：

mv docker-ce.repo /etc/yum.repos.d/
yum install docker-ce-18.09.5-3.el7.x86_64.rpm -y

在所有node节点启动docker,并配置自启动：

systemctl start docker
systemctl enable docker

3. 创建目录

在所有主机上执行如下命令，创建所需目录：

mkdir -p /opt/kubernetes/{cfg,bin,ssl,log}

4. 准备Kubernetes软件包

从github下载kubernetes 1.14版本的二进制软件包，下载地址: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#server-binaries

下载如下软件包：

[root@master-1 tmp]# ll
total 537520
-rw-r--r-- 1 root root 113938518 Jul 24 19:15 kubernetes-node-linux-amd64.tar.gz
-rw-r--r-- 1 root root 433740362 Jul 24 19:09 kubernetes-server-linux-amd64.tar.gz

解压：

tar xf kubernetes-server-linux-amd64.tar.gz

5.准备etcd和flannel组件

从github下载etcd v3.3.12和flannel v0.11.0 :

wget https://github.com/etcd-io/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz
wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz

制作CA证书

Kubernetes支持使用多种方式生成证书，可以使用easyrsa, openssl 或 cfssl任一一种生成。

参考链接

这里使用cfssl创建CA证书。

1.安装CFSSL

使用cfssl生成CA证书需要单独安装cfssl。

[root@master-1 ~]# cd /usr/local/src/

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /opt/kubernetes/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /opt/kubernetes/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /opt/kubernetes/bin/cfssl-certinfo
chmod +x /opt/kubernetes/bin/*

所有节点添加Kubernetes的bin目录到系统环境变量：

echo 'PATH=$PATH:/opt/kubernetes/bin' >>/etc/profile
source /etc/profile

2.生成证书

创建需要的配置文件：

[root@master-1 ~]# cd /opt/kubernetes/ssl/
[root@master-1 ssl]# cfssl  print-defaults config > config.json
[root@master-1 ssl]# cfssl print-defaults csr > csr.json
[root@master-1 ssl]# ll
total 8
-rw-r--r-- 1 root root 567 Jul 26 00:05 config.json
-rw-r--r-- 1 root root 287 Jul 26 00:05 csr.json
[root@master-1 ssl]# mv config.json ca-config.json
[root@master-1 ssl]# mv csr.json  ca-csr.json

修改生成的文件为如下内容：

ca-config.json文件：

[root@master-1 ssl]# vim ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

ca-csr.json文件：

[root@master-1 ssl]# vim ca-csr.json 
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
          "C": "CN",
          "ST": "BeiJing",
          "L": "BeiJing",
          "O": "k8s",
          "OU": "System"
        }
    ]
}

生成证书（ca-key.pem）和秘钥(ca.pem):

[root@master-1 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2018/07/26 00:27:00 [INFO] generating a new CA key and certificate from CSR
2018/07/26 00:27:00 [INFO] generate received request
2018/07/26 00:27:00 [INFO] received CSR
2018/07/26 00:27:00 [INFO] generating key: rsa-2048
2018/07/26 00:27:01 [INFO] encoded CSR
2018/07/26 00:27:01 [INFO] signed certificate with serial number 479065525331838190845576195908271097044538206777
[root@master-1 ssl]# ll
total 20
-rw-r--r-- 1 root root  386 Jul 26 00:16 ca-config.json
-rw-r--r-- 1 root root 1001 Jul 26 00:27 ca.csr
-rw-r--r-- 1 root root  255 Jul 26 00:20 ca-csr.json
-rw------- 1 root root 1679 Jul 26 00:27 ca-key.pem
-rw-r--r-- 1 root root 1359 Jul 26 00:27 ca.pem

分发证书到各个节点：

[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.45:/opt/kubernetes/ssl
[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.46:/opt/kubernetes/ssl
[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.47:/opt/kubernetes/ssl
[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.48:/opt/kubernetes/ssl
[root@master-1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.20.49:/opt/kubernetes/ssl

HA 节点部署

这里选择两个Master节点部署Haproxy 和 keepalived， keepalived上需要添加监控haproxy应用的脚本。

keepalived 配置

HA节点下载安装keepalive:

yum install keepalived -y

配置两个虚拟IP，一个用于k8s集群的apiserver代理，另一个用于nginx ingress入口（也可以单独配置），同时要设置对haproxy的状态判断，如果节点上的haproxy进程结束需要自动切换VIP到另一节点上，主HA配置如下：

# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

vrrp_script check_haproxy {
        script "/etc/keepalived/check_haproxy.sh"
        interval 3
        weight -20
}

vrrp_instance K8S {
    state backup 
    interface eth0
    virtual_router_id 44
    priority 200
    advert_int 5
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.20.50
        192.168.20.60

    }
    track_script {
        check_haproxy
   }

}

从HA配置如下：

! Configuration File for keepalived

vrrp_script check_haproxy {
        script "/etc/keepalived/check_haproxy.sh"
        interval 3
        weight -20
}

vrrp_instance K8S {
    state backup
    interface eth0
    virtual_router_id 44
    priority 190
    advert_int 5
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.20.50
        192.168.20.60

    }
    track_script {
        check_haproxy
   }
}

在这两个节点上配置对应的监测脚本：

vim /etc/keepalived/check_haproxy.sh

#!/bin/bash
active_status=`netstat -lntp|grep haproxy|wc -l`
if [ $active_status -gt 0 ]; then
    exit 0
else
    exit 1
fi

需要添加权限

chmod +x /etc/keepalived/check_haproxy.sh

部署Haproxy

官方配置手册

需要先确认已经配置了内核参数：

echo 'net.ipv4.ip_nonlocal_bind = 1'>>/etc/sysctl.conf
echo 'net.ipv4.ip_forward = 1'>>/etc/sysctl.conf

sysctl -p

安装haproxy

yum install haproxy -y

配置haproxy, 我们为k8s集群设计的VIP为 192.168.20.50，使用4层代理的方式，配置文件如下：

# cat /etc/haproxy/haproxy.cfg |egrep -v "^#"

global

    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

defaults
    mode                    tcp           # 修改默认为四层代理
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend  main 192.168.20.50:6443
    acl url_static       path_beg       -i /static /images /javascript /stylesheets
    acl url_static       path_end       -i .jpg .gif .png .css .js

    default_backend             k8s-node

backend k8s-node
    mode        tcp             # 修改为tcp
    balance     roundrobin
    server  k8s-node-1  192.168.20.44:6443 check     # 三个master主机
    server  k8s-node-2  192.168.20.45:6443 check
    server  k8s-node-3  192.168.20.46:6443 check

配置完成之后检查IP是否可以自动切换。

部署ETCD集群

1.安装etcd

执行如下命令，完成etcd的安装：

[root@master-1 ~]# cd /tmp/
[root@master-1 tmp]# tar xf etcd-v3.3.12-linux-amd64.tar.gz 
[root@master-1 tmp]# cd etcd-v3.3.12-linux-amd64
[root@master-1 tmp]# cp etcd* /opt/kubernetes/bin/
[root@master-1 tmp]# scp etcd* 192.168.20.45:/opt/kubernetes/bin/
[root@master-1 tmp]# scp etcd* 192.168.20.46:/opt/kubernetes/bin/

2.生成etcd的专属证书

1.创建etcd证书签名请求

[root@master-1 ~]# vim etcd-csr.json
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
"192.168.20.44",
"192.168.20.45",
"192.168.20.46"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

2.生成etcd证书

[root@master-1 ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem  \
-ca-key=/opt/kubernetes/ssl/ca-key.pem  \
-config=/opt/kubernetes/ssl/ca-config.json  \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd

会生成如下文件：

[root@master-1 ~]# ll
total 16
-rw-r--r-- 1 root root 1062 Jul 26 01:18 etcd.csr
-rw-r--r-- 1 root root  287 Jul 26 00:50 etcd-csr.json
-rw------- 1 root root 1679 Jul 26 01:18 etcd-key.pem
-rw-r--r-- 1 root root 1436 Jul 26 01:18 etcd.pem

[root@master-1 ~]#  cp etcd*.pem /opt/kubernetes/ssl
[root@master-1 ~]# scp etcd*.pem 192.168.20.45:/opt/kubernetes/ssl
[root@master-1 ~]# scp etcd*.pem 192.168.20.46:/opt/kubernetes/ssl

3.配置etcd

配置ETCD配置文件

master-1上的配置为：

[root@master-1 ~]# vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.20.44:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.44:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.44:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node-1=https://192.168.20.44:2380,etcd-node-2=https://192.168.20.45:2380,etcd-node-3=https://192.168.20.46:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.44:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

master-2上的配置为：

[root@master-2 tmp]# vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node-2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.20.45:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.45:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.45:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node-1=https://192.168.20.44:2380,etcd-node-2=https://192.168.20.45:2380,etcd-node-3=https://192.168.20.46:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.45:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

master-3上的配置为：

[root@master-3 ~]# vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node-3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.20.46:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.46:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.20.46:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node-1=https://192.168.20.44:2380,etcd-node-2=https://192.168.20.45:2380,etcd-node-3=https://192.168.20.46:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.46:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

在三个节点上创建etcd的systemd文件：

[root@master-1 ~]# vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd"

[Install]
WantedBy=multi-user.target

启动ETCD服务,在三个节点执行如下命令:

mkdir /var/lib/etcd
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

确认所以节点的etcd服务启动。

4.验证集群

[root@master-1 ~]# etcdctl --endpoints=https://192.168.20.44:2379 \
 --ca-file=/opt/kubernetes/ssl/ca.pem \
 --cert-file=/opt/kubernetes/ssl/etcd.pem \
 --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health
member 32922a109cfe00b2 is healthy: got healthy result from https://192.168.20.46:2379
member 4fa519fdd3e64a84 is healthy: got healthy result from https://192.168.20.45:2379
member cab6e832332e8b2a is healthy: got healthy result from https://192.168.20.44:2379
cluster is healthy

Master节点部署

1.部署Kubernetes软件包

[root@master-1 ~]# cd /tmp/kubernetes/server/bin/
[root@master-1 bin]# cp kube-apiserver /opt/kubernetes/bin/
[root@master-1 bin]# cp kube-controller-manager /opt/kubernetes/bin/
[root@master-1 bin]# cp kube-scheduler /opt/kubernetes/bin/

2.生成API Server的认证文件

参考链接

1.创建用于生成CSR的JSON文件，这里需要指定 HA代理的IP ，和集群的ClusterIP：

[root@master-1 ~]# cd /opt/kubernetes/ssl
[root@master-1 ssl]# vim kubernetes-csr.json
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.20.50",
    "10.1.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

2.生成Kubernetes的证书和私钥

[root@master-1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
   -ca-key=/opt/kubernetes/ssl/ca-key.pem \
   -config=/opt/kubernetes/ssl/ca-config.json \
   -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

将私钥分发到其它所有的node节点中：

[root@master-1 ssl]# scp kubernetes*.pem 192.168.20.46:/opt/kubernetes/ssl/
...

创建 API Server使用的token文件

[root@master-1 ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
197f33fcbbfab2d15603dcc4408358f5
[root@master-1 ~]# vim /opt/kubernetes/ssl/bootstrap-token.csv
197f33fcbbfab2d15603dcc4408358f5,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

创建基础用户名，密码认证配置

[root@k8s-node-1 ~]#  vim /opt/kubernetes/ssl/basic-auth.csv
admin,admin,1
readonly,readonly,2

将ssl目录的文件拷贝到其它的master节点上

scp -r -p /opt/kubernetes/ssl/*  k8s-node-1:/opt/kubernetes/ssl/
scp -r -p /opt/kubernetes/ssl/*  k8s-node-2:/opt/kubernetes/ssl/
scp -r -p /opt/kubernetes/ssl/*  k8s-node-3:/opt/kubernetes/ssl/

3.部署kube-apiserver

创建kube-apiserver的systemd文件

[root@k8s-node-1 ~]#  vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \
  --enable-admission-plugins=MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
  --bind-address=192.168.20.44 \
  --insecure-bind-address=127.0.0.1 \
  --authorization-mode=Node,RBAC \
  --runtime-config=rbac.authorization.k8s.io/v1 \
  --kubelet-https=true \
  --anonymous-auth=false \
  --basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \
  --enable-bootstrap-token-auth \
  --token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \
  --service-cluster-ip-range=10.1.0.0/16 \
  --service-node-port-range=20000-40000 \
  --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --etcd-cafile=/opt/kubernetes/ssl/ca.pem \
  --etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \
  --etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \
  --etcd-servers=https://192.168.20.44:2379,https://192.168.20.45:2379,https://192.168.20.46:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/opt/kubernetes/log/api-audit.log \
  --event-ttl=1h \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动kube-apiserver服务

[root@k8s-node-1 ~]# systemctl daemon-reload
[root@k8s-node-1 ~]# systemctl start kube-apiserver
[root@k8s-node-1 ~]# systemctl enable kube-apiserver

查看服务状态是否正常

[root@master-1 ~]# systemctl status kube-apiserver
[root@master-1 ~]# netstat -lntp|grep kube-apiserver
tcp        0      0 192.168.20.44:6443      0.0.0.0:*               LISTEN      4289/kube-apiserver 
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      4289/kube-apiserver

4.部署controller-manager

生成controller-manager的systemd文件

[root@master-1 ~]# vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kubernetes/bin/kube-controller-manager \
  --bind-address=127.0.0.1 \
  --master=http://127.0.0.1:8080 \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=10.1.0.0/16 \
  --cluster-cidr=10.2.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --root-ca-file=/opt/kubernetes/ssl/ca.pem \
  --leader-elect=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

启动kube-controller-manager

[root@master-1 ~]# systemctl daemon-reload
[root@master-1 ~]# systemctl start kube-controller-manager
[root@master-1 ~]# systemctl enable kube-controller-manager

查看服务状态

[root@master-1 ~]# systemctl status kube-controller-manager
[root@master-1 ~]# netstat -lntp|grep kube-con
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      4390/kube-controlle

5.部署Kubernetes Scheduler

创建systemd文件：

[root@master-1 ~]# vim /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kubernetes/bin/kube-scheduler \
  --address=127.0.0.1 \
  --master=http://127.0.0.1:8080 \
  --leader-elect=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

启动服务

[root@master-1 ~]# systemctl daemon-reload
[root@master-1 ~]# systemctl start kube-scheduler
[root@master-1 ~]# systemctl enable kube-scheduler

查看服务状态

[root@master-1 ~]# systemctl status kube-scheduler
[root@master-1 ~]# netstat -lntp|grep kube-scheduler
tcp        0      0 127.0.0.1:10251         0.0.0.0:*               LISTEN      4445/kube-scheduler

6. Master节点部署kube-proxy(可选)

(参见node节点部署部分，需要创建对应的kube-proxy家目录)

7. 使用上述方式，配置master-1和master-2

将master-1上的ssl,cfg,bin 文件拷贝到其它master节点的对应位置。
配置各个服务的启动文件，并启动。

8.部署kubectl命令行工具

安装二进制包

[root@master-1 ~]# cd /tmp/kubernetes/node/bin/
[root@master-1 bin]# cp kubectl /opt/kubernetes/bin/

2.创建admin证书签名

[root@master-1 ~]# vim /opt/kubernetes/ssl/admin-csr.json
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

3.生成admin证书和私钥

[root@master-1 ~]# cd /opt/kubernetes/ssl/
[root@master-1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
   -ca-key=/opt/kubernetes/ssl/ca-key.pem \
   -config=/opt/kubernetes/ssl/ca-config.json \
   -profile=kubernetes admin-csr.json | cfssljson -bare admin

设置集群参数

[root@master-1 ~]#  kubectl config set-cluster kubernetes \
    --certificate-authority=/opt/kubernetes/ssl/ca.pem \
    --embed-certs=true \
    --server=https://192.168.20.50:6443
Cluster "kubernetes" set.

5.设置客户端认证参数：

[root@naster-1 ~]# kubectl config set-credentials admin \
    --client-certificate=/opt/kubernetes/ssl/admin.pem \
    --embed-certs=true \
    --client-key=/opt/kubernetes/ssl/admin-key.pem
User "admin" set.

6.设置上下文参数

[root@master-1 ~]# kubectl config set-context kubernetes \
    --cluster=kubernetes \
    --user=admin
Context "kubernetes" created.

7.设置默认上下文：

[root@master-1 ~]# kubectl config use-context kubernetes
Switched to context "kubernetes".

8.使用Kubectl工具查看当前状态：

[root@master-1 ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}

Node节点部署

1.安装所需的服务

将kubernetes-node-linux-amd64.tar.gz压缩包在node节点解压，执行如下操作

[root@k8s-node-1 ~]# cd /tmp/kubernetes/node/bin
[root@k8s-node-1 bin]# cp kubelet kube-proxy  /opt/kubernetes/bin/
[root@k8s-node-1 bin]# scp kubelet kube-proxy  192.168.20.48:/opt/kubernetes/bin/
[root@k8s-node-1 bin]# scp kubelet kube-proxy  192.168.20.49:/opt/kubernetes/bin/

2.配置角色和认证参数

在master-1上创建角色绑定

[root@master-1 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io "kubelet-bootstrap" created

创建kubelet bootstrapping kubeconfig文件，设置集群参数

[root@master-1 ~]# kubectl config set-cluster kubernetes \
   --certificate-authority=/opt/kubernetes/ssl/ca.pem \
   --embed-certs=true \
   --server=https://192.168.20.50:6443 \
   --kubeconfig=bootstrap.kubeconfig
Cluster "kubernetes" set.

3.设置客户端认证参数

[root@master-1 ~]# kubectl config set-credentials kubelet-bootstrap \
   --token=197f33fcbbfab2d15603dcc4408358f5 \
   --kubeconfig=bootstrap.kubeconfig   

User "kubelet-bootstrap" set.

4.设置上下文认证参数

[root@master-1 ~]# kubectl config set-context default \
    --cluster=kubernetes \
    --user=kubelet-bootstrap \
    --kubeconfig=bootstrap.kubeconfig
Context "default" created.

5.选择默认上下文

[root@master-1 ~]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
Switched to context "default"

6.执行上面的操作后，会在当前目录生成一个bootstrap.kubeconfig的config文件，将此文件分发到各节点：

[root@k8s-node-1 ~]# cp bootstrap.kubeconfig /opt/kubernetes/cfg/
[root@k8s-node-1 ~]# scp bootstrap.kubeconfig 192.168.20.47:/opt/kubernetes/cfg/
[root@k8s-node-1 ~]# scp bootstrap.kubeconfig 192.168.20.48:/opt/kubernetes/cfg/
[root@k8s-node-1 ~]# scp bootstrap.kubeconfig 192.168.20.49:/opt/kubernetes/cfg/

将master上更新的配置拷贝到其它master节点。

3.设置支持CNI

以下操作需要在所有node节点执行

设置Kubernetes对CNI的支持：

[root@k8s-node-2 ~]# mkdir -p /etc/cni/net.d
[root@k8s-node-2 ~]# vim /etc/cni/net.d/10-default.conf
{
        "name": "flannel",
        "type": "flannel",
        "delegate": {
            "bridge": "docker0",
            "isDefaultGateway": true,
            "mtu": 1400
        }
}

4.配置Kubelet服务

以下操作需要在所有node节点执行

创建kubelet服务配置文件

[root@k8s-node-2 ~]# mkdir /var/lib/kubelet
[root@k8s-node-2 ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \
  --address=192.168.20.48 \
  --hostname-override=192.168.20.48 \
  --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.1 \
  --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
  --cert-dir=/opt/kubernetes/ssl \
  --network-plugin=cni \
  --cni-conf-dir=/etc/cni/net.d \
  --cni-bin-dir=/opt/kubernetes/bin/cni \
  --cluster-dns=10.1.0.2 \
  --cluster-domain=cluster.local. \
  --hairpin-mode hairpin-veth \
  --allow-privileged=true \
  --fail-swap-on=false \
  --logtostderr=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

启动Kubelet

[root@k8s-node-2 ~]# systemctl daemon-reload
[root@k8s-node-2 ~]# systemctl start kubelet
[root@k8s-node-2 ~]# systemctl enable kubelet
[root@k8s-node-2 ~]# systemctl status kubelet

在master节点上查看是否收到node节点的csr请求：

[root@master-1 ~]# kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-FDH7Y3rghf1WPsEJH2EYnofvOSeyHn2f-l_-4rH-LEk   2m        kubelet-bootstrap   Pending

批准kubelet的TLS请求

[root@master-1 ~]# kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve
certificatesigningrequest.certificates.k8s.io "node-csr-FDH7Y3rghf1WPsEJH2EYnofvOSeyHn2f-l_-4rH-LEk" approved

[root@kmaster-1 ~]# kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-FDH7Y3rghf1WPsEJH2EYnofvOSeyHn2f-l_-4rH-LEk   11m       kubelet-bootstrap   Approved,Issued

之后查看node节点状态：

[root@master-1 ~]# kubectl get node
NAME            STATUS    ROLES     AGE       VERSION
192.168.20.48   Ready     <none>    35s       v1.14.1

node节点上查看kubelet 服务

[root@k8s-node-2 ~]# netstat -lntp|grep kubelet
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      7917/kubelet        
tcp        0      0 192.168.20.32:10250     0.0.0.0:*               LISTEN      7917/kubelet        
tcp        0      0 192.168.20.32:10255     0.0.0.0:*               LISTEN      7917/kubelet        
tcp        0      0 192.168.20.32:4194      0.0.0.0:*               LISTEN      7917/kubelet

5.部署kube-proxy

1.配置kube-proxy使用LVS，所有节点执行：

yum install -y ipvsadm ipset conntrack

2.创建证书请求

[root@master-1 ~]# vim kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

3.生成证书

[root@master-1 ~]#  cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
    -ca-key=/opt/kubernetes/ssl/ca-key.pem \
    -config=/opt/kubernetes/ssl/ca-config.json \
    -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy

4.分发证书到所有node节点

[root@master-1 ~]# cp kube-proxy*.pem /opt/kubernetes/ssl/
[root@master-1 ~]# scp kube-proxy*.pem 192.168.20.47:/opt/kubernetes/ssl/
[root@master-1 ~]# scp kube-proxy*.pem 192.168.20.48:/opt/kubernetes/ssl/
[root@master-1 ~]# scp kube-proxy*.pem 192.168.20.49:/opt/kubernetes/ssl/

5.创建kube-proxy配置文件

[root@k8s-node-2 ~]# kubectl config set-cluster kubernetes  \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem   \
  --embed-certs=true   \
  --server=https://192.168.20.50:6443 \
  --kubeconfig=kube-proxy.kubeconfig
Cluster "kubernetes" set.

6.创建kube-proxy用户：

[root@k8s-node-2 ~]# kubectl config set-credentials kube-proxy \
    --client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \
    --client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \
    --embed-certs=true \
    --kubeconfig=kube-proxy.kubeconfig
User "kube-proxy" set.

7.设置默认上下文：

[root@k8s-node-2 ~]# kubectl config set-context default \
    --cluster=kubernetes \
    --user=kube-proxy \
    --kubeconfig=kube-proxy.kubeconfig
Context "default" created.

8.切换上下文为default:

[root@k8s-node-2 ~]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
Switched to context "default".

9.分发kube-proxy.kubeconfig配置文件到所有

[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.44:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.45:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.46:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.47:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.48:/opt/kubernetes/cfg/
[root@k8s-node-2 ~]# scp kube-proxy.kubeconfig 192.168.20.459/opt/kubernetes/cfg/

10.创建kube-proxy服务配置文件

所有节点执行，注意配置文件中IP需要修改为本机对应的IP

[root@k8s-node-1 ~]# mkdir /var/lib/kube-proxy
[root@k8s-node-1 ~]# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \
  --bind-address=192.168.20.47 \
  --hostname-override=192.168.20.47 \
  --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \
  --masquerade-all \
  --feature-gates=SupportIPVSProxyMode=true \
  --proxy-mode=ipvs \
  --ipvs-min-sync-period=5s \
  --ipvs-sync-period=5s \
  --ipvs-scheduler=rr \
  --logtostderr=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

11.启动服务

systemctl start kube-proxy
systemctl enable kube-proxy
systemctl status kube-proxy

12.查看服务状态,lvs状态

[root@k8s-node-1 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.1.0.1:443 rr
  -> 192.168.20.44:6443           Masq    1      0          0         
  -> 192.168.20.45:6443           Masq    1      0          0         
  -> 192.168.20.46:6443           Masq    1      1          0

所有node节点配置成功后可以看到如下结果：

[root@master-1 ~]# kubectl get node
NAME            STATUS   ROLES    AGE     VERSION
192.168.20.47   Ready    <none>   6d21h   v1.14.1
192.168.20.48   Ready    <none>   4d1h    v1.14.1
192.168.20.49   Ready    <none>   4d1h    v1.14.1

Flannel 网络部署

所有节点都需要部署flannel。

1.创建Flannel证书

1.生成证书文件

[root@master-1 ~]# vim flanneld-csr.json
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

2.生成证书

[root@master-1 ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
    -ca-key=/opt/kubernetes/ssl/ca-key.pem \
    -config=/opt/kubernetes/ssl/ca-config.json \
    -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

3.分发证书

[root@master-1 ~]# cp flanneld*.pem /opt/kubernetes/ssl/
[root@master-1 ~]# scp flanneld*.pem {all-k8s-node}:/opt/kubernetes/ssl/

2.部署flannel

1.将之前下载的flannel压缩包解压，并分发到其它节点如下操作：

cp mk-docker-opts.sh flanneld /opt/kubernetes/bin/
scp mk-docker-opts.sh flanneld {all-k8s-node}:/opt/kubernetes/bin/

2.创建如下文件，分发到各个node节点：

[root@k8s-node-1 tmp]# vim remove-docker0.sh
#!/bin/bash
# Delete default docker bridge, so that docker can start with flannel network.

# exit on any error
set -e

rc=0
ip link show docker0 >/dev/null 2>&1 || rc="$?"
if [[ "$rc" -eq "0" ]]; then
  ip link set dev docker0 down
  ip link delete docker0
fi

[root@k8s-node-1 tmp]# cp remove-docker0.sh /opt/kubernetes/bin/
[root@k8s-node-1 tmp]# scp remove-docker0.sh 192.168.20.48:/opt/kubernetes/bin/
[root@k8s-node-1 tmp]# scp remove-docker0.sh 192.168.20.49:/opt/kubernetes/bin/

3.配置flannel

[root@k8s-node-1 ~]# vim /opt/kubernetes/cfg/flannel
FLANNEL_ETCD="-etcd-endpoints=https://192.168.20.31:2379,https://192.168.20.32:2379,https://192.168.20.33:2379"
FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network"
FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem"
FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem"
FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem"

创建flannel服务文件

[root@k8s-node-1 ~]# vim /usr/lib/systemd/system/flannel.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
Before=docker.service

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/flannel
ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh
ExecStart=/opt/kubernetes/bin/flanneld ${FLANNEL_ETCD} ${FLANNEL_ETCD_KEY} ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE}
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker

Type=notify

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

5.分发创建的配置文件到各个节点：

scp /opt/kubernetes/cfg/flannel {all-k8s-node}:/opt/kubernetes/cfg/
scp /usr/lib/systemd/system/flannel.service {all-k8s-node}:/usr/lib/systemd/system/

3. Flannel CNI集成

1.下载CNI插件

wget https://github.com/containernetworking/plugins/releases/download/v0.7.5/cni-plugins-amd64-v0.7.5.tgz
[root@k8s-node-1 tmp]# mkdir /opt/kubernetes/bin/cni
[root@k8s-node-1 tmp]# tar xf cni-plugins-amd64-v0.7.5.tgz -C /opt/kubernetes/bin/cni

2.分发软件到各个节点：

[root@k8s-node-1 ~]# scp -r /opt/kubernetes/bin/cni/* {all-k8s-node}:/opt/kubernetes/bin/cni/

3.在etcd中创建key

[root@master-1 ~]# /opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file /opt/kubernetes/ssl/flanneld-key.pem \
     --no-sync -C https://192.168.20.44:2379,https://192.168.20.45:2379,https://192.168.20.46:2379 \
     mk /kubernetes/network/config '{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}' >/dev/null 2>&1

4.各个节点启动flannel

[root@k8s-node-1 ~]# chmod +x /opt/kubernetes/bin/*
[root@k8s-node-1 ~]# systemctl daemon-reload
[root@k8s-node-1 ~]# systemctl start flannel 
[root@k8s-node-1 ~]# systemctl enable flannel

配置Docker使用Flannel

1.修改docker的systemd的文件：

[Unit] #在Unit下面修改After和增加Requires
After=network-online.target firewalld.service flannel.service
Wants=network-online.target
Requires=flannel.service

[Service] #增加EnvironmentFile=-/run/flannel/docker
Type=notify
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_OPTS

2.其它NODE节点也做相同的修改

[root@k8s-node-2 ~]# scp /usr/lib/systemd/system/docker.service {k8s-node}:/usr/lib/systemd/system/

3.重启docker, 出现docker0网卡，且在10.2.0.0/16网段，说明配置成功

[root@k8s-node-3 ~]# systemctl daemon-reload
[root@k8s-node-3 ~]# systemctl restart docker
[root@k8s-node-3 ~]# ip a| grep -A 3 'docker0'
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:e9:2b:36:86 brd ff:ff:ff:ff:ff:ff
    inet 10.2.79.1/24 scope global docker0
       valid_lft forever preferred_lft forever

插件部署

1.创建CoreDNS

创建coredns.yaml，内容如下：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
  labels:
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: Reconcile
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
    addonmanager.kubernetes.io/mode: EnsureExists
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
  labels:
      addonmanager.kubernetes.io/mode: EnsureExists
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local. in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
    }
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: coredns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "CoreDNS"
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: coredns
  template:
    metadata:
      labels:
        k8s-app: coredns
    spec:
      serviceAccountName: coredns
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      containers:
      - name: coredns
        image: coredns/coredns:1.4.0
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: coredns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: coredns
  clusterIP: 10.1.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP

执行此文件：

[root@master-1 tmp]# kubectl create -f coredns.yaml

确认DNS服务运行：

[root@master-1 ~]# kubectl get pod  -n kube-system  -o wide
NAME                                    READY   STATUS    RESTARTS   AGE    IP          NODE            NOMINATED NODE   READINESS GATES
coredns-76fcfc9f65-9fkfh                1/1     Running   2          3d7h   10.2.45.3   192.168.20.49   <none>           <none>
coredns-76fcfc9f65-zfplt                1/1     Running   1          3d6h   10.2.24.2   192.168.20.48   <none>           <none>

2. 部署Dashboard

1.执行目录中的yaml,部署Dashboard:

[root@master-1 ~]# ll /tmp/dashboard/
total 20
-rw-r--r-- 1 root root  356 Jul 27 03:43 admin-user-sa-rbac.yaml
-rw-r--r-- 1 root root 4253 Jul 27 03:47 kubernetes-dashboard.yaml
-rw-r--r-- 1 root root  458 Jul 27 03:49 ui-admin-rbac.yaml
-rw-r--r-- 1 root root  477 Jul 27 03:50 ui-read-rbac.yaml

[root@master-1 ~]# kubectl create -f /tmp/dashboard/

2.确认服务是否正常运行：

[root@master-1 ~]# kubectl get pod -n kube-system
NAME                                    READY   STATUS    RESTARTS   AGE
coredns-76fcfc9f65-9fkfh                1/1     Running   2          3d7h
coredns-76fcfc9f65-zfplt                1/1     Running   1          3d6h
kubernetes-dashboard-68ddcc97fc-w4bxf   1/1     Running   1          3d2h

[root@master-1 ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.20.50:6443
CoreDNS is running at https://192.168.20.50:6443/api/v1/namespaces/kube-system/services/coredns:dns/proxy
kubernetes-dashboard is running at https://192.168.20.50:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

3.根据提示信息，使用dashboard的url,登录，账号admin/admin, 使用如下命令生成token:

[root@master-1 ~]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

4.复制token,选择使用令牌的方式登录：

K8S1.14高可用生产集群部署方案

3. Heapster 部署 (可选)

1.使用如下文件部署Heastper:

[root@master-1 ~]# ll heastper/
total 12
-rw-r--r-- 1 root root 2306 Jul 26 20:28 grafana.yaml
-rw-r--r-- 1 root root 1562 Jul 26 20:29 heapster.yaml
-rw-r--r-- 1 root root 1161 Jul 26 20:29 influxdb.yaml

[root@k8s-node-1 ~]# kubectl create -f heastper/

登录dashboard，查看石头出现资源利用率的图表。

K8S1.14高可用生产集群部署方案

使用 kubectl cluster-info 命令，查看当前服务的url地址。

补充说明

etcd无证书配置说明

在实际的生产环境中，如果都是使用的内网环境，可以将etd集群配置为无证书的模式，这样在配置和后续的故障恢复中会更简单。

etcd无证书配置需要使用http访问，安装上述文档，需要修改如下配置：

etcd的配置文件注释掉安全证书部分，并将所有url改为http方式：

# cat /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="http://192.168.20.31:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.20.31:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.20.31:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node-1=http://192.168.20.31:2380,etcd-node-2=http://192.168.20.32:2380,etcd-node-3=http://192.168.20.33:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.20.31:2379"
#[security]
#CLIENT_CERT_AUTH="true"
#ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
#ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
#ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
#PEER_CLIENT_CERT_AUTH="true"
#ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
#ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
#ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

fannel网络部分注释掉etcd的证书配置参数，并将URL该位http:

# cat /opt/kubernetes/cfg/flannel 

FLANNEL_ETCD="-etcd-endpoints=http://192.168.20.31:2379,http://192.168.20.32:2379,http://192.168.20.33:2379"
FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network"
#FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem"
#FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem"
#FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem"

3.kube-apiserver中去掉etcd的证书配置，这个文件中需要把参数直接删除，并将url改为http:

# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \
  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
  --bind-address=192.168.20.31 \
  --insecure-bind-address=127.0.0.1 \
  --authorization-mode=Node,RBAC \
  --runtime-config=rbac.authorization.k8s.io/v1 \
  --kubelet-https=true \
  --anonymous-auth=false \
  --basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \
  --enable-bootstrap-token-auth \
  --token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \
  --service-cluster-ip-range=10.1.0.0/16 \
  --service-node-port-range=20000-40000 \
  --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --etcd-servers=http://192.168.20.31:2379,http://192.168.20.32:2379,http://192.168.20.33:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/opt/kubernetes/log/api-audit.log \
  --event-ttl=1h \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

4.分别重启flannel、kubelet、kube-apiserver等服务。

以上就是本文的全部内容，希望本文的内容对大家的学习或者工作能带来一定的帮助，也希望大家多多支持码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。

码农书籍

电脑报（上下册）

电脑报社 / 西南师范大学出版社 / 2006-12-01 / 45.00元

全套上、下两册，浓缩2006年电脑报精华文章。附录包含70余篇简明IT应用指南，覆盖软件、硬盘、数码、网络四大领域。配赠权威实用的2006-2007中国计算机年鉴DVD光盘，近1.4GB海量信息与资源超值奉献。8大正版超值软件，涵盖系统维护、系统安全、办公应用、多媒体娱乐等四大领域。微软、腾讯、友立等知名厂商，新年献礼。提供2006-2007全系列硬件、数码产品资讯，兼具知识性与资料性。官方网站全......一起来看看《电脑报（上下册）》这本书的介绍吧!

码农工具

K8S1.14高可用生产集群部署方案

系统说明

系统组件版本

架构示意图

系统初始化

1. 主机初始化

2. 安装Docker

3. 创建目录

4. 准备Kubernetes软件包

5.准备etcd和flannel组件

制作CA证书

1.安装CFSSL

2.生成证书

HA 节点部署

keepalived 配置

部署Haproxy

部署ETCD集群

1.安装etcd

2.生成etcd的专属证书

3.配置etcd

4.验证集群

Master节点部署

1.部署Kubernetes软件包

2.生成API Server的认证文件

3.部署kube-apiserver

4.部署controller-manager

5.部署Kubernetes Scheduler

6. Master节点部署kube-proxy(可选)

7. 使用上述方式，配置master-1和master-2

8.部署kubectl命令行工具

Node节点部署

1.安装所需的服务

2.配置角色和认证参数

3.设置支持CNI

4.配置Kubelet服务

5.部署kube-proxy

Flannel 网络部署

1.创建Flannel证书

2.部署flannel

3. Flannel CNI集成

配置Docker使用Flannel

插件部署

1.创建CoreDNS

2. 部署Dashboard

3. Heapster 部署 (可选)

补充说明

etcd无证书配置说明

电脑报（上下册）

Markdown 在线编辑器

UNIX 时间戳转换

正则表达式在线测试