内容简介:Kubernetes v1.13版本发布后,kubeadm才正式进入GA,可以生产使用,用kubeadm部署kubernetes集群也是以后的发展趋势。目前Kubernetes的对应镜像仓库,在国内阿里云也有了镜像站点,使用kubeadm部署Kubernetes集群变得简单并且容易了很多,本文使用kubeadm带领大家快速部署Kubernetes v1.13.2版本。注意:请不要把目光仅仅放在部署上,如果你是新手,推荐先熟悉用二进制文件部署后,再来学习用kubeadm部署。二进制文件部署请查看我博客的其他文
1. 简介
Kubernetes v1.13版本发布后,kubeadm才正式进入GA,可以生产使用,用kubeadm部署kubernetes集群也是以后的发展趋势。目前Kubernetes的对应镜像仓库,在国内阿里云也有了镜像站点,使用kubeadm部署Kubernetes集群变得简单并且容易了很多,本文使用kubeadm带领大家快速部署Kubernetes v1.13.2版本。
注意:请不要把目光仅仅放在部署上,如果你是新手,推荐先熟悉用二进制文件部署后,再来学习用kubeadm部署。二进制文件部署请查看我博客的其他文章。
2. 架构信息
系统版本:CentOS 7.6 内核:3.10.0-957.el7.x86_64 Kubernetes: v1.13.2 Docker-ce: 18.06 推荐硬件配置:2核2G Keepalived保证apiserever服务器的IP高可用 Haproxy实现apiserver的负载均衡
为了减少服务器数量,haproxy、keepalived配置在node-01和node-02。
节点名称 | 角色 | IP | 安装软件 |
---|---|---|---|
负载VIP | VIP | 10.31.90.200 | |
node-01 | master | 10.31.90.201 | kubeadm、kubelet、kubectl、 docker 、haproxy、keepalived |
node-02 | master | 10.31.90.202 | kubeadm、kubelet、kubectl、docker、haproxy、keepalived |
node-03 | master | 10.31.90.203 | kubeadm、kubelet、kubectl、docker |
node-04 | node | 10.31.90.204 | kubeadm、kubelet、kubectl、docker |
node-05 | node | 10.31.90.205 | kubeadm、kubelet、kubectl、docker |
node-06 | node | 10.31.90.206 | kubeadm、kubelet、kubectl、docker |
service网段 | 10.245.0.0/16 |
2.部署前准备工作
1) 关闭selinux和防火墙
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config setenforce 0 systemctl disable firewalld systemctl stop firewalld
2) 关闭swap
swapoff -a
3) 为每台服务器添加host解析记录
cat >>/etc/hosts<<EOF 10.31.90.201 node-01 10.31.90.202 node-02 10.31.90.203 node-03 10.31.90.204 node-04 10.31.90.205 node-05 10.31.90.206 node-06 EOF
4) 创建并分发密钥
在node-01创建ssh密钥。
[root@node-01 ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:26z6DcUarn7wP70dqOZA28td+K/erv7NlaJPLVE1BTA root@node-01 The key's randomart image is: +---[RSA 2048]----+ | E..o+| | . o| | . | | . . | | S o . | | .o X oo .| | oB +.o+oo.| | .o*o+++o+o| | .++o+Bo+=B*B| +----[SHA256]-----+
分发node-01的公钥,用于免密登录其他服务器
for n in `seq -w 01 06`;do ssh-copy-id node-$n;done
5) 配置内核参数
cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 vm.swappiness=0 EOF sysctl --system
6) 加载ipvs模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
7) 添加yum源
cat << EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF wget http://mirrors.aliyun.com/repo/Centos-7.repo -O /etc/yum.repos.d/CentOS-Base.repo wget http://mirrors.aliyun.com/repo/epel-7.repo -O /etc/yum.repos.d/epel.repo wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
2. 部署keepalived和haproxy
1) 安装keepalived和haproxy
在node-01和node-02安装keepalived和haproxy
yum install -y keepalived haproxy
2) 修改配置
keepalived配置
node-01的 priority
为100,node-02的 priority
为90,其他配置一样。
[root@node-01 ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { feng110498@163.com } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_1 } vrrp_instance VI_1 { state MASTER interface eth0 lvs_sync_daemon_inteface eth0 virtual_router_id 88 advert_int 1 priority 100 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.31.90.200/24 } }
haproxy配置
node-01和node-02的haproxy配置是一样的。此处我们监听的是10.31.90.200的8443端口,因为haproxy是和k8s apiserver是部署在同一台服务器上,都用6443会冲突。
global chroot /var/lib/haproxy daemon group haproxy user haproxy log 127.0.0.1:514 local0 warning pidfile /var/lib/haproxy.pid maxconn 20000 spread-checks 3 nbproc 8 defaults log global mode tcp retries 3 option redispatch listen https-apiserver bind 10.31.90.200:8443 mode tcp balance roundrobin timeout server 15s timeout connect 15s server apiserver01 10.31.90.201:6443 check port 6443 inter 5000 fall 5 server apiserver02 10.31.90.202:6443 check port 6443 inter 5000 fall 5 server apiserver03 10.31.90.203:6443 check port 6443 inter 5000 fall 5
3) 启动服务
systemctl enable keepalived && systemctl start keepalived systemctl enable haproxy && systemctl start haproxy
3. 部署kubernetes
1) 安装软件
由于kubeadm对Docker的版本是有要求的,需要安装与kubeadm匹配的版本。
由于版本更新频繁,请指定对应的版本号,本文采用1.13.2版本,其它版本未经测试。
yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1 ipvsadm ipset docker-ce-18.06.1.ce #启动docker systemctl enable docker && systemctl start docker #设置kubelet开机自启动 systemctl enable kubelet
2) 修改初始化配置
[root@node-01 ~]# cat kubeadm-init.yaml apiVersion: kubeadm.k8s.io/v1beta1 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 10.31.90.201 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: node-01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration apiServer: timeoutForControlPlane: 4m0s certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: "10.31.90.200:8443" dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers kubernetesVersion: v1.13.2 networking: dnsDomain: cluster.local podSubnet: "" serviceSubnet: "10.245.0.0/16" scheduler: {} controllerManager: {} --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: "ipvs"
3) 预下载镜像
[root@node-01 ~]# kubeadm config images pull --config kubeadm-init.yaml [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.13.2 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.13.2 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.13.2 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.13.2 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.2.24 [config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.2.6
4) 初始化
[root@node-01 ~]# kubeadm init --config kubeadm-init.yaml [init] Using Kubernetes version: v1.13.2 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [node-01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.12.0.1 10.31.90.201 10.31.90.200] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [node-01 localhost] and IPs [10.31.90.201 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [node-01 localhost] and IPs [10.31.90.201 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "kubelet.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 22.503955 seconds [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node-01" as an annotation [mark-control-plane] Marking the node node-01 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node node-01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: abcdef.0123456789abcdef [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join 10.31.90.200:8443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:84201a329ec4388263e97303c6e4de50c2de2aa157a3b961cb8a6f325fadedb1
kubeadm init主要执行了以下操作:
-
[init]:指定版本进行初始化操作
-
[preflight] :初始化前的检查和下载所需要的Docker镜像文件
-
[kubelet-start] :生成kubelet的配置文件”/var/lib/kubelet/config.yaml”,没有这个文件kubelet无法启动,所以初始化之前的kubelet实际上启动失败。
-
[certificates]:生成Kubernetes使用的证书,存放在/etc/kubernetes/pki目录中。
-
[kubeconfig] :生成 KubeConfig 文件,存放在/etc/kubernetes目录中,组件之间通信需要使用对应文件。
-
[control-plane]:使用/etc/kubernetes/manifest目录下的YAML文件,安装 Master 组件。
-
[etcd]:使用/etc/kubernetes/manifest/etcd.yaml安装Etcd服务。
-
[wait-control-plane]:等待control-plan部署的Master组件启动。
-
[apiclient]:检查Master组件服务状态。
-
[uploadconfig]:更新配置
-
[kubelet]:使用configMap配置kubelet。
-
[patchnode]:更新CNI信息到Node上,通过注释的方式记录。
-
[mark-control-plane]:为当前节点打标签,打了角色Master,和不可调度标签,这样默认就不会使用Master节点来运行Pod。
-
[bootstrap-token]:生成token记录下来,后边使用kubeadm join往集群中添加节点时会用到
- [addons]:安装附加组件CoreDNS和kube-proxy
5) 为kubectl准备Kubeconfig文件
kubectl默认会在执行的用户家目录下面的.kube目录下寻找config文件。这里是将在初始化时[kubeconfig]步骤生成的admin.conf拷贝到.kube/config。
[root@node-01 ~]# mkdir -p $HOME/.kube [root@node-01 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@node-01 ~]# chown $(id -u):$(id -g)$HOME/.kube/config
在该配置文件中,记录了API Server的访问地址,所以后面直接执行kubectl命令就可以正常连接到API Server中。
6) 查看组件状态
[root@node-01 ~]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health": "true"}
[root@node-01 ~]# kubectl get node NAME STATUS ROLES AGE VERSION node-01 NotReady master 14m v1.13.2
目前只有一个节点,角色是Master,状态是NotReady。
7) 其他master部署
在node-01将证书文件拷贝至其他master节点
USER=root CONTROL_PLANE_IPS="node-02 node-03" for host in ${CONTROL_PLANE_IPS}; do ssh "${USER}"@$host "mkdir -p /etc/kubernetes/pki/etcd" scp /etc/kubernetes/pki/ca.* "${USER}"@$host:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/sa.* "${USER}"@$host:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/front-proxy-ca.* "${USER}"@$host:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/etcd/ca.* "${USER}"@$host:/etc/kubernetes/pki/etcd/ scp /etc/kubernetes/admin.conf "${USER}"@$host:/etc/kubernetes/ done
在其他master执行,注意 --experimental-control-plane
参数
kubeadm join 10.31.90.200:8443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:84201a329ec4388263e97303c6e4de50c2de2aa157a3b961cb8a6f325fadedb1 --experimental-control-plane
8) node部署
在node-04、node-05、node-06执行,注意没有 --experimental-control-plane
参数
kubeadm join 10.31.90.200:8443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:84201a329ec4388263e97303c6e4de50c2de2aa157a3b961cb8a6f325fadedb1
9) 部署网络插件weave
Master节点NotReady的原因就是因为没有使用任何的网络插件,此时Node和Master的连接还不正常。目前最流行的Kubernetes网络插件有Flannel、Calico、Canal、Weave这里选择使用Weave。
[root@node-01 ~]# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
10) 查看节点状态
所有的节点已经处于Ready状态。
[root@node-01 ~]# kubectl get node NAME STATUS ROLES AGE VERSION node-01 Ready master 35m v1.13.2 node-02 Ready master 36m v1.13.2 node-03 Ready master 36m v1.13.2 node-04 Ready <none> 40m v1.13.2 node-05 Ready <none> 40m v1.13.2 node-06 Ready <none> 40m v1.13.2
查看pod
[root@node-01 ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-89cc84847-j8mmg 1/1 Running 0 1d coredns-89cc84847-rbjxs 1/1 Running 0 1d etcd-node-01 1/1 Running 1 1d etcd-node-02 1/1 Running 0 1d etcd-node-03 1/1 Running 0 1d kube-apiserver-node-01 1/1 Running 0 1d kube-apiserver-node-02 1/1 Running 0 1d kube-apiserver-node-03 1/1 Running 0 1d kube-controller-manager-node-01 1/1 Running 2 1d kube-controller-manager-node-02 1/1 Running 0 1d kube-controller-manager-node-03 1/1 Running 0 1d kube-proxy-jfbmv 1/1 Running 0 1d kube-proxy-lvkms 1/1 Running 0 1d kube-proxy-qx7kh 1/1 Running 0 1d kube-proxy-xst5v 1/1 Running 0 1d kube-proxy-zfwrk 1/1 Running 0 1d kube-proxy-ztg6j 1/1 Running 0 1d kube-scheduler-node-01 1/1 Running 1 1d kube-scheduler-node-02 1/1 Running 1 1d kube-scheduler-node-03 1/1 Running 1 1d weave-net-2xr7j 2/2 Running 0 1d weave-net-6x2gn 2/2 Running 1 1d weave-net-l6fdw 2/2 Running 1 1d weave-net-mswmj 2/2 Running 2 1d weave-net-skdw7 2/2 Running 1 1d weave-net-zm7zf 2/2 Running 1 1d
查看ipvs的状态
[root@node-01 ~]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.245.0.1:443 rr -> 10.31.90.201:6443 Masq 1 2 0 -> 10.31.90.202:6443 Masq 1 0 0 -> 10.31.90.203:6443 Masq 1 2 0 TCP 10.245.0.10:53 rr -> 10.32.0.3:53 Masq 1 0 0 -> 10.32.0.4:53 Masq 1 0 0 TCP 10.245.90.161:80 rr -> 10.45.0.1:80 Masq 1 0 0 TCP 10.245.90.161:443 rr -> 10.45.0.1:443 Masq 1 0 0 TCP 10.245.149.227:1 rr -> 10.31.90.204:1 Masq 1 0 0 -> 10.31.90.205:1 Masq 1 0 0 -> 10.31.90.206:1 Masq 1 0 0 TCP 10.245.181.126:80 rr -> 10.34.0.2:80 Masq 1 0 0 -> 10.45.0.0:80 Masq 1 0 0 -> 10.46.0.0:80 Masq 1 0 0 UDP 10.245.0.10:53 rr -> 10.32.0.3:53 Masq 1 0 0 -> 10.32.0.4:53 Masq 1 0 0
至此kubernetes集群部署完成。如有问题欢迎在下面留言交流。希望大家多多关注和点赞,谢谢!
以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网
猜你喜欢:- 构建生产环境可用的高可用kubernetes集群
- MongoDB高可用集群搭建
- 构建高可用ZooKeeper集群
- 部署高可用Kubernetes集群
- 构建高可用ZooKeeper集群
- Kubernet 高可用集群搭建(下)
本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
Python for Data Analysis
Wes McKinney / O'Reilly Media / 2012-11-1 / USD 39.99
Finding great data analysts is difficult. Despite the explosive growth of data in industries ranging from manufacturing and retail to high technology, finance, and healthcare, learning and accessing d......一起来看看 《Python for Data Analysis》 这本书的介绍吧!