k8s集群水平扩展(HPA)

栏目: 编程工具 · 发布时间: 5年前

内容简介:Horizontal Pod Autoscaling,简称HPA,是Kubernetes中实现POD水平自动伸缩的功能。K8S集群可以通过Replication Controller的scale机制完成服务的扩容或缩容,实现具有伸缩性的服务。K8S自动伸缩分为:

Horizontal Pod Autoscaling,简称HPA,是Kubernetes中实现POD水平自动伸缩的功能。

简介

K8S集群可以通过Replication Controller的scale机制完成服务的扩容或缩容,实现具有伸缩性的服务。

K8S自动伸缩分为:

自动扩展主要分为两种:

  • 水平扩展(scale out),针对于实例数目的增减。
  • 垂直扩展(scal up),即单个实例可以使用的资源的增减, 比如增加cpu和增大内存。

HPA属于前者。它可以根据CPU使用率或应用自定义metrics自动扩展Pod数量(支持 replication controller、deployment 和 replica set)。

k8s集群水平扩展(HPA)

获取metrics的两种方式:

  • Heapster:heapster提供metrics服务,但是在v1(autoscaling/v1)版本中仅支持以CPU作为扩展度量指标。而其他比如:内存,网络流量,qps等目前处于beta阶段(autoscaling/v2beta1)。
  • Cousom:同样处于beta阶段(autoscaling/v2beta1),但是涉及到自定义的REST API的开发,复杂度会大一些,并且当需要从自定义的监控中获取数据时,只能设置绝对值,无法设置使用率。

工作流程

  • 创建HPA资源,设定目标CPU使用率限额,以及最大/最小实例数,一定要设置Pod的资源限制参数: request ,否则HPA不会工作。
  • 控制管理器每隔30s(在 kube-controller-manager.service 中可以通过 –horizontal-pod-autoscaler-sync-period 修改)查询metrics的资源使用情况。
  • 然后与创建时设定的值和指标做对比(平均值之和/限额),求出目标调整的实例个数。
  • 目标调整的实例数不能超过第一条中设定的最大/最小实例数。如果没有超过,则扩容;超过,则扩容至最大的实例个数。
  • 重复第2-4步。

自动伸缩算法

HPA Controller会通过调整副本数量使得CPU使用率尽量向期望值靠近,而且不是完全相等。另官方考虑到自动扩展的决策可能需要一段时间才会生效:例如当pod所需要的CPU负荷过大,从而在创建一个新pod的过程中,系统的CPU使用量可能会同样在有一个攀升的过程。所以在每一次作出决策后的一段时间内,将不再进行扩展决策。对于扩容而言,这个时间段为3分钟,缩容为5分钟(可以通过 --horizontal-pod-autoscaler-downscale-delay--horizontal-pod-autoscaler-upscale-delay 进行调整)。

  • HPA Controller中有一个tolerance(容忍力)的概念,它允许一定范围内的使用量的不稳定,现在默认为0.1,这也是出于维护系统稳定性的考虑。例如设定HPA调度策略为cpu使用率高于50%触发扩容,那么只有当使用率大于55%或者小于45%才会触发伸缩活动,HPA会尽力把Pod的使用率控制在这个范围之间。
  • 具体的每次扩容或者缩容的多少Pod的算法为:Ceil(前采集到的使用率 / 用户自定义的使用率) * Pod数量)。
  • 每次最大扩容pod数量不会超过当前副本数量的2倍。

环境说明

角色 IP 操作系统版本
master 192.168.1.201 centos 7.4
etcd1 192.168.1.201 centos 7.4
etcd2 192.168.1.202 centos 7.4
etcd3 192.168.1.203 centos 7.4
node1 192.168.1.204 centos 7.4
node2 192.168.1.205 centos 7.4
环境 软件版本
kubectl server v1.9.2
kubectl client v1.9.2
Go go1.9.2
etcdctl 3.2.15
etcd 3.2.15
flanneld v0.10.0
cfssl 1.2.0
docker 18.09.1-beta1
[root@master ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.1.201:6443
Heapster is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/heapster/proxy
monitoring-grafana is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
monitoring-influxdb is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@master ~]#
[root@master ~]# kubectl -s http://192.168.1.201:8080 get componentstatuses
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
etcd-2               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
[root@master ~]#
[root@master ~]# kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
192.168.1.204   Ready     <none>    21h       v1.9.2
192.168.1.205   Ready     <none>    21h       v1.9.2
[root@master ~]#

部署HPA

先准备一套K8S集群环境,环境部署略。

创建Deployment POD应用nginx

[root@master ~]# cat nginx.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx-hpa
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 0.01
            memory: 25Mi
          limits:
            cpu: 0.05
            memory: 60Mi
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx-hpa
spec:
  selector:
    app: nginx-hpa
  type: NodePort
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30080
[root@master ~]#
[root@master ~]# kubectl apply -f nginx.yml
[root@master ~]# kubectl get pod -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-5dcf548595-bk9cr   1/1       Running   1          14h       172.30.94.2   192.168.1.205
[root@master ~]#

创建nginx应用的HPA

[root@master ~]# cat nginx-hpa-cpu.yml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: nginx
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 70
[root@master ~]#
[root@master ~]# kubectl apply -f nginx-hpa-cpu.yml
[root@master ~]# kubectl get hpa
NAME        REFERENCE          TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   <unknown> / 70%   1         5         1          14h
[root@master ~]#

Q1

这时发现nginx-hpa获取不到当前的CPU情况(TARGETS)。等待几分钟后执行 kubectl describe hpa 发现HPA报错信息如下:

[root@master ~]# kubectl describe hpa
Name:                                                  nginx-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxReplic...
CreationTimestamp:                                     Sat, 26 Jan 2019 22:23:08 +0800
Reference:                                             Deployment/nginx
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 70%
Min replicas:                                          1
Max replicas:                                          5
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedComputeMetricsReplicas  1m (x12 over 3m)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
  Warning  FailedGetResourceMetric       1m (x13 over 3m) horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
[root@master ~]#

大概意思是HPA无法通过API获取到metrics值。

解决办法:

/etc/systemd/system/kube-controller-manager.service 配置文件中新增 --horizontal-pod-autoscaler-use-rest-clients=false 配置参数。然后重启kube-controller-manager服务即可。

kube-controller-manager's parameter --horizontal-pod-autoscaler-use-rest-clients in k8s 1.9.0 default value is true , while in k8s 1.8.x is false
change it to false and it works.
[root@master ~]# cat /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/usr/local/k8s/bin/kube-controller-manager \
  --address=127.0.0.1 \
  --master=http://192.168.1.201:8080 \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=172.16.0.0/16 \
  --cluster-cidr=172.30.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  --leader-elect=true \
  --horizontal-pod-autoscaler-use-rest-clients=false \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
[root@master ~]#
[root@master ~]# systemctl daemon-reload
[root@master ~]# systemctl restart kube-controller-manager

Q2

配置并重启完成kube-controller-manager服务后,执行 kubectl delete -f nginx-hpa-cpu.ymlkubectl apply -f nginx-hpa-cpu.yml 重新创建服务后,发现出现新的错误,信息如下:

[root@master ~]# kubectl describe hpa
Name:                                                  nginx-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
scaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec{"maxRepl...
CreationTimestamp:                                     Sun, 27 Jan 2019 00:18:02 +0800
Reference:                                             Deployment/nginx
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 70%
Min replicas:                                          1
Max replicas:                                          5
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics r resource cpu: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockoptconnection timed out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelect=app%3Dnginx-hpa'") has prevented the request from succeeding (get services http:heapster:)
Events:
  Type     Reason                        Age               From                       Message
  ----     ------                        ----              ----                       -------
  Warning  FailedUpdateStatus            2m                horizontal-pod-autoscaler  Operation cannot be fulfilled on hozontalpodautoscalers.autoscaling "nginx-hpa": the object has been modified; please apply your changes to the latest versi and try again
  Warning  FailedGetResourceMetric       24s (x3 over 4m)  horizontal-pod-autoscaler  unable to get metrics for resource u: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockopt: connection med out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginhpa'") has prevented the request from succeeding (get services http:heapster:)
  Warning  FailedComputeMetricsReplicas  24s (x3 over 4m)  horizontal-pod-autoscaler  failed to get cpu utilization: unab to get metrics for resource cpu: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/defaulpods?labelSelector=app%3Dnginx-hpa'") has prevented the request from succeeding (get services http:heapster:)
[root@master ~]#

意思是HPA无法连接heapster服务。于是检查heapster服务是否异常。

[root@master ~]# kubectl get pod -o wide -n kube-system
NAME                                   READY     STATUS    RESTARTS   AGE         IP           NODE
heapster-6d5c495969-2rgcr              1/1       Running   2          20h         172.30.9.4   192.168.1.204
kubernetes-dashboard-cbbf9945c-bkvbk   1/1       Running   2          20h         172.30.9.3   192.168.1.204
monitoring-grafana-67d68bf9c6-zv928    1/1       Running   2          20h         172.30.9.2   192.168.1.204
monitoring-influxdb-7c4c46745f-kbxgb   1/1       Running   0          <invalid>   172.30.9.5   192.168.1.204
[root@master ~]#

访问kube-dashboard发现POD是可以通过heapster获取到CPU内存的信息的。如下,说明heapster工作正常。

k8s集群水平扩展(HPA)

于是到node节点手动curl访问连接异常的URL。经测试在node1节点上访问正常。

[root@node1 ~]# curl 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginx-hpa'
{
  "metadata": {},
  "items": [
   {
    "metadata": {
     "name": "nginx-5dcf548595-bk9cr",
     "namespace": "default",
     "creationTimestamp": "2019-01-27T07:29:43Z"
    },
    "timestamp": "2019-01-27T07:29:00Z",
    "window": "1m0s",
    "containers": [
     {
      "name": "nginx",
      "usage": {
       "cpu": "0",
       "memory": "2820Ki"
      }
     }
    ]
   }
  ]
 }
 [root@node1 ~]#

于是到kube-master上访问测试,发现HPA无法访问到heapster。

[root@master ~]# curl 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginx-hpa'
curl: (7) Failed connect to 172.30.9.4:8082; Connection timed out
[root@master ~]#

接下来我们来测试下网络情况,发现kube-master无法Ping通heapster的POD地址。

[root@master ~]# ping 172.30.9.4
PING 172.30.9.4 (172.30.9.4) 56(84) bytes of data.
^C
--- 172.30.9.4 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1002ms

[root@master ~]# telnet 172.30.9.4 8082
Trying 172.30.9.4...
telnet: connect to address 172.30.9.4: Connection timed out
[root@master ~]#

测试发现是网络不通导致的。解决办法是在kube-master上安装flannel网络。

如果flannel网络的IP地址丢失,重启flannel网卡 systemctl restart flanneld 即可解决。

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:48:f6:1d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.201/24 brd 192.168.1.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::22d8:9dda:6705:ec09/64 scope link 
       valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN 
    link/ether 6e:05:c0:9c:34:3f brd ff:ff:ff:ff:ff:ff
    inet 172.30.13.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::6c05:c0ff:fe9c:343f/64 scope link 
       valid_lft forever preferred_lft forever
[root@localhost ~]#

再测试下kube-master到heapster POD的网络情况:

[root@master ~]# ping 172.30.9.4 -c 4
PING 172.30.9.4 (172.30.9.4) 56(84) bytes of data.
64 bytes from 172.30.9.4: icmp_seq=1 ttl=63 time=2.15 ms
64 bytes from 172.30.9.4: icmp_seq=2 ttl=63 time=1.27 ms
64 bytes from 172.30.9.4: icmp_seq=3 ttl=63 time=1.30 ms
64 bytes from 172.30.9.4: icmp_seq=4 ttl=63 time=1.66 ms

--- 172.30.9.4 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 1.277/1.599/2.150/0.354 ms
[root@master ~]# telnet 172.30.9.4 8082
Trying 172.30.9.4...
telnet: connect to address 172.30.9.4: Connection refused
[root@master ~]#

重新导入nginx-hpa-cpu.yml文件,然后等待几分钟…

[root@localhost ~]# kubectl delete -f nginx-hpa-cpu.yml
horizontalpodautoscaler "nginx-hpa" deleted
[root@localhost ~]#
[root@localhost ~]# kubectl apply -f nginx-hpa-cpu.yml
horizontalpodautoscaler "nginx-hpa" created
[root@localhost ~]#

OK,HPA连接heapster成功。

[root@localhost ~]# kubectl get hpa
NAME        REFERENCE          TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   0% / 70%   1         5         1          39s
[root@localhost ~]#
[root@localhost ~]# kubectl describe hpa
Name:                                                  nginx-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxRepl...
CreationTimestamp:                                     Sun, 27 Jan 2019 01:04:25 +0800
Reference:                                             Deployment/nginx
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (0) / 70%
Min replicas:                                          1
Max replicas:                                          5
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  the last scale time was sufficiently old as to warrant a new scale
  ScalingActive   True    ValidMetricFound  the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is increasing faster than the maximum scale rate
Events:           <none>
[root@localhost ~]#

HPA测试

截至目前,HPA支持的API版本有三个。分别是 autoscaling/v1autoscaling/v2beta1autoscaling/v2beta2 。其中 autoscaling/v1 只支持CPU一种伸缩指标;在 autoscaling/v2beta1 中增加支持custom metrics;在 autoscaling/v2beta2 中增加支持external metrics。

详细说明参考:

官方说明,在k8s 1.11版本,HPA将不再从heapster上获取指标。

The HorizontalPodAutoscaler normally fetches metrics from a series of aggregated APIs (metrics.k8s.io, custom.metrics.k8s.io, and external.metrics.k8s.io). The metrics.k8s.io API is usually provided by metrics-server, which needs to be launched separately. See metrics-server for instructions. The HorizontalPodAutoscaler can also fetch metrics directly from Heapster.

Note:
FEATURE STATE: Kubernetes 1.11 deprecated
Fetching metrics from Heapster is deprecated as of Kubernetes 1.11.

autoscaling/v1

[root@master ~]# cat nginx-hpa-cpu.yml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: nginx
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 70
[root@master ~]#

这里只针对CPU的HPA 压力测试。

压测命令

[root@node1 ~]# cat test.sh
while true
do
	wget -q -O- http://192.168.1.204:30080
done
[root@node1 ~]# sh test.sh

观察HPA当前负载和POD的情况

[root@master ~]# kubectl get hpa
NAME        REFERENCE          TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   0% / 70%   1         5         1          14h
[root@master ~]#
[root@master ~]# kubectl get hpa
NAME        REFERENCE          TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   14% / 70%   1         5         1          14h
[root@master ~]#

当负载飙升时,HPA会按照定义的规则开始创建新的POD副本(定义POD的CPU阈值为70%)。

[root@master ~]# kubectl get hpa
NAME        REFERENCE          TARGETS      MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   180% / 70%   1         5         3          14h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-5dcf548595-bk9cr   1/1       Running   1          15h       172.30.94.2   192.168.1.205
nginx-5dcf548595-pdndb   1/1       Running   0          1m        172.30.94.4   192.168.1.205
nginx-5dcf548595-z9d6h   1/1       Running   0          1m        172.30.94.3   192.168.1.205
[root@master ~]#

继续压测,会发现POD副本数量继续增加(REPLICAS从3到5)。

[root@master ~]# kubectl get hpa
NAME        REFERENCE          TARGETS      MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   139% / 70%   1         5         5          14h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME                     READY     STATUS              RESTARTS   AGE       IP            NODE
nginx-5dcf548595-9gmqf   0/1       ContainerCreating   0          39s       <none>        192.168.1.204
nginx-5dcf548595-bk9cr   1/1       Running             1          15h       172.30.94.2   192.168.1.205
nginx-5dcf548595-pdndb   1/1       Running             0          10m       172.30.94.4   192.168.1.205
nginx-5dcf548595-r7n4b   1/1       Running             0          39s       172.30.94.5   192.168.1.205
nginx-5dcf548595-z9d6h   1/1       Running             0          10m       172.30.94.3   192.168.1.205
[root@master ~]#

当REPLICAS达到定义的上限时,即使当前CPU的压力仍然很大,REPLICAS也不会再增加了。

[root@master ~]# kubectl get hpa
NAME        REFERENCE          TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   112% / 70%   1         5         5          14h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-5dcf548595-9gmqf   1/1       Running   0          2m        172.30.9.6    192.168.1.204
nginx-5dcf548595-bk9cr   1/1       Running   1          15h       172.30.94.2   192.168.1.205
nginx-5dcf548595-pdndb   1/1       Running   0          12m       172.30.94.4   192.168.1.205
nginx-5dcf548595-r7n4b   1/1       Running   0          2m        172.30.94.5   192.168.1.205
nginx-5dcf548595-z9d6h   1/1       Running   0          12m       172.30.94.3   192.168.1.205
[root@master ~]#

停止压测,当CPU负载降低时,HPA会自动减少POD的数量。

[root@master ~]# kubectl get hpa
NAME        REFERENCE          TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   40% / 70%   1         5         3          14h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-5dcf548595-pdndb   1/1       Running   0          16m       172.30.94.4   192.168.1.205
nginx-5dcf548595-r7n4b   1/1       Running   0          6m        172.30.94.5   192.168.1.205
nginx-5dcf548595-z9d6h   1/1       Running   0          16m       172.30.94.3   192.168.1.205
[root@master ~]#

慢慢的,HPA会减少POD的数量,直到降低到最小POD数(MINPODS)。

[root@master ~]# kubectl get hpa
NAME        REFERENCE          TARGETS      MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   0% / 70%   1         5         1          15h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-5dcf548595-z9d6h   1/1       Running   0          1h        172.30.94.3   192.168.1.205
[root@master ~]#

通过kube-dashboard观察这个过程的变化。

k8s集群水平扩展(HPA) k8s集群水平扩展(HPA) k8s集群水平扩展(HPA)

通过HPA的日志信息查看到它伸缩的过程。

[root@master ~]# kubectl describe hpa
Name:                                                  nginx-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxRepl...CreationTimestamp: Sun, 27 Jan 2019 01:04:25 +0800
Reference:                                             Deployment/nginx
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (0) / 70%
Min replicas:                                          1
Max replicas:                                          5
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     False   BackoffDownscale  the time since the previous scale is still within the downscale forbidden window
  ScalingActive   True    ValidMetricFound  the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request)  ScalingLimited  True    TooFewReplicas    the desired replica count is increasing faster than the maximum scale rate
Events:
  Type    Reason             Age               From                       Message
  ----    ------             ----              ----                       -------
  Normal  SuccessfulRescale  41m (x2 over 1h)  horizontal-pod-autoscaler  New size: 5; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  29m (x2 over 1h)  horizontal-pod-autoscaler  New size: 3; reason: All metrics below target
  Normal  SuccessfulRescale  17m               horizontal-pod-autoscaler  New size: 2; reason: All metrics below target
  Normal  SuccessfulRescale  8m (x2 over 1h)   horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  3m (x2 over 12m)  horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
[root@master ~]#

autoscaling/v2beta1

autoscaling/v2beta1 中增加支持custom metrics。

[root@master ~]# cat nginx-hpa-v2beta1.yml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: nginx
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: memory
        targetAverageUtilization: 70
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 70
[root@master ~]#
[root@master ~]# kubectl apply -f nginx-hpa-v2beta1.yml

等待几分钟后…

观察发现前面10%是内存的使用百分比,后面0%是CPU的使用百分比。

[root@master ~]# kubectl get hpa nginx-hpa
NAME        REFERENCE          TARGETS               MINPODS   MAXPODS   REPLICAS   AGE
nginx-hpa   Deployment/nginx   10% / 70%, 0% / 70%   1         5         1          51s
[root@master ~]#
[root@master ~]# kubectl describe hpa nginx-hpa
Name:                                                     nginx-hpa
Namespace:                                                default
Labels:                                                   <none>
Annotations:                                              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":
"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"ma...CreationTimestamp: Mon, 28 Jan 2019 22:22:01 +0800
Reference:                                                Deployment/nginx
Metrics:                                                  ( current / target )
  resource memory on pods  (as a percentage of request):  10% (2670592) / 70%
  resource cpu on pods  (as a percentage of request):     0% (0) / 70%
Min replicas:                                             1
Max replicas:                                             5
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
  ScalingActive   True    ValidMetricFound    the HPA was able to succesfully calculate a replica count from memory resou
rce utilization (percentage of request)  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           <none>

[root@master ~]#

autoscaling/v2beta2

autoscaling/v2beta2 测试发现目前k8s 1.9.2暂不支持这个API版本。

[root@master ~]# kubectl get hpa.v2beta2.autoscaling -o yaml
the server doesn't have a resource type "hpa" in group "v2beta2.autoscaling"
[root@master ~]#

以上所述就是小编给大家介绍的《k8s集群水平扩展(HPA)》,希望对大家有所帮助,如果大家有任何疑问请给我留言,小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持!

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

Principles of Object-Oriented JavaScript

Principles of Object-Oriented JavaScript

Nicholas C. Zakas / No Starch Press / 2014-2 / USD 24.95

If you've used a more traditional object-oriented language, such as C++ or Java, JavaScript probably doesn't seem object-oriented at all. It has no concept of classes, and you don't even need to defin......一起来看看 《Principles of Object-Oriented JavaScript》 这本书的介绍吧!

在线进制转换器
在线进制转换器

各进制数互转换器

Markdown 在线编辑器
Markdown 在线编辑器

Markdown 在线编辑器

正则表达式在线测试
正则表达式在线测试

正则表达式在线测试