内容简介:Horizontal Pod Autoscaling,简称HPA,是Kubernetes中实现POD水平自动伸缩的功能。K8S集群可以通过Replication Controller的scale机制完成服务的扩容或缩容,实现具有伸缩性的服务。K8S自动伸缩分为:
Horizontal Pod Autoscaling,简称HPA,是Kubernetes中实现POD水平自动伸缩的功能。
简介
K8S集群可以通过Replication Controller的scale机制完成服务的扩容或缩容,实现具有伸缩性的服务。
K8S自动伸缩分为:
- sacle手动伸缩。见 k8s滚动升级(RollingUpdate)
- autoscale自动伸缩,见HPA。
自动扩展主要分为两种:
- 水平扩展(scale out),针对于实例数目的增减。
- 垂直扩展(scal up),即单个实例可以使用的资源的增减, 比如增加cpu和增大内存。
HPA属于前者。它可以根据CPU使用率或应用自定义metrics自动扩展Pod数量(支持 replication controller、deployment 和 replica set)。
获取metrics的两种方式:
- Heapster:heapster提供metrics服务,但是在v1(autoscaling/v1)版本中仅支持以CPU作为扩展度量指标。而其他比如:内存,网络流量,qps等目前处于beta阶段(autoscaling/v2beta1)。
- Cousom:同样处于beta阶段(autoscaling/v2beta1),但是涉及到自定义的REST API的开发,复杂度会大一些,并且当需要从自定义的监控中获取数据时,只能设置绝对值,无法设置使用率。
工作流程
- 创建HPA资源,设定目标CPU使用率限额,以及最大/最小实例数,一定要设置Pod的资源限制参数:
request
,否则HPA不会工作。 - 控制管理器每隔30s(在
kube-controller-manager.service
中可以通过–horizontal-pod-autoscaler-sync-period
修改)查询metrics的资源使用情况。 - 然后与创建时设定的值和指标做对比(平均值之和/限额),求出目标调整的实例个数。
- 目标调整的实例数不能超过第一条中设定的最大/最小实例数。如果没有超过,则扩容;超过,则扩容至最大的实例个数。
- 重复第2-4步。
自动伸缩算法
HPA Controller会通过调整副本数量使得CPU使用率尽量向期望值靠近,而且不是完全相等。另官方考虑到自动扩展的决策可能需要一段时间才会生效:例如当pod所需要的CPU负荷过大,从而在创建一个新pod的过程中,系统的CPU使用量可能会同样在有一个攀升的过程。所以在每一次作出决策后的一段时间内,将不再进行扩展决策。对于扩容而言,这个时间段为3分钟,缩容为5分钟(可以通过 --horizontal-pod-autoscaler-downscale-delay
, --horizontal-pod-autoscaler-upscale-delay
进行调整)。
- HPA Controller中有一个tolerance(容忍力)的概念,它允许一定范围内的使用量的不稳定,现在默认为0.1,这也是出于维护系统稳定性的考虑。例如设定HPA调度策略为cpu使用率高于50%触发扩容,那么只有当使用率大于55%或者小于45%才会触发伸缩活动,HPA会尽力把Pod的使用率控制在这个范围之间。
- 具体的每次扩容或者缩容的多少Pod的算法为:Ceil(前采集到的使用率 / 用户自定义的使用率) * Pod数量)。
- 每次最大扩容pod数量不会超过当前副本数量的2倍。
环境说明
角色 | IP | 操作系统版本 |
---|---|---|
master | 192.168.1.201 | centos 7.4 |
etcd1 | 192.168.1.201 | centos 7.4 |
etcd2 | 192.168.1.202 | centos 7.4 |
etcd3 | 192.168.1.203 | centos 7.4 |
node1 | 192.168.1.204 | centos 7.4 |
node2 | 192.168.1.205 | centos 7.4 |
环境 | 软件版本 |
---|---|
kubectl server | v1.9.2 |
kubectl client | v1.9.2 |
Go | go1.9.2 |
etcdctl | 3.2.15 |
etcd | 3.2.15 |
flanneld | v0.10.0 |
cfssl | 1.2.0 |
docker | 18.09.1-beta1 |
[root@master ~]# kubectl cluster-info Kubernetes master is running at https://192.168.1.201:6443 Heapster is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/heapster/proxy monitoring-grafana is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy monitoring-influxdb is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. [root@master ~]#
[root@master ~]# kubectl -s http://192.168.1.201:8080 get componentstatuses NAME STATUS MESSAGE ERROR controller-manager Healthy ok etcd-2 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} scheduler Healthy ok etcd-0 Healthy {"health": "true"} [root@master ~]#
[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.1.204 Ready <none> 21h v1.9.2 192.168.1.205 Ready <none> 21h v1.9.2 [root@master ~]#
部署HPA
先准备一套K8S集群环境,环境部署略。
创建Deployment POD应用nginx
[root@master ~]# cat nginx.yml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx spec: replicas: 1 template: metadata: labels: app: nginx-hpa spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 name: http protocol: TCP resources: requests: cpu: 0.01 memory: 25Mi limits: cpu: 0.05 memory: 60Mi --- apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx-hpa spec: selector: app: nginx-hpa type: NodePort ports: - name: http protocol: TCP port: 80 targetPort: 80 nodePort: 30080 [root@master ~]#
[root@master ~]# kubectl apply -f nginx.yml [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-bk9cr 1/1 Running 1 14h 172.30.94.2 192.168.1.205 [root@master ~]#
创建nginx应用的HPA
[root@master ~]# cat nginx-hpa-cpu.yml apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: nginx-hpa spec: scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: nginx minReplicas: 1 maxReplicas: 5 targetCPUUtilizationPercentage: 70 [root@master ~]#
[root@master ~]# kubectl apply -f nginx-hpa-cpu.yml [root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx <unknown> / 70% 1 5 1 14h [root@master ~]#
Q1
这时发现nginx-hpa获取不到当前的CPU情况(TARGETS)。等待几分钟后执行 kubectl describe hpa
发现HPA报错信息如下:
[root@master ~]# kubectl describe hpa Name: nginx-hpa Namespace: default Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxReplic... CreationTimestamp: Sat, 26 Jan 2019 22:23:08 +0800 Reference: Deployment/nginx Metrics: ( current / target ) resource cpu on pods (as a percentage of request): <unknown> / 70% Min replicas: 1 Max replicas: 5 Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedComputeMetricsReplicas 1m (x12 over 3m) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io) Warning FailedGetResourceMetric 1m (x13 over 3m) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io) [root@master ~]#
大概意思是HPA无法通过API获取到metrics值。
解决办法:
在 /etc/systemd/system/kube-controller-manager.service
配置文件中新增 --horizontal-pod-autoscaler-use-rest-clients=false
配置参数。然后重启kube-controller-manager服务即可。
kube-controller-manager's parameter --horizontal-pod-autoscaler-use-rest-clients in k8s 1.9.0 default value is true , while in k8s 1.8.x is false change it to false and it works.
[root@master ~]# cat /etc/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/usr/local/k8s/bin/kube-controller-manager \ --address=127.0.0.1 \ --master=http://192.168.1.201:8080 \ --allocate-node-cidrs=true \ --service-cluster-ip-range=172.16.0.0/16 \ --cluster-cidr=172.30.0.0/16 \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \ --root-ca-file=/etc/kubernetes/ssl/ca.pem \ --leader-elect=true \ --horizontal-pod-autoscaler-use-rest-clients=false \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target [root@master ~]#
[root@master ~]# systemctl daemon-reload [root@master ~]# systemctl restart kube-controller-manager
Q2
配置并重启完成kube-controller-manager服务后,执行 kubectl delete -f nginx-hpa-cpu.yml
和 kubectl apply -f nginx-hpa-cpu.yml
重新创建服务后,发现出现新的错误,信息如下:
[root@master ~]# kubectl describe hpa Name: nginx-hpa Namespace: default Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au scaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec{"maxRepl... CreationTimestamp: Sun, 27 Jan 2019 00:18:02 +0800 Reference: Deployment/nginx Metrics: ( current / target ) resource cpu on pods (as a percentage of request): <unknown> / 70% Min replicas: 1 Max replicas: 5 Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics r resource cpu: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockoptconnection timed out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelect=app%3Dnginx-hpa'") has prevented the request from succeeding (get services http:heapster:) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedUpdateStatus 2m horizontal-pod-autoscaler Operation cannot be fulfilled on hozontalpodautoscalers.autoscaling "nginx-hpa": the object has been modified; please apply your changes to the latest versi and try again Warning FailedGetResourceMetric 24s (x3 over 4m) horizontal-pod-autoscaler unable to get metrics for resource u: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockopt: connection med out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginhpa'") has prevented the request from succeeding (get services http:heapster:) Warning FailedComputeMetricsReplicas 24s (x3 over 4m) horizontal-pod-autoscaler failed to get cpu utilization: unab to get metrics for resource cpu: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/defaulpods?labelSelector=app%3Dnginx-hpa'") has prevented the request from succeeding (get services http:heapster:) [root@master ~]#
意思是HPA无法连接heapster服务。于是检查heapster服务是否异常。
[root@master ~]# kubectl get pod -o wide -n kube-system NAME READY STATUS RESTARTS AGE IP NODE heapster-6d5c495969-2rgcr 1/1 Running 2 20h 172.30.9.4 192.168.1.204 kubernetes-dashboard-cbbf9945c-bkvbk 1/1 Running 2 20h 172.30.9.3 192.168.1.204 monitoring-grafana-67d68bf9c6-zv928 1/1 Running 2 20h 172.30.9.2 192.168.1.204 monitoring-influxdb-7c4c46745f-kbxgb 1/1 Running 0 <invalid> 172.30.9.5 192.168.1.204 [root@master ~]#
访问kube-dashboard发现POD是可以通过heapster获取到CPU内存的信息的。如下,说明heapster工作正常。
于是到node节点手动curl访问连接异常的URL。经测试在node1节点上访问正常。
[root@node1 ~]# curl 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginx-hpa' { "metadata": {}, "items": [ { "metadata": { "name": "nginx-5dcf548595-bk9cr", "namespace": "default", "creationTimestamp": "2019-01-27T07:29:43Z" }, "timestamp": "2019-01-27T07:29:00Z", "window": "1m0s", "containers": [ { "name": "nginx", "usage": { "cpu": "0", "memory": "2820Ki" } } ] } ] } [root@node1 ~]#
于是到kube-master上访问测试,发现HPA无法访问到heapster。
[root@master ~]# curl 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginx-hpa' curl: (7) Failed connect to 172.30.9.4:8082; Connection timed out [root@master ~]#
接下来我们来测试下网络情况,发现kube-master无法Ping通heapster的POD地址。
[root@master ~]# ping 172.30.9.4 PING 172.30.9.4 (172.30.9.4) 56(84) bytes of data. ^C --- 172.30.9.4 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1002ms [root@master ~]# telnet 172.30.9.4 8082 Trying 172.30.9.4... telnet: connect to address 172.30.9.4: Connection timed out [root@master ~]#
测试发现是网络不通导致的。解决办法是在kube-master上安装flannel网络。
如果flannel网络的IP地址丢失,重启flannel网卡 systemctl restart flanneld
即可解决。
[root@localhost ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:48:f6:1d brd ff:ff:ff:ff:ff:ff inet 192.168.1.201/24 brd 192.168.1.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::22d8:9dda:6705:ec09/64 scope link valid_lft forever preferred_lft forever 3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN link/ether 6e:05:c0:9c:34:3f brd ff:ff:ff:ff:ff:ff inet 172.30.13.0/32 scope global flannel.1 valid_lft forever preferred_lft forever inet6 fe80::6c05:c0ff:fe9c:343f/64 scope link valid_lft forever preferred_lft forever [root@localhost ~]#
再测试下kube-master到heapster POD的网络情况:
[root@master ~]# ping 172.30.9.4 -c 4 PING 172.30.9.4 (172.30.9.4) 56(84) bytes of data. 64 bytes from 172.30.9.4: icmp_seq=1 ttl=63 time=2.15 ms 64 bytes from 172.30.9.4: icmp_seq=2 ttl=63 time=1.27 ms 64 bytes from 172.30.9.4: icmp_seq=3 ttl=63 time=1.30 ms 64 bytes from 172.30.9.4: icmp_seq=4 ttl=63 time=1.66 ms --- 172.30.9.4 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3003ms rtt min/avg/max/mdev = 1.277/1.599/2.150/0.354 ms [root@master ~]# telnet 172.30.9.4 8082 Trying 172.30.9.4... telnet: connect to address 172.30.9.4: Connection refused [root@master ~]#
重新导入nginx-hpa-cpu.yml文件,然后等待几分钟…
[root@localhost ~]# kubectl delete -f nginx-hpa-cpu.yml horizontalpodautoscaler "nginx-hpa" deleted [root@localhost ~]# [root@localhost ~]# kubectl apply -f nginx-hpa-cpu.yml horizontalpodautoscaler "nginx-hpa" created [root@localhost ~]#
OK,HPA连接heapster成功。
[root@localhost ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 0% / 70% 1 5 1 39s [root@localhost ~]# [root@localhost ~]# kubectl describe hpa Name: nginx-hpa Namespace: default Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxRepl... CreationTimestamp: Sun, 27 Jan 2019 01:04:25 +0800 Reference: Deployment/nginx Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 0% (0) / 70% Min replicas: 1 Max replicas: 5 Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rate Events: <none> [root@localhost ~]#
HPA测试
截至目前,HPA支持的API版本有三个。分别是 autoscaling/v1
, autoscaling/v2beta1
, autoscaling/v2beta2
。其中 autoscaling/v1
只支持CPU一种伸缩指标;在 autoscaling/v2beta1
中增加支持custom metrics;在 autoscaling/v2beta2
中增加支持external metrics。
详细说明参考:
官方说明,在k8s 1.11版本,HPA将不再从heapster上获取指标。
The HorizontalPodAutoscaler normally fetches metrics from a series of aggregated APIs (metrics.k8s.io, custom.metrics.k8s.io, and external.metrics.k8s.io). The metrics.k8s.io API is usually provided by metrics-server, which needs to be launched separately. See metrics-server for instructions. The HorizontalPodAutoscaler can also fetch metrics directly from Heapster. Note: FEATURE STATE: Kubernetes 1.11 deprecated Fetching metrics from Heapster is deprecated as of Kubernetes 1.11.
autoscaling/v1
[root@master ~]# cat nginx-hpa-cpu.yml apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: nginx-hpa spec: scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: nginx minReplicas: 1 maxReplicas: 5 targetCPUUtilizationPercentage: 70 [root@master ~]#
这里只针对CPU的HPA 压力测试。
压测命令
[root@node1 ~]# cat test.sh while true do wget -q -O- http://192.168.1.204:30080 done [root@node1 ~]# sh test.sh
观察HPA当前负载和POD的情况
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 0% / 70% 1 5 1 14h [root@master ~]#
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 14% / 70% 1 5 1 14h [root@master ~]#
当负载飙升时,HPA会按照定义的规则开始创建新的POD副本(定义POD的CPU阈值为70%)。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 180% / 70% 1 5 3 14h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-bk9cr 1/1 Running 1 15h 172.30.94.2 192.168.1.205 nginx-5dcf548595-pdndb 1/1 Running 0 1m 172.30.94.4 192.168.1.205 nginx-5dcf548595-z9d6h 1/1 Running 0 1m 172.30.94.3 192.168.1.205 [root@master ~]#
继续压测,会发现POD副本数量继续增加(REPLICAS从3到5)。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 139% / 70% 1 5 5 14h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-9gmqf 0/1 ContainerCreating 0 39s <none> 192.168.1.204 nginx-5dcf548595-bk9cr 1/1 Running 1 15h 172.30.94.2 192.168.1.205 nginx-5dcf548595-pdndb 1/1 Running 0 10m 172.30.94.4 192.168.1.205 nginx-5dcf548595-r7n4b 1/1 Running 0 39s 172.30.94.5 192.168.1.205 nginx-5dcf548595-z9d6h 1/1 Running 0 10m 172.30.94.3 192.168.1.205 [root@master ~]#
当REPLICAS达到定义的上限时,即使当前CPU的压力仍然很大,REPLICAS也不会再增加了。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 112% / 70% 1 5 5 14h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-9gmqf 1/1 Running 0 2m 172.30.9.6 192.168.1.204 nginx-5dcf548595-bk9cr 1/1 Running 1 15h 172.30.94.2 192.168.1.205 nginx-5dcf548595-pdndb 1/1 Running 0 12m 172.30.94.4 192.168.1.205 nginx-5dcf548595-r7n4b 1/1 Running 0 2m 172.30.94.5 192.168.1.205 nginx-5dcf548595-z9d6h 1/1 Running 0 12m 172.30.94.3 192.168.1.205 [root@master ~]#
停止压测,当CPU负载降低时,HPA会自动减少POD的数量。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 40% / 70% 1 5 3 14h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-pdndb 1/1 Running 0 16m 172.30.94.4 192.168.1.205 nginx-5dcf548595-r7n4b 1/1 Running 0 6m 172.30.94.5 192.168.1.205 nginx-5dcf548595-z9d6h 1/1 Running 0 16m 172.30.94.3 192.168.1.205 [root@master ~]#
慢慢的,HPA会减少POD的数量,直到降低到最小POD数(MINPODS)。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 0% / 70% 1 5 1 15h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-z9d6h 1/1 Running 0 1h 172.30.94.3 192.168.1.205 [root@master ~]#
通过kube-dashboard观察这个过程的变化。
通过HPA的日志信息查看到它伸缩的过程。
[root@master ~]# kubectl describe hpa Name: nginx-hpa Namespace: default Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxRepl...CreationTimestamp: Sun, 27 Jan 2019 01:04:25 +0800 Reference: Deployment/nginx Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 0% (0) / 70% Min replicas: 1 Max replicas: 5 Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale False BackoffDownscale the time since the previous scale is still within the downscale forbidden window ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rate Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 41m (x2 over 1h) horizontal-pod-autoscaler New size: 5; reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 29m (x2 over 1h) horizontal-pod-autoscaler New size: 3; reason: All metrics below target Normal SuccessfulRescale 17m horizontal-pod-autoscaler New size: 2; reason: All metrics below target Normal SuccessfulRescale 8m (x2 over 1h) horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 3m (x2 over 12m) horizontal-pod-autoscaler New size: 1; reason: All metrics below target [root@master ~]#
autoscaling/v2beta1
autoscaling/v2beta1
中增加支持custom metrics。
[root@master ~]# cat nginx-hpa-v2beta1.yml apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: nginx-hpa spec: scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: nginx minReplicas: 1 maxReplicas: 5 metrics: - type: Resource resource: name: memory targetAverageUtilization: 70 - type: Resource resource: name: cpu targetAverageUtilization: 70 [root@master ~]#
[root@master ~]# kubectl apply -f nginx-hpa-v2beta1.yml
等待几分钟后…
观察发现前面10%是内存的使用百分比,后面0%是CPU的使用百分比。
[root@master ~]# kubectl get hpa nginx-hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 10% / 70%, 0% / 70% 1 5 1 51s [root@master ~]#
[root@master ~]# kubectl describe hpa nginx-hpa Name: nginx-hpa Namespace: default Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion": "autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"ma...CreationTimestamp: Mon, 28 Jan 2019 22:22:01 +0800 Reference: Deployment/nginx Metrics: ( current / target ) resource memory on pods (as a percentage of request): 10% (2670592) / 70% resource cpu on pods (as a percentage of request): 0% (0) / 70% Min replicas: 1 Max replicas: 5 Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from memory resou rce utilization (percentage of request) ScalingLimited False DesiredWithinRange the desired count is within the acceptable range Events: <none> [root@master ~]#
autoscaling/v2beta2
autoscaling/v2beta2
测试发现目前k8s 1.9.2暂不支持这个API版本。
[root@master ~]# kubectl get hpa.v2beta2.autoscaling -o yaml the server doesn't have a resource type "hpa" in group "v2beta2.autoscaling" [root@master ~]#
参考:
http://blog.51cto.com/ylw6006/2113848
https://blog.frognew.com/2017/01/kubernetes-pod-scale.html
https://k8smeetup.github.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
https://blog.csdn.net/qq_17016649/article/details/79297796
https://github.com/kubernetes/kubernetes/issues/57673附件:
HPA测试配置文件.zip以上所述就是小编给大家介绍的《k8s集群水平扩展(HPA)》,希望对大家有所帮助,如果大家有任何疑问请给我留言,小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持!
猜你喜欢:- Redis集群水平扩展(二)
- Redis集群水平扩展(一)
- Linux集群类型、系统扩展方式及调度方法
- 在 Kubernetes 集群中部署可扩展的 WordPress 站点
- Vitess 10 发布,MySQL 数据库集群水平扩展系统
- Vitess 4.0 发布,MySQL 数据库集群水平扩展系统
本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
Principles of Object-Oriented JavaScript
Nicholas C. Zakas / No Starch Press / 2014-2 / USD 24.95
If you've used a more traditional object-oriented language, such as C++ or Java, JavaScript probably doesn't seem object-oriented at all. It has no concept of classes, and you don't even need to defin......一起来看看 《Principles of Object-Oriented JavaScript》 这本书的介绍吧!