内容简介：Horizontal Pod Autoscaling，简称HPA，是Kubernetes中实现POD水平自动伸缩的功能。K8S集群可以通过Replication Controller的scale机制完成服务的扩容或缩容，实现具有伸缩性的服务。K8S自动伸缩分为：
Horizontal Pod Autoscaling，简称HPA，是Kubernetes中实现POD水平自动伸缩的功能。
简介
K8S集群可以通过Replication Controller的scale机制完成服务的扩容或缩容，实现具有伸缩性的服务。
K8S自动伸缩分为：
- sacle手动伸缩。见 k8s滚动升级(RollingUpdate)
- autoscale自动伸缩，见HPA。
自动扩展主要分为两种：
- 水平扩展(scale out)，针对于实例数目的增减。
- 垂直扩展(scal up)，即单个实例可以使用的资源的增减, 比如增加cpu和增大内存。
HPA属于前者。它可以根据CPU使用率或应用自定义metrics自动扩展Pod数量(支持 replication controller、deployment 和 replica set)。
获取metrics的两种方式：
- Heapster：heapster提供metrics服务，但是在v1(autoscaling/v1)版本中仅支持以CPU作为扩展度量指标。而其他比如：内存，网络流量，qps等目前处于beta阶段(autoscaling/v2beta1)。
- Cousom：同样处于beta阶段(autoscaling/v2beta1)，但是涉及到自定义的REST API的开发，复杂度会大一些，并且当需要从自定义的监控中获取数据时，只能设置绝对值，无法设置使用率。
工作流程
- 创建HPA资源，设定目标CPU使用率限额，以及最大/最小实例数，一定要设置Pod的资源限制参数:
request，否则HPA不会工作。 - 控制管理器每隔30s(在
kube-controller-manager.service中可以通过–horizontal-pod-autoscaler-sync-period修改)查询metrics的资源使用情况。 - 然后与创建时设定的值和指标做对比(平均值之和/限额)，求出目标调整的实例个数。
- 目标调整的实例数不能超过第一条中设定的最大/最小实例数。如果没有超过，则扩容；超过，则扩容至最大的实例个数。
- 重复第2-4步。
自动伸缩算法
HPA Controller会通过调整副本数量使得CPU使用率尽量向期望值靠近，而且不是完全相等。另官方考虑到自动扩展的决策可能需要一段时间才会生效：例如当pod所需要的CPU负荷过大，从而在创建一个新pod的过程中，系统的CPU使用量可能会同样在有一个攀升的过程。所以在每一次作出决策后的一段时间内，将不再进行扩展决策。对于扩容而言，这个时间段为3分钟，缩容为5分钟(可以通过 --horizontal-pod-autoscaler-downscale-delay ， --horizontal-pod-autoscaler-upscale-delay 进行调整)。
- HPA Controller中有一个tolerance（容忍力）的概念，它允许一定范围内的使用量的不稳定，现在默认为0.1，这也是出于维护系统稳定性的考虑。例如设定HPA调度策略为cpu使用率高于50%触发扩容，那么只有当使用率大于55%或者小于45%才会触发伸缩活动，HPA会尽力把Pod的使用率控制在这个范围之间。
- 具体的每次扩容或者缩容的多少Pod的算法为：Ceil(前采集到的使用率 / 用户自定义的使用率) * Pod数量)。
- 每次最大扩容pod数量不会超过当前副本数量的2倍。
环境说明
| 角色 | IP | 操作系统版本 |
|---|---|---|
| master | 192.168.1.201 | centos 7.4 |
| etcd1 | 192.168.1.201 | centos 7.4 |
| etcd2 | 192.168.1.202 | centos 7.4 |
| etcd3 | 192.168.1.203 | centos 7.4 |
| node1 | 192.168.1.204 | centos 7.4 |
| node2 | 192.168.1.205 | centos 7.4 |
| 环境 | 软件版本 |
|---|---|
| kubectl server | v1.9.2 |
| kubectl client | v1.9.2 |
| Go | go1.9.2 |
| etcdctl | 3.2.15 |
| etcd | 3.2.15 |
| flanneld | v0.10.0 |
| cfssl | 1.2.0 |
| docker | 18.09.1-beta1 |
[root@master ~]# kubectl cluster-info Kubernetes master is running at https://192.168.1.201:6443 Heapster is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/heapster/proxy monitoring-grafana is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy monitoring-influxdb is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. [root@master ~]#
[root@master ~]# kubectl -s http://192.168.1.201:8080 get componentstatuses
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
[root@master ~]#
[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION 192.168.1.204 Ready <none> 21h v1.9.2 192.168.1.205 Ready <none> 21h v1.9.2 [root@master ~]#
部署HPA
先准备一套K8S集群环境，环境部署略。
创建Deployment POD应用nginx
[root@master ~]# cat nginx.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
template:
metadata:
labels:
app: nginx-hpa
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
name: http
protocol: TCP
resources:
requests:
cpu: 0.01
memory: 25Mi
limits:
cpu: 0.05
memory: 60Mi
---
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx-hpa
spec:
selector:
app: nginx-hpa
type: NodePort
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
nodePort: 30080
[root@master ~]#
[root@master ~]# kubectl apply -f nginx.yml [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-bk9cr 1/1 Running 1 14h 172.30.94.2 192.168.1.205 [root@master ~]#
创建nginx应用的HPA
[root@master ~]# cat nginx-hpa-cpu.yml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx
minReplicas: 1
maxReplicas: 5
targetCPUUtilizationPercentage: 70
[root@master ~]#
[root@master ~]# kubectl apply -f nginx-hpa-cpu.yml [root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx <unknown> / 70% 1 5 1 14h [root@master ~]#
Q1
这时发现nginx-hpa获取不到当前的CPU情况（TARGETS）。等待几分钟后执行 kubectl describe hpa 发现HPA报错信息如下：
[root@master ~]# kubectl describe hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxReplic...
CreationTimestamp: Sat, 26 Jan 2019 22:23:08 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 1m (x12 over 3m) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Warning FailedGetResourceMetric 1m (x13 over 3m) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
[root@master ~]#
大概意思是HPA无法通过API获取到metrics值。
解决办法：
在 /etc/systemd/system/kube-controller-manager.service 配置文件中新增 --horizontal-pod-autoscaler-use-rest-clients=false 配置参数。然后重启kube-controller-manager服务即可。
kube-controller-manager's parameter --horizontal-pod-autoscaler-use-rest-clients in k8s 1.9.0 default value is true , while in k8s 1.8.x is false change it to false and it works.
[root@master ~]# cat /etc/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/usr/local/k8s/bin/kube-controller-manager \ --address=127.0.0.1 \ --master=http://192.168.1.201:8080 \ --allocate-node-cidrs=true \ --service-cluster-ip-range=172.16.0.0/16 \ --cluster-cidr=172.30.0.0/16 \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \ --root-ca-file=/etc/kubernetes/ssl/ca.pem \ --leader-elect=true \ --horizontal-pod-autoscaler-use-rest-clients=false \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target [root@master ~]#
[root@master ~]# systemctl daemon-reload [root@master ~]# systemctl restart kube-controller-manager
Q2
配置并重启完成kube-controller-manager服务后，执行 kubectl delete -f nginx-hpa-cpu.yml 和 kubectl apply -f nginx-hpa-cpu.yml 重新创建服务后，发现出现新的错误，信息如下：
[root@master ~]# kubectl describe hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
scaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec{"maxRepl...
CreationTimestamp: Sun, 27 Jan 2019 00:18:02 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics r resource cpu: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockoptconnection timed out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelect=app%3Dnginx-hpa'") has prevented the request from succeeding (get services http:heapster:)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedUpdateStatus 2m horizontal-pod-autoscaler Operation cannot be fulfilled on hozontalpodautoscalers.autoscaling "nginx-hpa": the object has been modified; please apply your changes to the latest versi and try again
Warning FailedGetResourceMetric 24s (x3 over 4m) horizontal-pod-autoscaler unable to get metrics for resource u: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockopt: connection med out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginhpa'") has prevented the request from succeeding (get services http:heapster:)
Warning FailedComputeMetricsReplicas 24s (x3 over 4m) horizontal-pod-autoscaler failed to get cpu utilization: unab to get metrics for resource cpu: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/defaulpods?labelSelector=app%3Dnginx-hpa'") has prevented the request from succeeding (get services http:heapster:)
[root@master ~]#
意思是HPA无法连接heapster服务。于是检查heapster服务是否异常。
[root@master ~]# kubectl get pod -o wide -n kube-system NAME READY STATUS RESTARTS AGE IP NODE heapster-6d5c495969-2rgcr 1/1 Running 2 20h 172.30.9.4 192.168.1.204 kubernetes-dashboard-cbbf9945c-bkvbk 1/1 Running 2 20h 172.30.9.3 192.168.1.204 monitoring-grafana-67d68bf9c6-zv928 1/1 Running 2 20h 172.30.9.2 192.168.1.204 monitoring-influxdb-7c4c46745f-kbxgb 1/1 Running 0 <invalid> 172.30.9.5 192.168.1.204 [root@master ~]#
访问kube-dashboard发现POD是可以通过heapster获取到CPU内存的信息的。如下，说明heapster工作正常。
于是到node节点手动curl访问连接异常的URL。经测试在node1节点上访问正常。
[root@node1 ~]# curl 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginx-hpa'
{
"metadata": {},
"items": [
{
"metadata": {
"name": "nginx-5dcf548595-bk9cr",
"namespace": "default",
"creationTimestamp": "2019-01-27T07:29:43Z"
},
"timestamp": "2019-01-27T07:29:00Z",
"window": "1m0s",
"containers": [
{
"name": "nginx",
"usage": {
"cpu": "0",
"memory": "2820Ki"
}
}
]
}
]
}
[root@node1 ~]#
于是到kube-master上访问测试，发现HPA无法访问到heapster。
[root@master ~]# curl 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginx-hpa' curl: (7) Failed connect to 172.30.9.4:8082; Connection timed out [root@master ~]#
接下来我们来测试下网络情况，发现kube-master无法Ping通heapster的POD地址。
[root@master ~]# ping 172.30.9.4 PING 172.30.9.4 (172.30.9.4) 56(84) bytes of data. ^C --- 172.30.9.4 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1002ms [root@master ~]# telnet 172.30.9.4 8082 Trying 172.30.9.4... telnet: connect to address 172.30.9.4: Connection timed out [root@master ~]#
测试发现是网络不通导致的。解决办法是在kube-master上安装flannel网络。
如果flannel网络的IP地址丢失，重启flannel网卡 systemctl restart flanneld 即可解决。
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:48:f6:1d brd ff:ff:ff:ff:ff:ff
inet 192.168.1.201/24 brd 192.168.1.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::22d8:9dda:6705:ec09/64 scope link
valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
link/ether 6e:05:c0:9c:34:3f brd ff:ff:ff:ff:ff:ff
inet 172.30.13.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::6c05:c0ff:fe9c:343f/64 scope link
valid_lft forever preferred_lft forever
[root@localhost ~]#
再测试下kube-master到heapster POD的网络情况：
[root@master ~]# ping 172.30.9.4 -c 4 PING 172.30.9.4 (172.30.9.4) 56(84) bytes of data. 64 bytes from 172.30.9.4: icmp_seq=1 ttl=63 time=2.15 ms 64 bytes from 172.30.9.4: icmp_seq=2 ttl=63 time=1.27 ms 64 bytes from 172.30.9.4: icmp_seq=3 ttl=63 time=1.30 ms 64 bytes from 172.30.9.4: icmp_seq=4 ttl=63 time=1.66 ms --- 172.30.9.4 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3003ms rtt min/avg/max/mdev = 1.277/1.599/2.150/0.354 ms [root@master ~]# telnet 172.30.9.4 8082 Trying 172.30.9.4... telnet: connect to address 172.30.9.4: Connection refused [root@master ~]#
重新导入nginx-hpa-cpu.yml文件，然后等待几分钟…
[root@localhost ~]# kubectl delete -f nginx-hpa-cpu.yml horizontalpodautoscaler "nginx-hpa" deleted [root@localhost ~]# [root@localhost ~]# kubectl apply -f nginx-hpa-cpu.yml horizontalpodautoscaler "nginx-hpa" created [root@localhost ~]#
OK，HPA连接heapster成功。
[root@localhost ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 0% / 70% 1 5 1 39s
[root@localhost ~]#
[root@localhost ~]# kubectl describe hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxRepl...
CreationTimestamp: Sun, 27 Jan 2019 01:04:25 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (0) / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rate
Events: <none>
[root@localhost ~]#
HPA测试
截至目前，HPA支持的API版本有三个。分别是 autoscaling/v1 ， autoscaling/v2beta1 ， autoscaling/v2beta2 。其中 autoscaling/v1 只支持CPU一种伸缩指标；在 autoscaling/v2beta1 中增加支持custom metrics；在 autoscaling/v2beta2 中增加支持external metrics。
详细说明参考：
官方说明，在k8s 1.11版本，HPA将不再从heapster上获取指标。
The HorizontalPodAutoscaler normally fetches metrics from a series of aggregated APIs (metrics.k8s.io, custom.metrics.k8s.io, and external.metrics.k8s.io). The metrics.k8s.io API is usually provided by metrics-server, which needs to be launched separately. See metrics-server for instructions. The HorizontalPodAutoscaler can also fetch metrics directly from Heapster. Note: FEATURE STATE: Kubernetes 1.11 deprecated Fetching metrics from Heapster is deprecated as of Kubernetes 1.11.
autoscaling/v1
[root@master ~]# cat nginx-hpa-cpu.yml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx
minReplicas: 1
maxReplicas: 5
targetCPUUtilizationPercentage: 70
[root@master ~]#
这里只针对CPU的HPA 压力测试。
压测命令
[root@node1 ~]# cat test.sh while true do wget -q -O- http://192.168.1.204:30080 done [root@node1 ~]# sh test.sh
观察HPA当前负载和POD的情况
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 0% / 70% 1 5 1 14h [root@master ~]#
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 14% / 70% 1 5 1 14h [root@master ~]#
当负载飙升时，HPA会按照定义的规则开始创建新的POD副本(定义POD的CPU阈值为70%)。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 180% / 70% 1 5 3 14h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-bk9cr 1/1 Running 1 15h 172.30.94.2 192.168.1.205 nginx-5dcf548595-pdndb 1/1 Running 0 1m 172.30.94.4 192.168.1.205 nginx-5dcf548595-z9d6h 1/1 Running 0 1m 172.30.94.3 192.168.1.205 [root@master ~]#
继续压测，会发现POD副本数量继续增加（REPLICAS从3到5）。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 139% / 70% 1 5 5 14h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-9gmqf 0/1 ContainerCreating 0 39s <none> 192.168.1.204 nginx-5dcf548595-bk9cr 1/1 Running 1 15h 172.30.94.2 192.168.1.205 nginx-5dcf548595-pdndb 1/1 Running 0 10m 172.30.94.4 192.168.1.205 nginx-5dcf548595-r7n4b 1/1 Running 0 39s 172.30.94.5 192.168.1.205 nginx-5dcf548595-z9d6h 1/1 Running 0 10m 172.30.94.3 192.168.1.205 [root@master ~]#
当REPLICAS达到定义的上限时，即使当前CPU的压力仍然很大，REPLICAS也不会再增加了。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 112% / 70% 1 5 5 14h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-9gmqf 1/1 Running 0 2m 172.30.9.6 192.168.1.204 nginx-5dcf548595-bk9cr 1/1 Running 1 15h 172.30.94.2 192.168.1.205 nginx-5dcf548595-pdndb 1/1 Running 0 12m 172.30.94.4 192.168.1.205 nginx-5dcf548595-r7n4b 1/1 Running 0 2m 172.30.94.5 192.168.1.205 nginx-5dcf548595-z9d6h 1/1 Running 0 12m 172.30.94.3 192.168.1.205 [root@master ~]#
停止压测，当CPU负载降低时，HPA会自动减少POD的数量。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 40% / 70% 1 5 3 14h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-pdndb 1/1 Running 0 16m 172.30.94.4 192.168.1.205 nginx-5dcf548595-r7n4b 1/1 Running 0 6m 172.30.94.5 192.168.1.205 nginx-5dcf548595-z9d6h 1/1 Running 0 16m 172.30.94.3 192.168.1.205 [root@master ~]#
慢慢的，HPA会减少POD的数量，直到降低到最小POD数(MINPODS)。
[root@master ~]# kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 0% / 70% 1 5 1 15h [root@master ~]# [root@master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-5dcf548595-z9d6h 1/1 Running 0 1h 172.30.94.3 192.168.1.205 [root@master ~]#
通过kube-dashboard观察这个过程的变化。
通过HPA的日志信息查看到它伸缩的过程。
[root@master ~]# kubectl describe hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxRepl...CreationTimestamp: Sun, 27 Jan 2019 01:04:25 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (0) / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale False BackoffDownscale the time since the previous scale is still within the downscale forbidden window
ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rate
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 41m (x2 over 1h) horizontal-pod-autoscaler New size: 5; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 29m (x2 over 1h) horizontal-pod-autoscaler New size: 3; reason: All metrics below target
Normal SuccessfulRescale 17m horizontal-pod-autoscaler New size: 2; reason: All metrics below target
Normal SuccessfulRescale 8m (x2 over 1h) horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 3m (x2 over 12m) horizontal-pod-autoscaler New size: 1; reason: All metrics below target
[root@master ~]#
autoscaling/v2beta1
autoscaling/v2beta1 中增加支持custom metrics。
[root@master ~]# cat nginx-hpa-v2beta1.yml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: memory
targetAverageUtilization: 70
- type: Resource
resource:
name: cpu
targetAverageUtilization: 70
[root@master ~]#
[root@master ~]# kubectl apply -f nginx-hpa-v2beta1.yml
等待几分钟后…
观察发现前面10%是内存的使用百分比，后面0%是CPU的使用百分比。
[root@master ~]# kubectl get hpa nginx-hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE nginx-hpa Deployment/nginx 10% / 70%, 0% / 70% 1 5 1 51s [root@master ~]#
[root@master ~]# kubectl describe hpa nginx-hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":
"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"ma...CreationTimestamp: Mon, 28 Jan 2019 22:22:01 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource memory on pods (as a percentage of request): 10% (2670592) / 70%
resource cpu on pods (as a percentage of request): 0% (0) / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from memory resou
rce utilization (percentage of request) ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events: <none>
[root@master ~]#
autoscaling/v2beta2
autoscaling/v2beta2 测试发现目前k8s 1.9.2暂不支持这个API版本。
[root@master ~]# kubectl get hpa.v2beta2.autoscaling -o yaml the server doesn't have a resource type "hpa" in group "v2beta2.autoscaling" [root@master ~]#
参考：
http://blog.51cto.com/ylw6006/2113848
https://blog.frognew.com/2017/01/kubernetes-pod-scale.html
https://k8smeetup.github.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
https://blog.csdn.net/qq_17016649/article/details/79297796
https://github.com/kubernetes/kubernetes/issues/57673附件：
HPA测试配置文件.zip以上所述就是小编给大家介绍的《k8s集群水平扩展(HPA)》，希望对大家有所帮助，如果大家有任何疑问请给我留言，小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持！
猜你喜欢:
- Redis集群水平扩展（二）
- Redis集群水平扩展（一）
- Linux集群类型、系统扩展方式及调度方法
- 在 Kubernetes 集群中部署可扩展的 WordPress 站点
- Vitess 10 发布，MySQL 数据库集群水平扩展系统
- Vitess 4.0 发布，MySQL 数据库集群水平扩展系统
本站部分资源来源于网络，本站转载出于传递更多信息之目的，版权归原作者或者来源机构所有，如转载稿涉及版权问题，请联系我们。
Data Structures and Algorithm Analysis in Java
Mark A. Weiss / Pearson / 2006-3-3 / USD 143.00
As the speed and power of computers increases, so does the need for effective programming and algorithm analysis. By approaching these skills in tandem, Mark Allen Weiss teaches readers to develop wel......一起来看看 《Data Structures and Algorithm Analysis in Java》 这本书的介绍吧!
