内容简介:自动扩展是一种根据资源使用情况自动扩展或缩小工作负载的方法。 Kubernetes中的自动缩放有两个维度:Cluster Autoscaler处理节点扩展操作,Horizontal Pod Autoscaler自动扩展部署或副本集中的pod数量。 Cluster Autoscaling与Horizontal Pod Autoscaler一起用于动态调整计算能力以及系统满足SLA所需的并行度。虽然Cluster Autoscaler高度依赖托管您的集群的云提供商的基础功能,但HPA可以独立于您的Iaa
k8s与HPA--通过 Prometheus adaptor 来自定义监控指标
自动扩展是一种根据资源使用情况自动扩展或缩小工作负载的方法。 Kubernetes中的自动缩放有两个维度:Cluster Autoscaler处理节点扩展操作,Horizontal Pod Autoscaler自动扩展部署或副本集中的pod数量。 Cluster Autoscaling与Horizontal Pod Autoscaler一起用于动态调整计算能力以及系统满足SLA所需的并行度。虽然Cluster Autoscaler高度依赖托管您的集群的云提供商的基础功能,但HPA可以独立于您的IaaS / PaaS提供商运营。
Horizontal Pod Autoscaler功能最初是在Kubernetes v1.1中引入的,并且从那时起已经发展了很多。 HPA缩放容器的版本1基于观察到的CPU利用率,后来基于内存使用情况。在Kubernetes 1.6中,引入了一个新的API Custom Metrics API,使HPA能够访问任意指标。 Kubernetes 1.7引入了聚合层,允许第三方应用程序通过将自己注册为API附加组件来扩展Kubernetes API。 Custom Metrics API和聚合层使Prometheus等监控系统可以向HPA控制器公开特定于应用程序的指标。
Horizontal Pod Autoscaler实现为一个控制循环,定期查询Resource Metrics API以获取CPU /内存等核心指标和针对特定应用程序指标的Custom Metrics API。
以下是为Kubernetes 1.9或更高版本配置HPA v2的分步指南。您将安装提供核心指标的Metrics Server附加组件,然后您将使用演示应用程序根据CPU和内存使用情况展示pod自动扩展。在本指南的第二部分中,您将部署Prometheus和自定义API服务器。您将使用聚合器层注册自定义API服务器,然后使用演示应用程序提供的自定义指标配置HPA。
在开始之前,您需要安装 Go 1.8或更高版本并在GOPATH中克隆k8s-prom-hpa repo。
cd $GOPATH git clone https://github.com/stefanprodan/k8s-prom-hpa
部署 Metrics Server
kubernetes Metrics Server是资源使用数据的集群范围聚合器,是Heapster的后继者。度量服务器通过汇集来自kubernetes.summary_api的数据来收集节点和pod的CPU和内存使用情况。摘要API是一种内存高效的API,用于将数据从Kubelet / cAdvisor传递到度量服务器。
在HPA的第一个版本中,您需要Heapster来提供CPU和内存指标,在HPA v2和Kubernetes 1.8中,只有在启用horizontal-pod-autoscaler-use-rest-clients时才需要指标服务器。默认情况下,Kubernetes 1.9中启用了HPA rest客户端。 GKE 1.9附带预安装的Metrics Server。
在kube-system命名空间中部署Metrics Server:
kubectl create -f ./metrics-server
一分钟后,度量服务器开始报告节点和pod的CPU和内存使用情况。
查看nodes metrics:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .
结果如下:
{ "kind": "NodeMetricsList", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes" }, "items": [ { "metadata": { "name": "ip-10-1-50-61.ec2.internal", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-50-61.ec2.internal", "creationTimestamp": "2019-02-13T08:34:05Z" }, "timestamp": "2019-02-13T08:33:38Z", "window": "30s", "usage": { "cpu": "78322168n", "memory": "563180Ki" } }, { "metadata": { "name": "ip-10-1-57-40.ec2.internal", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-57-40.ec2.internal", "creationTimestamp": "2019-02-13T08:34:05Z" }, "timestamp": "2019-02-13T08:33:42Z", "window": "30s", "usage": { "cpu": "48926263n", "memory": "554472Ki" } }, { "metadata": { "name": "ip-10-1-62-29.ec2.internal", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-62-29.ec2.internal", "creationTimestamp": "2019-02-13T08:34:05Z" }, "timestamp": "2019-02-13T08:33:36Z", "window": "30s", "usage": { "cpu": "36700681n", "memory": "326088Ki" } } ] }
查看pods metrics:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq .
结果如下:
{ "kind": "PodMetricsList", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/metrics.k8s.io/v1beta1/pods" }, "items": [ { "metadata": { "name": "kube-proxy-77nt2", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-77nt2", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:00Z", "window": "30s", "containers": [ { "name": "kube-proxy", "usage": { "cpu": "2370555n", "memory": "13184Ki" } } ] }, { "metadata": { "name": "cluster-autoscaler-n2xsl", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/cluster-autoscaler-n2xsl", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:12Z", "window": "30s", "containers": [ { "name": "cluster-autoscaler", "usage": { "cpu": "1477997n", "memory": "54584Ki" } } ] }, { "metadata": { "name": "core-dns-autoscaler-b4785d4d7-j64xd", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/core-dns-autoscaler-b4785d4d7-j64xd", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:08Z", "window": "30s", "containers": [ { "name": "autoscaler", "usage": { "cpu": "191293n", "memory": "7956Ki" } } ] }, { "metadata": { "name": "spot-interrupt-handler-8t2xk", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/spot-interrupt-handler-8t2xk", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:04Z", "window": "30s", "containers": [ { "name": "spot-interrupt-handler", "usage": { "cpu": "844907n", "memory": "4608Ki" } } ] }, { "metadata": { "name": "kube-proxy-t5kqm", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-t5kqm", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:08Z", "window": "30s", "containers": [ { "name": "kube-proxy", "usage": { "cpu": "1194766n", "memory": "12204Ki" } } ] }, { "metadata": { "name": "kube-proxy-zxmqb", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-zxmqb", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:06Z", "window": "30s", "containers": [ { "name": "kube-proxy", "usage": { "cpu": "3021117n", "memory": "13628Ki" } } ] }, { "metadata": { "name": "aws-node-rcz5c", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-rcz5c", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:15Z", "window": "30s", "containers": [ { "name": "aws-node", "usage": { "cpu": "1217989n", "memory": "24976Ki" } } ] }, { "metadata": { "name": "aws-node-z2qxs", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-z2qxs", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:15Z", "window": "30s", "containers": [ { "name": "aws-node", "usage": { "cpu": "1025780n", "memory": "46424Ki" } } ] }, { "metadata": { "name": "php-apache-899d75b96-8ppk4", "namespace": "default", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/php-apache-899d75b96-8ppk4", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:08Z", "window": "30s", "containers": [ { "name": "php-apache", "usage": { "cpu": "24612n", "memory": "27556Ki" } } ] }, { "metadata": { "name": "load-generator-779c5f458c-9sglg", "namespace": "default", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/load-generator-779c5f458c-9sglg", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:34:56Z", "window": "30s", "containers": [ { "name": "load-generator", "usage": { "cpu": "0", "memory": "336Ki" } } ] }, { "metadata": { "name": "aws-node-v9jxs", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-v9jxs", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:00Z", "window": "30s", "containers": [ { "name": "aws-node", "usage": { "cpu": "1303458n", "memory": "28020Ki" } } ] }, { "metadata": { "name": "kube2iam-m2ktt", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-m2ktt", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:11Z", "window": "30s", "containers": [ { "name": "kube2iam", "usage": { "cpu": "1328864n", "memory": "9724Ki" } } ] }, { "metadata": { "name": "kube2iam-w9cqf", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-w9cqf", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:03Z", "window": "30s", "containers": [ { "name": "kube2iam", "usage": { "cpu": "1294379n", "memory": "8812Ki" } } ] }, { "metadata": { "name": "custom-metrics-apiserver-657644489c-pk8rb", "namespace": "monitoring", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/monitoring/pods/custom-metrics-apiserver-657644489c-pk8rb", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:04Z", "window": "30s", "containers": [ { "name": "custom-metrics-apiserver", "usage": { "cpu": "22409370n", "memory": "42468Ki" } } ] }, { "metadata": { "name": "kube2iam-qghgt", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-qghgt", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:11Z", "window": "30s", "containers": [ { "name": "kube2iam", "usage": { "cpu": "2078992n", "memory": "16356Ki" } } ] }, { "metadata": { "name": "spot-interrupt-handler-ps745", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/spot-interrupt-handler-ps745", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:10Z", "window": "30s", "containers": [ { "name": "spot-interrupt-handler", "usage": { "cpu": "611566n", "memory": "4336Ki" } } ] }, { "metadata": { "name": "coredns-68fb7946fb-2xnpp", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-68fb7946fb-2xnpp", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:12Z", "window": "30s", "containers": [ { "name": "coredns", "usage": { "cpu": "1610381n", "memory": "10480Ki" } } ] }, { "metadata": { "name": "coredns-68fb7946fb-9ctjf", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-68fb7946fb-9ctjf", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:13Z", "window": "30s", "containers": [ { "name": "coredns", "usage": { "cpu": "1418850n", "memory": "9852Ki" } } ] }, { "metadata": { "name": "prometheus-7d4f6d4454-v4fnd", "namespace": "monitoring", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/monitoring/pods/prometheus-7d4f6d4454-v4fnd", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:00Z", "window": "30s", "containers": [ { "name": "prometheus", "usage": { "cpu": "17951807n", "memory": "202316Ki" } } ] }, { "metadata": { "name": "metrics-server-7cdd54ccb4-k2x7m", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/metrics-server-7cdd54ccb4-k2x7m", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:04Z", "window": "30s", "containers": [ { "name": "metrics-server-nanny", "usage": { "cpu": "144656n", "memory": "5716Ki" } }, { "name": "metrics-server", "usage": { "cpu": "568327n", "memory": "16268Ki" } } ] } ] }
基于CPU和内存使用情况的Auto Scaling
您将使用基于Golang的小型Web应用程序来测试Horizontal Pod Autoscaler(HPA)。
将podinfo部署到默认命名空间:
kubectl create -f ./podinfo/podinfo-svc.yaml,./podinfo/podinfo-dep.yaml
使用NodePort服务访问podinfo,地址为http:// <K8S_PUBLIC_IP>:31198。
接下来定义一个至少维护两个副本的HPA,如果CPU平均值超过80%或内存超过200Mi,则最多可扩展到10个:
apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: podinfo spec: scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: podinfo minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 80 - type: Resource resource: name: memory targetAverageValue: 200Mi
创建这个hpa:
kubectl create -f ./podinfo/podinfo-hpa.yaml
几秒钟后,HPA控制器联系度量服务器,然后获取CPU和内存使用情况:
kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE podinfo Deployment/podinfo 2826240 / 200Mi, 15% / 80% 2 10 2 5m
为了增加CPU使用率,请使用rakyll / hey运行负载测试:
#install hey go get -u github.com/rakyll/hey #do 10K requests hey -n 10000 -q 10 -c 5 http://<K8S_PUBLIC_IP>:31198/
您可以使用以下方式监控HPA事件:
$ kubectl describe hpa Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 7m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 3m horizontal-pod-autoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target
暂时删除podinfo。稍后将在本教程中再次部署它:
kubectl delete -f ./podinfo/podinfo-hpa.yaml,./podinfo/podinfo-dep.yaml,./podinfo/podinfo-svc.yaml
部署 Custom Metrics Server
要根据自定义指标进行扩展,您需要拥有两个组件。一个组件,用于从应用程序收集指标并将其存储在Prometheus时间序列数据库中。第二个组件使用collect( k8s-prometheus-adapter )提供的指标扩展了Kubernetes自定义指标API。
您将在专用命名空间中部署Prometheus和适配器。
创建 monitoring 命名空间:
kubectl create -f ./namespaces.yaml
在 monitoring 命名空间中部署Prometheus v2:
kubectl create -f ./prometheus
生成Prometheus适配器所需的TLS证书:
make certs
生成以下几个文件:
# ls output apiserver.csr apiserver-key.pem apiserver.pem
部署Prometheus自定义指标API适配器:
kubectl create -f ./custom-metrics-api
列出Prometheus提供的自定义指标:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
获取 monitoring 命名空间中所有pod的FS使用情况:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/*/fs_usage_bytes" | jq .
查询结果如下:
{ "kind": "MetricValueList", "apiVersion": "custom.metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/%2A/fs_usage_bytes" }, "items": [ { "describedObject": { "kind": "Pod", "namespace": "monitoring", "name": "custom-metrics-apiserver-657644489c-pk8rb", "apiVersion": "/v1" }, "metricName": "fs_usage_bytes", "timestamp": "2019-02-13T08:52:30Z", "value": "94253056" }, { "describedObject": { "kind": "Pod", "namespace": "monitoring", "name": "prometheus-7d4f6d4454-v4fnd", "apiVersion": "/v1" }, "metricName": "fs_usage_bytes", "timestamp": "2019-02-13T08:52:30Z", "value": "24576" } ] }
基于custom metrics 自动伸缩
在默认命名空间中创建podinfo NodePort服务和部署:
kubectl create -f ./podinfo/podinfo-svc.yaml,./podinfo/podinfo-dep.yaml
podinfo应用程序公开名为http_requests_total的自定义指标。 Prometheus适配器删除_total后缀并将度量标记为计数器度量标准。
从自定义指标API获取每秒的总请求数:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
{ "kind": "MetricValueList", "apiVersion": "custom.metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests" }, "items": [ { "describedObject": { "kind": "Pod", "namespace": "default", "name": "podinfo-6b86c8ccc9-kv5g9", "apiVersion": "/__internal" }, "metricName": "http_requests", "timestamp": "2018-01-10T16:49:07Z", "value": "901m" }, { "describedObject": { "kind": "Pod", "namespace": "default", "name": "podinfo-6b86c8ccc9-nm7bl", "apiVersion": "/__internal" }, "metricName": "http_requests", "timestamp": "2018-01-10T16:49:07Z", "value": "898m" } ] }
建一个HPA,如果请求数超过每秒10个,将扩展podinfo部署:
apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: podinfo spec: scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: podinfo minReplicas: 2 maxReplicas: 10 metrics: - type: Pods pods: metricName: http_requests targetAverageValue: 10
在默认命名空间中部署podinfo HPA:
kubectl create -f ./podinfo/podinfo-hpa-custom.yaml
几秒钟后,HPA从指标API获取http_requests值:
kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE podinfo Deployment/podinfo 899m / 10 2 10 2 1m
在podinfo服务上应用一些负载,每秒25个请求:
#install hey go get -u github.com/rakyll/hey #do 10K requests rate limited at 25 QPS hey -n 10000 -q 5 -c 5 http://<K8S-IP>:31198/healthz
几分钟后,HPA开始扩展部署:
kubectl describe hpa Name: podinfo Namespace: default Reference: Deployment/podinfo Metrics: ( current / target ) "http_requests" on pods: 9059m / 10 Min replicas: 2 Max replicas: 10 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 2m horizontal-pod-autoscaler New size: 3; reason: pods metric http_requests above target
按照当前的每秒请求速率,部署永远不会达到10个pod的最大值。三个复制品足以使每个吊舱的RPS保持在10以下。
负载测试完成后,HPA会将部署缩到其初始副本:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 3; reason: pods metric http_requests above target Normal SuccessfulRescale 21s horizontal-pod-autoscaler New size: 2; reason: All metrics below target
您可能已经注意到自动缩放器不会立即对使用峰值做出反应。默认情况下,度量标准同步每30秒发生一次,只有在最后3-5分钟内没有重新缩放时才能进行扩展/缩小。通过这种方式,HPA可以防止快速执行冲突的决策,并为Cluster Autoscaler提供时间。
结论
并非所有系统都可以通过单独依赖CPU /内存使用指标来满足其SLA,大多数Web和移动后端需要基于每秒请求进行自动扩展以处理任何流量突发。对于ETL应用程序,可以通过作业队列长度超过某个阈值等来触发自动缩放。通过使用Prometheus检测应用程序并公开正确的自动缩放指标,您可以对应用程序进行微调,以更好地处理突发并确保高可用性。
以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持 码农网
猜你喜欢:- zabbix监控tomcat 自定义监控项
- 监控之路5-zabbix定义一次完整的监控
- Zabbix自定义监控脚本配置详解
- zabbix 日志监控(自定义插件开发)
- 分布式监控系统 WGCLOUD 更新,支持自定义告警脚本
- Zabbix监控Nginx和fpm(网站并发数)自定义key
本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
Numerical Methods and Methods of Approximation in Science and En
Karan Surana / CRC Press / 2018-10-31
ABOUT THIS BOOK Numerical Methods and Methods of Approximation in Science and Engineering prepares students and other readers for advanced studies involving applied numerical and computational anal......一起来看看 《Numerical Methods and Methods of Approximation in Science and En》 这本书的介绍吧!