内容简介:自动扩展是一种根据资源使用情况自动扩展或缩小工作负载的方法。 Kubernetes中的自动缩放有两个维度:Cluster Autoscaler处理节点扩展操作,Horizontal Pod Autoscaler自动扩展部署或副本集中的pod数量。 Cluster Autoscaling与Horizontal Pod Autoscaler一起用于动态调整计算能力以及系统满足SLA所需的并行度。虽然Cluster Autoscaler高度依赖托管您的集群的云提供商的基础功能,但HPA可以独立于您的Iaa
k8s与HPA--通过 Prometheus adaptor 来自定义监控指标
自动扩展是一种根据资源使用情况自动扩展或缩小工作负载的方法。 Kubernetes中的自动缩放有两个维度:Cluster Autoscaler处理节点扩展操作,Horizontal Pod Autoscaler自动扩展部署或副本集中的pod数量。 Cluster Autoscaling与Horizontal Pod Autoscaler一起用于动态调整计算能力以及系统满足SLA所需的并行度。虽然Cluster Autoscaler高度依赖托管您的集群的云提供商的基础功能,但HPA可以独立于您的IaaS / PaaS提供商运营。
Horizontal Pod Autoscaler功能最初是在Kubernetes v1.1中引入的,并且从那时起已经发展了很多。 HPA缩放容器的版本1基于观察到的CPU利用率,后来基于内存使用情况。在Kubernetes 1.6中,引入了一个新的API Custom Metrics API,使HPA能够访问任意指标。 Kubernetes 1.7引入了聚合层,允许第三方应用程序通过将自己注册为API附加组件来扩展Kubernetes API。 Custom Metrics API和聚合层使Prometheus等监控系统可以向HPA控制器公开特定于应用程序的指标。
Horizontal Pod Autoscaler实现为一个控制循环,定期查询Resource Metrics API以获取CPU /内存等核心指标和针对特定应用程序指标的Custom Metrics API。
以下是为Kubernetes 1.9或更高版本配置HPA v2的分步指南。您将安装提供核心指标的Metrics Server附加组件,然后您将使用演示应用程序根据CPU和内存使用情况展示pod自动扩展。在本指南的第二部分中,您将部署Prometheus和自定义API服务器。您将使用聚合器层注册自定义API服务器,然后使用演示应用程序提供的自定义指标配置HPA。
在开始之前,您需要安装 Go 1.8或更高版本并在GOPATH中克隆k8s-prom-hpa repo。
cd $GOPATH git clone https://github.com/stefanprodan/k8s-prom-hpa
部署 Metrics Server
kubernetes Metrics Server是资源使用数据的集群范围聚合器,是Heapster的后继者。度量服务器通过汇集来自kubernetes.summary_api的数据来收集节点和pod的CPU和内存使用情况。摘要API是一种内存高效的API,用于将数据从Kubelet / cAdvisor传递到度量服务器。
在HPA的第一个版本中,您需要Heapster来提供CPU和内存指标,在HPA v2和Kubernetes 1.8中,只有在启用horizontal-pod-autoscaler-use-rest-clients时才需要指标服务器。默认情况下,Kubernetes 1.9中启用了HPA rest客户端。 GKE 1.9附带预安装的Metrics Server。
在kube-system命名空间中部署Metrics Server:
kubectl create -f ./metrics-server
一分钟后,度量服务器开始报告节点和pod的CPU和内存使用情况。
查看nodes metrics:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .
结果如下:
{ "kind": "NodeMetricsList", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes" }, "items": [ { "metadata": { "name": "ip-10-1-50-61.ec2.internal", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-50-61.ec2.internal", "creationTimestamp": "2019-02-13T08:34:05Z" }, "timestamp": "2019-02-13T08:33:38Z", "window": "30s", "usage": { "cpu": "78322168n", "memory": "563180Ki" } }, { "metadata": { "name": "ip-10-1-57-40.ec2.internal", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-57-40.ec2.internal", "creationTimestamp": "2019-02-13T08:34:05Z" }, "timestamp": "2019-02-13T08:33:42Z", "window": "30s", "usage": { "cpu": "48926263n", "memory": "554472Ki" } }, { "metadata": { "name": "ip-10-1-62-29.ec2.internal", "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-62-29.ec2.internal", "creationTimestamp": "2019-02-13T08:34:05Z" }, "timestamp": "2019-02-13T08:33:36Z", "window": "30s", "usage": { "cpu": "36700681n", "memory": "326088Ki" } } ] }
查看pods metrics:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq .
结果如下:
{ "kind": "PodMetricsList", "apiVersion": "metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/metrics.k8s.io/v1beta1/pods" }, "items": [ { "metadata": { "name": "kube-proxy-77nt2", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-77nt2", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:00Z", "window": "30s", "containers": [ { "name": "kube-proxy", "usage": { "cpu": "2370555n", "memory": "13184Ki" } } ] }, { "metadata": { "name": "cluster-autoscaler-n2xsl", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/cluster-autoscaler-n2xsl", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:12Z", "window": "30s", "containers": [ { "name": "cluster-autoscaler", "usage": { "cpu": "1477997n", "memory": "54584Ki" } } ] }, { "metadata": { "name": "core-dns-autoscaler-b4785d4d7-j64xd", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/core-dns-autoscaler-b4785d4d7-j64xd", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:08Z", "window": "30s", "containers": [ { "name": "autoscaler", "usage": { "cpu": "191293n", "memory": "7956Ki" } } ] }, { "metadata": { "name": "spot-interrupt-handler-8t2xk", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/spot-interrupt-handler-8t2xk", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:04Z", "window": "30s", "containers": [ { "name": "spot-interrupt-handler", "usage": { "cpu": "844907n", "memory": "4608Ki" } } ] }, { "metadata": { "name": "kube-proxy-t5kqm", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-t5kqm", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:08Z", "window": "30s", "containers": [ { "name": "kube-proxy", "usage": { "cpu": "1194766n", "memory": "12204Ki" } } ] }, { "metadata": { "name": "kube-proxy-zxmqb", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-zxmqb", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:06Z", "window": "30s", "containers": [ { "name": "kube-proxy", "usage": { "cpu": "3021117n", "memory": "13628Ki" } } ] }, { "metadata": { "name": "aws-node-rcz5c", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-rcz5c", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:15Z", "window": "30s", "containers": [ { "name": "aws-node", "usage": { "cpu": "1217989n", "memory": "24976Ki" } } ] }, { "metadata": { "name": "aws-node-z2qxs", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-z2qxs", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:15Z", "window": "30s", "containers": [ { "name": "aws-node", "usage": { "cpu": "1025780n", "memory": "46424Ki" } } ] }, { "metadata": { "name": "php-apache-899d75b96-8ppk4", "namespace": "default", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/php-apache-899d75b96-8ppk4", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:08Z", "window": "30s", "containers": [ { "name": "php-apache", "usage": { "cpu": "24612n", "memory": "27556Ki" } } ] }, { "metadata": { "name": "load-generator-779c5f458c-9sglg", "namespace": "default", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/load-generator-779c5f458c-9sglg", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:34:56Z", "window": "30s", "containers": [ { "name": "load-generator", "usage": { "cpu": "0", "memory": "336Ki" } } ] }, { "metadata": { "name": "aws-node-v9jxs", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-v9jxs", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:00Z", "window": "30s", "containers": [ { "name": "aws-node", "usage": { "cpu": "1303458n", "memory": "28020Ki" } } ] }, { "metadata": { "name": "kube2iam-m2ktt", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-m2ktt", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:11Z", "window": "30s", "containers": [ { "name": "kube2iam", "usage": { "cpu": "1328864n", "memory": "9724Ki" } } ] }, { "metadata": { "name": "kube2iam-w9cqf", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-w9cqf", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:03Z", "window": "30s", "containers": [ { "name": "kube2iam", "usage": { "cpu": "1294379n", "memory": "8812Ki" } } ] }, { "metadata": { "name": "custom-metrics-apiserver-657644489c-pk8rb", "namespace": "monitoring", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/monitoring/pods/custom-metrics-apiserver-657644489c-pk8rb", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:04Z", "window": "30s", "containers": [ { "name": "custom-metrics-apiserver", "usage": { "cpu": "22409370n", "memory": "42468Ki" } } ] }, { "metadata": { "name": "kube2iam-qghgt", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-qghgt", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:11Z", "window": "30s", "containers": [ { "name": "kube2iam", "usage": { "cpu": "2078992n", "memory": "16356Ki" } } ] }, { "metadata": { "name": "spot-interrupt-handler-ps745", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/spot-interrupt-handler-ps745", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:10Z", "window": "30s", "containers": [ { "name": "spot-interrupt-handler", "usage": { "cpu": "611566n", "memory": "4336Ki" } } ] }, { "metadata": { "name": "coredns-68fb7946fb-2xnpp", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-68fb7946fb-2xnpp", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:12Z", "window": "30s", "containers": [ { "name": "coredns", "usage": { "cpu": "1610381n", "memory": "10480Ki" } } ] }, { "metadata": { "name": "coredns-68fb7946fb-9ctjf", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-68fb7946fb-9ctjf", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:13Z", "window": "30s", "containers": [ { "name": "coredns", "usage": { "cpu": "1418850n", "memory": "9852Ki" } } ] }, { "metadata": { "name": "prometheus-7d4f6d4454-v4fnd", "namespace": "monitoring", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/monitoring/pods/prometheus-7d4f6d4454-v4fnd", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:00Z", "window": "30s", "containers": [ { "name": "prometheus", "usage": { "cpu": "17951807n", "memory": "202316Ki" } } ] }, { "metadata": { "name": "metrics-server-7cdd54ccb4-k2x7m", "namespace": "kube-system", "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/metrics-server-7cdd54ccb4-k2x7m", "creationTimestamp": "2019-02-13T08:35:19Z" }, "timestamp": "2019-02-13T08:35:04Z", "window": "30s", "containers": [ { "name": "metrics-server-nanny", "usage": { "cpu": "144656n", "memory": "5716Ki" } }, { "name": "metrics-server", "usage": { "cpu": "568327n", "memory": "16268Ki" } } ] } ] }
基于CPU和内存使用情况的Auto Scaling
您将使用基于Golang的小型Web应用程序来测试Horizontal Pod Autoscaler(HPA)。
将podinfo部署到默认命名空间:
kubectl create -f ./podinfo/podinfo-svc.yaml,./podinfo/podinfo-dep.yaml
使用NodePort服务访问podinfo,地址为http:// <K8S_PUBLIC_IP>:31198。
接下来定义一个至少维护两个副本的HPA,如果CPU平均值超过80%或内存超过200Mi,则最多可扩展到10个:
apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: podinfo spec: scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: podinfo minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 80 - type: Resource resource: name: memory targetAverageValue: 200Mi
创建这个hpa:
kubectl create -f ./podinfo/podinfo-hpa.yaml
几秒钟后,HPA控制器联系度量服务器,然后获取CPU和内存使用情况:
kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE podinfo Deployment/podinfo 2826240 / 200Mi, 15% / 80% 2 10 2 5m
为了增加CPU使用率,请使用rakyll / hey运行负载测试:
#install hey go get -u github.com/rakyll/hey #do 10K requests hey -n 10000 -q 10 -c 5 http://<K8S_PUBLIC_IP>:31198/
您可以使用以下方式监控HPA事件:
$ kubectl describe hpa Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 7m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 3m horizontal-pod-autoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target
暂时删除podinfo。稍后将在本教程中再次部署它:
kubectl delete -f ./podinfo/podinfo-hpa.yaml,./podinfo/podinfo-dep.yaml,./podinfo/podinfo-svc.yaml
部署 Custom Metrics Server
要根据自定义指标进行扩展,您需要拥有两个组件。一个组件,用于从应用程序收集指标并将其存储在Prometheus时间序列数据库中。第二个组件使用collect( k8s-prometheus-adapter )提供的指标扩展了Kubernetes自定义指标API。
您将在专用命名空间中部署Prometheus和适配器。
创建 monitoring 命名空间:
kubectl create -f ./namespaces.yaml
在 monitoring 命名空间中部署Prometheus v2:
kubectl create -f ./prometheus
生成Prometheus适配器所需的TLS证书:
make certs
生成以下几个文件:
# ls output apiserver.csr apiserver-key.pem apiserver.pem
部署Prometheus自定义指标API适配器:
kubectl create -f ./custom-metrics-api
列出Prometheus提供的自定义指标:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
获取 monitoring 命名空间中所有pod的FS使用情况:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/*/fs_usage_bytes" | jq .
查询结果如下:
{ "kind": "MetricValueList", "apiVersion": "custom.metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/%2A/fs_usage_bytes" }, "items": [ { "describedObject": { "kind": "Pod", "namespace": "monitoring", "name": "custom-metrics-apiserver-657644489c-pk8rb", "apiVersion": "/v1" }, "metricName": "fs_usage_bytes", "timestamp": "2019-02-13T08:52:30Z", "value": "94253056" }, { "describedObject": { "kind": "Pod", "namespace": "monitoring", "name": "prometheus-7d4f6d4454-v4fnd", "apiVersion": "/v1" }, "metricName": "fs_usage_bytes", "timestamp": "2019-02-13T08:52:30Z", "value": "24576" } ] }
基于custom metrics 自动伸缩
在默认命名空间中创建podinfo NodePort服务和部署:
kubectl create -f ./podinfo/podinfo-svc.yaml,./podinfo/podinfo-dep.yaml
podinfo应用程序公开名为http_requests_total的自定义指标。 Prometheus适配器删除_total后缀并将度量标记为计数器度量标准。
从自定义指标API获取每秒的总请求数:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .
{ "kind": "MetricValueList", "apiVersion": "custom.metrics.k8s.io/v1beta1", "metadata": { "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests" }, "items": [ { "describedObject": { "kind": "Pod", "namespace": "default", "name": "podinfo-6b86c8ccc9-kv5g9", "apiVersion": "/__internal" }, "metricName": "http_requests", "timestamp": "2018-01-10T16:49:07Z", "value": "901m" }, { "describedObject": { "kind": "Pod", "namespace": "default", "name": "podinfo-6b86c8ccc9-nm7bl", "apiVersion": "/__internal" }, "metricName": "http_requests", "timestamp": "2018-01-10T16:49:07Z", "value": "898m" } ] }
建一个HPA,如果请求数超过每秒10个,将扩展podinfo部署:
apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: podinfo spec: scaleTargetRef: apiVersion: extensions/v1beta1 kind: Deployment name: podinfo minReplicas: 2 maxReplicas: 10 metrics: - type: Pods pods: metricName: http_requests targetAverageValue: 10
在默认命名空间中部署podinfo HPA:
kubectl create -f ./podinfo/podinfo-hpa-custom.yaml
几秒钟后,HPA从指标API获取http_requests值:
kubectl get hpa NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE podinfo Deployment/podinfo 899m / 10 2 10 2 1m
在podinfo服务上应用一些负载,每秒25个请求:
#install hey go get -u github.com/rakyll/hey #do 10K requests rate limited at 25 QPS hey -n 10000 -q 5 -c 5 http://<K8S-IP>:31198/healthz
几分钟后,HPA开始扩展部署:
kubectl describe hpa Name: podinfo Namespace: default Reference: Deployment/podinfo Metrics: ( current / target ) "http_requests" on pods: 9059m / 10 Min replicas: 2 Max replicas: 10 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 2m horizontal-pod-autoscaler New size: 3; reason: pods metric http_requests above target
按照当前的每秒请求速率,部署永远不会达到10个pod的最大值。三个复制品足以使每个吊舱的RPS保持在10以下。
负载测试完成后,HPA会将部署缩到其初始副本:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 3; reason: pods metric http_requests above target Normal SuccessfulRescale 21s horizontal-pod-autoscaler New size: 2; reason: All metrics below target
您可能已经注意到自动缩放器不会立即对使用峰值做出反应。默认情况下,度量标准同步每30秒发生一次,只有在最后3-5分钟内没有重新缩放时才能进行扩展/缩小。通过这种方式,HPA可以防止快速执行冲突的决策,并为Cluster Autoscaler提供时间。
结论
并非所有系统都可以通过单独依赖CPU /内存使用指标来满足其SLA,大多数Web和移动后端需要基于每秒请求进行自动扩展以处理任何流量突发。对于ETL应用程序,可以通过作业队列长度超过某个阈值等来触发自动缩放。通过使用Prometheus检测应用程序并公开正确的自动缩放指标,您可以对应用程序进行微调,以更好地处理突发并确保高可用性。
以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持 码农网
猜你喜欢:- zabbix监控tomcat 自定义监控项
- 监控之路5-zabbix定义一次完整的监控
- Zabbix自定义监控脚本配置详解
- zabbix 日志监控(自定义插件开发)
- 分布式监控系统 WGCLOUD 更新,支持自定义告警脚本
- Zabbix监控Nginx和fpm(网站并发数)自定义key
本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们。
Python标准库
Doug Hellmann / 刘炽 / 机械工业出版社华章公司 / 2012-6-15 / 139.00元
本书由资深Python专家亲自执笔,Python语言的核心开发人员作序推荐,权威性毋庸置疑。 对于程序员而言,标准库与语言本身同样重要,它好比一个百宝箱,能为各种常见的任务提供完美的解决方案,所以本书是所有Python程序员都必备的工具书!本书以案例驱动的方式讲解了标准库中一百多个模块的使用方法(如何工作)和工作原理(为什么要这样工作),比标准库的官方文档更容易理解(一个简单的示例比一份手册......一起来看看 《Python标准库》 这本书的介绍吧!