使用自定义指标进行Pod弹性伸缩

Horizontal Pod Autoscaling，简称HPA，是Kubernetes中实现Pod水平自动伸缩功能的资源对象。Kubernetes 1.9 或更高版本支持HPA V2版本，允许我们是用自定义监控指标来控制Pod数目。
本文用阿里云容器服务来介绍HPA使用自定义指标进行弹性伸缩的功能。

安装配置Prometheus和Prometheus adaptor

我们使用Prometheus收集指标，通过Prometheus adaptor来扩展Kubernetes 自定义监控指标。
使用阿里云容器服务控制台创建好集群，并创建monitoring命名空间。然后进行如下步骤：

安装Prometheus

在应用目录-App Hub中搜索prometheus可以看到如下组件

点击prometheus进入安装页面，点击参数进行查看并配置（本文persistentVolume enabled设置为false，即使用emptyDir），选择集群命名空间后点击安装。

安装Prometheus adaptor

安装过程同上，注意配置参数中prometheus url要使用之前部署好的prometheus地址

验证安装是否成功

使用控制台的cloudshell或kubectl连接集群，执行如下命令：

kubectl api-versions

结果中可以看到如下内容，表明集群支持hpa v2并可以使用自定义监控指标

autoscaling/v2beta1custom.metrics.k8s.io/v1beta1

我们可以进一步查看监控项，如pod支持的监控指标

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" |jq . | grep "pods/"

      "name": "pods/start_time_seconds",      "name": "pods/cpu_user",      "name": "pods/fs_limit_bytes",      "name": "pods/cpu_cfs_throttled",      "name": "pods/memory_failcnt",      "name": "pods/fs_writes_bytes",      "name": "pods/kube_pod_created",      "name": "pods/kube_pod_status_phase",      "name": "pods/tasks_state",      "name": "pods/last_seen",      "name": "pods/kube_pod_container_status_running",      "name": "pods/fs_reads_merged",      "name": "pods/memory_swap",      "name": "pods/spec_cpu_quota",      "name": "pods/fs_io_current",      "name": "pods/kube_pod_container_resource_requests_memory_bytes",      "name": "pods/kube_pod_container_status_waiting",      "name": "pods/fs_sector_reads",      "name": "pods/kube_pod_status_scheduled",      "name": "pods/kube_pod_container_status_waiting_reason",      "name": "pods/fs_reads_bytes",      "name": "pods/kube_pod_container_resource_requests_cpu_cores",      "name": "pods/kube_pod_container_status_terminated_reason",      "name": "pods/cpu_system",      "name": "pods/fs_read",      "name": "pods/fs_writes",      "name": "pods/kube_pod_owner",      "name": "pods/memory_mapped_file",      "name": "pods/spec_cpu_shares",      "name": "pods/kube_pod_container_resource_limits_cpu_cores",      "name": "pods/cpu_usage",      "name": "pods/kube_pod_container_resource_limits",      "name": "pods/kube_pod_info",      "name": "pods/memory_working_set_bytes",      "name": "pods/fs_writes_merged",      "name": "pods/memory_usage_bytes",      "name": "pods/kube_pod_container_resource_requests",      "name": "pods/fs_io_time_weighted",      "name": "pods/spec_cpu_period",      "name": "pods/spec_memory_reservation_limit_bytes",      "name": "pods/cpu_cfs_throttled_periods",      "name": "pods/kube_pod_container_resource_limits_memory_bytes",      "name": "pods/cpu_cfs_periods",      "name": "pods/kube_pod_status_ready",      "name": "pods/kube_pod_container_status_terminated",      "name": "pods/memory_failures",      "name": "pods/memory_rss",      "name": "pods/spec_memory_swap_limit_bytes",      "name": "pods/memory_cache",      "name": "pods/kube_pod_start_time",      "name": "pods/fs_io_time",      "name": "pods/fs_reads",      "name": "pods/fs_inodes_free",      "name": "pods/kube_pod_container_status_ready",      "name": "pods/fs_sector_writes",      "name": "pods/cpu_load_average_10s",      "name": "pods/spec_memory_limit_bytes",      "name": "pods/kube_pod_labels",      "name": "pods/kube_pod_container_status_restarts",      "name": "pods/kube_pod_container_info",      "name": "pods/fs_usage_bytes",      "name": "pods/kube_pod_completion_time",      "name": "pods/kube_pod_container_status_last_terminated_reason",      "name": "pods/kube_pod_status_scheduled_time",      "name": "pods/memory_max_usage_bytes",      "name": "pods/fs_inodes",      "name": "pods/fs_write",

部署Deployment

在容器服务控制台中点击无状态->使用镜像创建，创建并部署一个nginx deployment，配置如下

在访问设置中配置服务

创建成功后可以在无状态列表中找到名为nginx的deployment，点击详情可以查看到相关信息

创建HPA

在上述详情页中点击容器组水平伸缩器，点击创建，配置相关内容（指标使用pods，参考上述自定义监控指标内容填写一项，这里使用memory_usage_bytes，阈值10485760=10M）

创建成功后可以在列表中查看到nginx-hpa

测试并触发自动伸缩

使用压测工具ab进行压测

for a in `seq 1 50`; do ab -rSqd -c 100 -n 2000 http://ng.cfc7d1315bdd14d529a7f102c2163736d.cn-beijing.alicontainer.com/;done

等待一段时间，可以看到pod数量的变化

总结

通过 Prometheus 来监控应用程序并暴露出用于弹性伸缩的指标，使用HPA利用这些监控指标对应用进行弹性伸缩，可以应对一些突发状况，提高应用的可用性。