玄德公笔记

prometheus监控k8s的metric详解-01（apiserver部分）

文章目录

1. apiserver admission（准入）和audit（审计）
- 1.1 准入控制器调用次数
- - 1.1.1 `apiserver_admission_controller_admission_duration_seconds_count`
  - 1.1.2 `apiserver_admission_controller_admission_duration_seconds_bucket`
  - 1.1.3 `apiserver_admission_controller_admission_duration_seconds_sum`
- 1.2 step（准入步骤）
- - 1.2.1 `apiserver_admission_step_admission_duration_seconds_count`
  - 1.2.2 `apiserver_admission_step_admission_duration_seconds_bucket`
  - 1.2.3 `apiserver_admission_step_admission_duration_seconds_sum`
- 1.3 summary（摘要）
- - 1.3.1 `apiserver_admission_step_admission_duration_seconds_summary`
  - 1.3.2 `apiserver_admission_step_admission_duration_seconds_summary_count`
  - 1.3.3 `apiserver_admission_step_admission_duration_seconds_summary_sum`
- 1.4 webhook
- - 1.4.1 `apiserver_admission_webhook_admission_duration_seconds_count`
  - 1.4.2 `apiserver_admission_webhook_admission_duration_seconds_bucket`
  - 1.4.5 `apiserver_admission_webhook_admission_duration_seconds_sum`
  - 1.4.6 `apiserver_admission_webhook_rejection_count`
  - 1.4.7 `apiserver_admission_webhook_request_total`
- 1.5 审计
- - 1.5.1 `apiserver_audit_event_total`
  - 1.5.2 `apiserver_audit_requests_rejected_total`
2. apiserver delegated（委托）
- 2.1 delegated authn（委托认证）
- - 2.1.1 `apiserver_delegated_authn_request_duration_seconds_count`
  - 2.1.2 `apiserver_delegated_authn_request_duration_seconds_bucket`
  - 2.1.3 `apiserver_delegated_authn_request_duration_seconds_sum`
  - 2.1.4 `apiserver_delegated_authn_request_total`
- 2.2 delegated authz（委托授权）
- - 2.2.1 `apiserver_delegated_authz_request_duration_seconds_count`
  - 2.2.2 `apiserver_delegated_authz_request_duration_seconds_bucket`
  - 2.2.3 `apiserver_delegated_authz_request_duration_seconds_sum`
- 2.2.5 `apiserver_delegated_authz_request_total`
3. apiserver flowcontrol
- 3.1 请求和速率（待整理）
- - 3.1.1 `apiserver_flowcontrol_current_executing_requests`
  - 3.1.2 `apiserver_flowcontrol_current_inqueue_requests`
  - 3.1.3 `apiserver_flowcontrol_current_r`（未验证）
  - 3.1.4 `apiserver_flowcontrol_dispatched_requests_total`
  - 3.1.5 `apiserver_flowcontrol_dispatch_r`（未验证）
  - 3.1.6 `apiserver_flowcontrol_latest_s`
  - 3.1.7 `apiserver_flowcontrol_next_discounted_s_bounds`
  - 3.1.8 `apiserver_flowcontrol_next_s_bounds`
- 3.2 priority level（优先级别）
- - 3.2.2 `apiserver_flowcontrol_priority_level_request_count_samples_count`
  - 3.2.1 `apiserver_flowcontrol_priority_level_request_count_samples_bucket`
  - 3.2.3 `apiserver_flowcontrol_priority_level_request_count_samples_sum`
  - 3.2.4 `apiserver_flowcontrol_priority_level_request_count_watermarks_count`
  - 3.2.5 `apiserver_flowcontrol_priority_level_request_count_watermarks_bucket`
  - 3.2.6 `apiserver_flowcontrol_priority_level_request_count_watermarks_sum`
  - 3.2.7 `apiserver_flowcontrol_priority_level_seat_count_samples_count`
  - 3.2.8 `apiserver_flowcontrol_priority_level_seat_count_samples_bucket`
  - 3.2.9 `apiserver_flowcontrol_priority_level_seat_count_samples_sum`
  - 3.2.12 `apiserver_flowcontrol_priority_level_seat_count_watermarks_count`
  - 3.2.11 `apiserver_flowcontrol_priority_level_seat_count_watermarks_bucket`
  - 3.2.13 `apiserver_flowcontrol_priority_level_seat_count_watermarks_sum`
- 3.3 读写请求
- - 3.3.1 `apiserver_flowcontrol_read_vs_write_request_count_samples_count`
  - 3.3.2 `apiserver_flowcontrol_read_vs_write_request_count_samples_bucket`
  - 3.3.3 `apiserver_flowcontrol_read_vs_write_request_count_samples_sum`
  - 3.3.4 `apiserver_flowcontrol_read_vs_write_request_count_watermarks_count`
  - 3.3.5 `apiserver_flowcontrol_read_vs_write_request_count_watermarks_bucket`
  - 3.3.6 `apiserver_flowcontrol_read_vs_write_request_count_watermarks_sum`
- 3.4 request（请求）
- - 3.4.1 `apiserver_flowcontrol_request_concurrency_in_use`
  - 3.4.2 `apiserver_flowcontrol_request_concurrency_limit`
  - 3.4.3 `apiserver_flowcontrol_request_execution_seconds_count`
  - 3.4.4 `apiserver_flowcontrol_request_execution_seconds_bucket`
  - 3.4.5 `apiserver_flowcontrol_request_execution_seconds_sum`
  - 3.4.6 `apiserver_flowcontrol_request_queue_length_after_enqueue_count`
  - 3.4.7 `apiserver_flowcontrol_request_queue_length_after_enqueue_bucket`
  - 3.4.8 `apiserver_flowcontrol_request_queue_length_after_enqueue_sum`
  - 3.4.10 `apiserver_flowcontrol_request_wait_duration_seconds_count`
  - 3.4.9 `apiserver_flowcontrol_request_wait_duration_seconds_bucket`
  - 3.4.12 `apiserver_flowcontrol_request_wait_duration_seconds_sum`
4. request 和 response
- 4.1 请求次数
- - 4.1.1 `apiserver_request_duration_seconds_count`
  - 4.1.2 `apiserver_request_duration_seconds_bucket`
  - 4.1.3 `apiserver_request_duration_seconds_sum`
  - 4.1.4 `apiserver_current_inflight_requests`
  - 4.1.5 `apiserver_current_inqueue_requests`
- 4.2 过滤器
- - 4.2.1 `apiserver_request_filter_duration_seconds_count`
  - 4.2.2 `apiserver_request_filter_duration_seconds_bucket`
  - 4.2.3 `apiserver_request_filter_duration_seconds_sum`
- 4.3 SLO（服务级别目标）
- - 4.3.1 `apiserver_request_slo_duration_seconds_count`
  - 4.3.2 `apiserver_request_slo_duration_seconds_bucket`
  - 4.3.3 `apiserver_request_slo_duration_seconds_sum`
- 4.4 其他
- - 4.4.1 `apiserver_requested_deprecated_apis`
  - 4.4.2 `apiserver_request_post_timeout_total`
  - 4.4.3 `apiserver_request_terminations_total`
  - 4.4.4 `apiserver_request_total`
- 4.5 response
- - 4.5.2 `apiserver_response_sizes_count`
  - 4.5.2 `apiserver_response_sizes_bucket`
  - 4.5.3 `apiserver_response_sizes_sum`
5. storage
- 5.1 data key（秘钥）
- - 5.1.1 `apiserver_storage_data_key_generation_duration_seconds_count`
  - 5.1.2 `apiserver_storage_data_key_generation_duration_seconds_bucket`
  - 5.1.3 `apiserver_storage_data_key_generation_duration_seconds_sum`
  - 5.1.4 `apiserver_storage_data_key_generation_failures_total`
- 5.2 list
- - 5.2.1 `apiserver_storage_list_evaluated_objects_total`
  - 5.2.2 `apiserver_storage_list_fetched_objects_total`
  - 5.2.3 `apiserver_storage_list_returned_objects_total`
  - 5.2.4 `apiserver_storage_list_total`
- 5.3 watch
- - 5.3.1 `apiserver_watch_events_sizes_count`
  - 5.3.2 `apiserver_watch_events_sizes_bucket`
  - 5.3.3 `apiserver_watch_events_sizes_sum`
  - 5.3.4 `apiserver_watch_events_total`
  - 5.4.2 `apiserver_storage_envelope_transformation_cache_misses_total`
6. 其他
- 6.1 cache list
- 6.1.1 `apiserver_cache_list_total`
- - 6.1.2 `apiserver_cache_list_fetched_objects_total`
  - 6.1.3 `apiserver_cache_list_returned_objects_total`
- 6.2 client
- - 6.2.1 `apiserver_client_certificate_expiration_seconds_count`
  - 6.2.2 `apiserver_client_certificate_expiration_seconds_bucket`
  - 27. `apiserver_client_certificate_expiration_seconds_sum`
- 6.3 x509 SAN字段
- 6.3.1 `apiserver_webhooks_x509_missing_san_total`
- - 6.3.2 `apiserver_kube_aggregator_x509_missing_san_total`
- 6.4 其他
- - 6.4.1 `apiserver_envelope_encryption_dek_cache_fill_percent`
  - 6.4.2 `apiserver_init_events_total`
  - 6.4.3 `apiserver_longrunning_gauge`
  - 6.4.4 `apiserver_longrunning_requests`
- 6.4.5 `apiserver_registered_watchers`
- - 6.4.6 `apiserver_selfrequest_total`
  - 6.4.7 `apiserver_tls_handshake_errors_total`

1. apiserver admission（准入）和audit（审计）

1.1 准入控制器调用次数

1.1.1 `apiserver_admission_controller_admission_duration_seconds_count`

作用：统计触发指定准入控制器的操作的次数。

直译为：“统计apiserver准入控制器的准入操作的时间” 这个行为的计数。

示例：
- 查询条件：
我们查询10.10.239.101:6443这个接口，Priority这个准入控制器的调用次数。
```
apiserver_admission_controller_admission_duration_seconds_count{instance="10.10.239.101:6443",name="Priority"}
```
- 查询结果
我们可以看到，关于权限的Priority准入控制器有两种子类型，admit（进行资源调整）和validate（验证资源是否符合策略）。每个子类型有三种操作分别是：CREATE、DELETE、UPDATE

apiserver_admission_controller_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="CREATE",rejected="false",type="admit"}	1323439
apiserver_admission_controller_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="CREATE",rejected="false",type="validate"}	1323429
apiserver_admission_controller_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="DELETE",rejected="false",type="admit"}	5507
apiserver_admission_controller_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="DELETE",rejected="false",type="validate"}	5507
apiserver_admission_controller_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="UPDATE",rejected="false",type="admit"}	22473897
apiserver_admission_controller_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="UPDATE",rejected="false",type="validate"}	22471410

说明：
- admission_controller：准入控制器，是Kubernetes API 服务器中的一组拦截机制
- name：准入控制器的名称，列举如下：
  - CertificateApproval：与证书相关的审批操作。
  - NamespaceLifecycle：与命名空间生命周期相关的操作。
  - LimitRanger：与资源限制（如 CPU 和内存限制）相关的操作。
  - ResourceQuota：与资源配额相关的操作。
  - DefaultStorageClass：与默认存储类相关的操作。
  - ValidatingAdmissionWebhook：与自定义验证 Webhook 相关的操作。
  - MutatingAdmissionWebhook：与自定义变更 Webhook 相关的操作。
  - Priority：与优先级相关的操作。
  - PodSecurity：与Pod安全相关的操作
- type：表示准入控制器执行的子类型，其值如下：
  - admit：进行资源调整。
  - validate：验证资源是否符合策略。

1.1.2 `apiserver_admission_controller_admission_duration_seconds_bucket`

作用：统计触发指定准入控制器的操作的次数，并放入桶中（桶按时间分布）
示例：
- 查询语句
为了便于对比，我们将刚才count的查询结果的第一条作为bucket的查询条件：
```
apiserver_admission_controller_admission_duration_seconds_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="CREATE",rejected="false",type="admit"}
```
- 查询结果
我们可以看见，bucket本质是将count的结果放到了不同的“桶”中

apiserver_admission_controller_admission_duration_seconds_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="+Inf",name="Priority",operation="CREATE",rejected="false",type="admit"}	1323585
apiserver_admission_controller_admission_duration_seconds_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.005",name="Priority",operation="CREATE",rejected="false",type="admit"}	1323585
apiserver_admission_controller_admission_duration_seconds_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.025",name="Priority",operation="CREATE",rejected="false",type="admit"}	1323585
apiserver_admission_controller_admission_duration_seconds_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.1",name="Priority",operation="CREATE",rejected="false",type="admit"}	1323585
apiserver_admission_controller_admission_duration_seconds_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.5",name="Priority",operation="CREATE",rejected="false",type="admit"}	1323585
apiserver_admission_controller_admission_duration_seconds_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1",name="Priority",operation="CREATE",rejected="false",type="admit"}	1323585
apiserver_admission_controller_admission_duration_seconds_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="2.5",name="Priority",operation="CREATE",rejected="false",type="admit"}	1323585

说明：
- le：桶的标签（或维度），这里的维度是时间（分别为无限大、0.005秒、0.025秒、0.1秒、0.5秒、1秒、2.5秒）
  - 同时，我们可以分析出，所有调用在0.005秒内全部完成了。

1.1.3 `apiserver_admission_controller_admission_duration_seconds_sum`

作用：统计触发指定准入控制器的操作所花费的总时间，单位为秒。
示例：
- 查询语句
```
apiserver_admission_controller_admission_duration_seconds_sum{instance="10.10.239.101:6443",name="Priority"}
```
- 查询结果
我们可以看见，bucket本质是将count的结果放到了不同的“桶”中

apiserver_admission_controller_admission_duration_seconds_sum{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="CREATE",rejected="false",type="admit"}	0.36450270500003595
apiserver_admission_controller_admission_duration_seconds_sum{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="CREATE",rejected="false",type="validate"}	0.19169566899938528
apiserver_admission_controller_admission_duration_seconds_sum{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="DELETE",rejected="false",type="admit"}	0.001428140999999997
apiserver_admission_controller_admission_duration_seconds_sum{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="DELETE",rejected="false",type="validate"}	0.0007803470000000063
apiserver_admission_controller_admission_duration_seconds_sum{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="UPDATE",rejected="false",type="admit"}	6.796822979032037
apiserver_admission_controller_admission_duration_seconds_sum{instance="10.10.239.101:6443",job="kubernetes-apiservers",name="Priority",operation="UPDATE",rejected="false",type="validate"}	2.9933640270575186

说明：和前边两个类似，只不过统计的是总时间，没有什么可说的

1.2 step（准入步骤）

1.2.1 `apiserver_admission_step_admission_duration_seconds_count`

作用：统计了某个准入步骤被执行的次数。

之前apiserver_admission_controller_admission_duration_seconds_count统计的是执行次数，且区分准入器。

示例：

查询语句

apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443"}

查询结果

apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CONNECT",rejected="false",type="admit"}	1117577
apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CONNECT",rejected="false",type="validate"}	1117577
apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CREATE",rejected="false",type="admit"}	1323657
apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CREATE",rejected="false",type="validate"}	1323325
apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CREATE",rejected="true",type="admit"}	34
apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CREATE",rejected="true",type="validate"}	322
apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="DELETE",rejected="false",type="admit"}	5507
apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="DELETE",rejected="false",type="validate"}	5507
apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="UPDATE",rejected="false",type="admit"}	22477345
apiserver_admission_step_admission_duration_seconds_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="UPDATE",rejected="false",type="validate"}	22474858

1.2.2 `apiserver_admission_step_admission_duration_seconds_bucket`

作用：统计了某个准入步骤被执行的次数，按条件放入bucket中。
说明：参加上边count

1.2.3 `apiserver_admission_step_admission_duration_seconds_sum`

作用：统计了某个准入步骤被执行的总时间，单位秒
说明：参加上边count

1.3 summary（摘要）

1.3.1 `apiserver_admission_step_admission_duration_seconds_summary`

作用：统计了某个准入步骤被执行的总时间，包括分位数，单位秒。

示例：

查询语句

apiserver_admission_step_admission_duration_seconds_summary{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CREATE",rejected="false",type="admit"}

查询结果

apiserver_admission_step_admission_duration_seconds_summary{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CREATE",quantile="0.5",rejected="false",type="admit"}	0.000047539
apiserver_admission_step_admission_duration_seconds_summary{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CREATE",quantile="0.9",rejected="false",type="admit"}	0.000073149
apiserver_admission_step_admission_duration_seconds_summary{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CREATE",quantile="0.99",rejected="false",type="admit"}	0.000098979

说明：
- Summary（摘要）是prometheus的四种数据类型之一，提供分位数（quantile）
- quantile="0.5"：50%的请求延时小于0.000047539秒

分位数计算：
假设收集了一组时间数据：[0.00002, 0.00004, 0.00005, 0.00007, 0.0001]。
按升序排列这些数据。
分位数对应的是某个百分比位置的值：
0.5 分位数：第 50% 的值（中间位置） → 0.00005。
0.9 分位数：第 90% 的值 → 0.00007。
0.99 分位数：第 99% 的值 → 0.0001。

1.3.2 `apiserver_admission_step_admission_duration_seconds_summary_count`

作用：统计了某个准入步骤被执行的关于Summary（摘要）的统计。

摘要会计算分位数，它是prometheus四种数据类型之一

示例：

apiserver_admission_step_admission_duration_seconds_summary_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",operation="CREATE",rejected="false",type="admit"}	1324084

说明：
- 我们可以看见summary只有count和sum两种

1.3.3 `apiserver_admission_step_admission_duration_seconds_summary_sum`

作用：统计了某个准入步骤被执行的关于Summary（摘要）的执行总时间，单位是秒
说明：参加上边count

1.4 webhook

1.4.1 `apiserver_admission_webhook_admission_duration_seconds_count`

作用：该指标统计了 Kubernetes API 服务器中处理的准入 Webhook 请求的次数

1.4.2 `apiserver_admission_webhook_admission_duration_seconds_bucket`

作用：该指标统计了 Kubernetes API 服务器中处理的准入 Webhook 请求的次数，按条件放入bucket

1.4.5 `apiserver_admission_webhook_admission_duration_seconds_sum`

作用：该指标统计了 Kubernetes API 服务器中处理的准入 Webhook 请求的总时间，单位秒。

1.4.6 `apiserver_admission_webhook_rejection_count`

作用：该指标统计了 Kubernetes API 服务器中被拒绝的准入 Webhook 请求的次数。

1.4.7 `apiserver_admission_webhook_request_total`

作用：该指标衡量了 Kubernetes API 服务器中所有准入 Webhook 请求的总次数。

1.5 审计

1.5.1 `apiserver_audit_event_total`

作用：该指标跟踪了 Kubernetes API 服务器中记录的审计事件总数。

1.5.2 `apiserver_audit_requests_rejected_total`

该指标统计了 Kubernetes API 服务器中被拒绝的审计请求总数。

2. apiserver delegated（委托）

delegated（委托）是指将某些功能或操作交由外部系统或服务来处理，而不是由 Kubernetes API Server 自己直接完成。这种设计通过分离职责来提升灵活性和可扩展性

2.1 delegated authn（委托认证）

是指 Kubernetes API Server 将认证（Authentication）的过程交由外部系统或服务处理，而不是由 API Server 自身直接完成身份验证

2.1.1 `apiserver_delegated_authn_request_duration_seconds_count`

作用：该指标统计了 Kubernetes API 服务器中处理的委托认证请求的次数。

示例：

查询语句

apiserver_delegated_authn_request_duration_seconds_count

查询结果

apiserver_delegated_authn_request_duration_seconds_count{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux"}	110167
apiserver_delegated_authn_request_duration_seconds_count{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m02",kubernetes_io_os="linux"}	110049
apiserver_delegated_authn_request_duration_seconds_count{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m03",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m03",kubernetes_io_os="linux"}	110112
apiserver_delegated_authn_request_duration_seconds_count{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-n01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n01",kubernetes_io_os="linux"}	43596
apiserver_delegated_authn_request_duration_seconds_count{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-n02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n02",kubernetes_io_os="linux"}	43524

2.1.2 `apiserver_delegated_authn_request_duration_seconds_bucket`

作用：该指标统计了 Kubernetes API 服务器中处理的委托认证请求的次数，放入bucket
示例：
- 查询语句
```
apiserver_delegated_authn_request_duration_seconds_bucket{instance="k8s-dev-m01"}
```
- 查询结果

apiserver_delegated_authn_request_duration_seconds_bucket{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux",le="+Inf"}	110168
apiserver_delegated_authn_request_duration_seconds_bucket{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux",le="0.25"}	110168
apiserver_delegated_authn_request_duration_seconds_bucket{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux",le="0.5"}	110168
apiserver_delegated_authn_request_duration_seconds_bucket{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux",le="0.7"}	110168
apiserver_delegated_authn_request_duration_seconds_bucket{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux",le="1"}	110168
apiserver_delegated_authn_request_duration_seconds_bucket{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux",le="1.5"}	110168
apiserver_delegated_authn_request_duration_seconds_bucket{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux",le="10"}	110168
apiserver_delegated_authn_request_duration_seconds_bucket{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux",le="3"}	110168
apiserver_delegated_authn_request_duration_seconds_bucket{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux",le="5"}	110168

2.1.3 `apiserver_delegated_authn_request_duration_seconds_sum`

作用：该指标统计了 Kubernetes API 服务器中处理的委托认证请求的总持续时间，单位为秒。

2.1.4 `apiserver_delegated_authn_request_total`

作用：该指标统计了 Kubernetes API 服务器中处理的委托认证请求的总次数。
示例：

apiserver_delegated_authn_request_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux"}	110170

和count的值此处是相同的

apiserver_delegated_authn_request_duration_seconds_count{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",code="201",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux"}	110170

2.2 delegated authz（委托授权）

委托授权：是指将请求的授权决策交由外部系统或服务来处理，而不是由 API Server 自己完成

2.2.1 `apiserver_delegated_authz_request_duration_seconds_count`

作用：服务器中处理的委托授权请求的次数。

2.2.2 `apiserver_delegated_authz_request_duration_seconds_bucket`

作用：服务器中处理的委托授权请求的次数，按条件放入bucket中。

2.2.3 `apiserver_delegated_authz_request_duration_seconds_sum`

作用：服务器中处理的委托授权请求的总持续时间，单位为秒。

2.2.5 `apiserver_delegated_authz_request_total`

作用：服务器中处理的委托授权请求的总次数

3. apiserver flowcontrol

3.1 请求和速率（待整理）

3.1.1 `apiserver_flowcontrol_current_executing_requests`

作用：该指标记录了 Kubernetes API 服务器流量控制中当前正在执行的请求数量。

示例：

查询语句

apiserver_flowcontrol_current_executing_requests{instance="10.10.239.101:6443"}

查询结果

apiserver_flowcontrol_current_executing_requests{flow_schema="catch-all",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="catch-all"}	0
apiserver_flowcontrol_current_executing_requests{flow_schema="global-default",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="global-default"}	0
apiserver_flowcontrol_current_executing_requests{flow_schema="kube-controller-manager",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	0
apiserver_flowcontrol_current_executing_requests{flow_schema="kube-scheduler",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	0
apiserver_flowcontrol_current_executing_requests{flow_schema="kube-system-service-accounts",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	0
apiserver_flowcontrol_current_executing_requests{flow_schema="service-accounts",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-low"}	1
apiserver_flowcontrol_current_executing_requests{flow_schema="system-leader-election",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="leader-election"}	0
apiserver_flowcontrol_current_executing_requests{flow_schema="system-node-high",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="node-high"}	0
apiserver_flowcontrol_current_executing_requests{flow_schema="system-nodes",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="system"}	0
apiserver_flowcontrol_current_executing_requests{flow_schema="workload-leader-election",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="leader-election"}	0

说明：

priority_level	描述	优先级等级	Flow Schema
`system`	系统级请求，主要用于节点心跳和关键系统组件的通信。	最高	`system-nodes`
`node-high`	高优先级的节点请求，如 kubelet 与 apiserver 的通信。	很高	`system-node-high`
`leader-election`	用于系统和工作负载的选举相关请求，保证服务的主从切换。	高	`system-leader-election`、`workload-leader-election`
`workload-high`	高优先级的工作负载请求，例如重要服务的 Pod 管理或资源操作。	中高	`kube-controller-manager`、`kube-scheduler`、`kube-system-service-accounts`
`global-default`	默认优先级的请求，未显式配置流控规则的请求都归入该类。	中	`global-default`
`workload-low`	较低优先级的工作负载请求，例如普通服务账户的请求。	较低	`service-accounts`
`catch-all`	最低优先级，所有未匹配到其他规则的请求。	最低	`catch-all`

3.1.2 `apiserver_flowcontrol_current_inqueue_requests`

作用：该指标衡量了 Kubernetes API 服务器流量控制中当前排队的请求数量。
说明：和4.1.1类似

3.1.3 `apiserver_flowcontrol_current_r`（未验证）

作用：该指标表示 Kubernetes API 服务器当前请求调度速率

3.1.4 `apiserver_flowcontrol_dispatched_requests_total`

作用：该指标统计了 Kubernetes API 服务器已经调度的请求总数。

3.1.5 `apiserver_flowcontrol_dispatch_r`（未验证）

作用：该指标表示 Kubernetes API 服务器请求调度速率。

3.1.6 `apiserver_flowcontrol_latest_s`

作用：该指标显示了 Kubernetes API 服务器流量控制的最新速率。

3.1.7 `apiserver_flowcontrol_next_discounted_s_bounds`

作用：给出的结果表示流量控制（Flow Control）中某个请求在下一次调整（折扣）的时间边界，单位是秒（seconds）。
说明：
- 该指标展示了一个最大等待时间，超过这个时间后，系统可能会调整该请求的优先级或采取其他流量控制措施

3.1.8 `apiserver_flowcontrol_next_s_bounds`

作用：下一个请求的时间边界，即请求的等待时间或处理延迟的界限。它通常用于显示请求在流量控制队列中能等待多久。
说明：next_discounted_s_bounds 是基于 next_s_bounds 的进一步调整，代表了流量控制系统经过优先级、负载等因素调整后的实际最大等待时间。

3.2 priority level（优先级别）

3.2.2 `apiserver_flowcontrol_priority_level_request_count_samples_count`

作用：该指标统计了不同优先级请求的样本数。

示例：

查询语句

apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443"}

查询结果

apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="catch-all"}	14702351176
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="global-default"}	14702349430
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="leader-election"}	14702353408
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="node-high"}	14702354335
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="system"}	14702351745
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="workload-high"}	14702351411
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="workload-low"}	14702354411
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="waiting",priority_level="catch-all"}	14702351171
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="waiting",priority_level="global-default"}	14702349425
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="waiting",priority_level="leader-election"}	14702353404
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="waiting",priority_level="node-high"}	14702354331
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="waiting",priority_level="system"}	14702351744
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="waiting",priority_level="workload-high"}	14702351409
apiserver_flowcontrol_priority_level_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="waiting",priority_level="workload-low"}	14702354411

说明：优先级说明如下：

priority_level	描述	优先级等级
`system`	系统级请求，主要用于节点心跳和关键系统组件的通信。	最高
`node-high`	高优先级的节点请求，如 kubelet 与 apiserver 的通信。	很高
`leader-election`	用于系统和工作负载的选举相关请求，保证服务的主从切换。	高
`workload-high`	高优先级的工作负载请求，例如重要服务的 Pod 管理或资源操作。	中高
`global-default`	默认优先级的请求，未显式配置流控规则的请求都归入该类。	中
`workload-low`	较低优先级的工作负载请求，例如普通服务账户的请求。	较低
`catch-all`	最低优先级，所有未匹配到其他规则的请求。	最低

大量的“waiting”并不是说现在它们处于waiting状态，而是曾经处于waiting状态

3.2.1 `apiserver_flowcontrol_priority_level_request_count_samples_bucket`

作用：该指标统计了不同优先级请求的样本数，并按要求放入桶中

3.2.3 `apiserver_flowcontrol_priority_level_request_count_samples_sum`

作用：该指标统计了不同优先级请求的总时间

3.2.4 `apiserver_flowcontrol_priority_level_request_count_watermarks_count`

作用：该指标统计了不同优先级请求的水位标记的样本数。

示例：

查询语句

apiserver_flowcontrol_priority_level_request_count_watermarks_count{instance="10.10.239.101:6443",priority_level="catch-all"}

查询结果

apiserver_flowcontrol_priority_level_request_count_watermarks_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",mark="high",phase="executing",priority_level="catch-all"}	14705233235
apiserver_flowcontrol_priority_level_request_count_watermarks_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",mark="high",phase="waiting",priority_level="catch-all"}	14705233230
apiserver_flowcontrol_priority_level_request_count_watermarks_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",mark="low",phase="executing",priority_level="catch-all"}	14705233235
apiserver_flowcontrol_priority_level_request_count_watermarks_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",mark="low",phase="waiting",priority_level="catch-all"}	14705233230

说明：
- mark：表示水位，有“high”和“low”

3.2.5 `apiserver_flowcontrol_priority_level_request_count_watermarks_bucket`

作用：该指标统计了不同优先级请求的水位标记的样本数，并放入桶中。

3.2.6 `apiserver_flowcontrol_priority_level_request_count_watermarks_sum`

作用：该指标统计了不同优先级请求的水位标记的总时间，单位秒

3.2.7 `apiserver_flowcontrol_priority_level_seat_count_samples_count`

作用：该指标统计了每个优先级的席位数量样本数。

示例：

查询语句

apiserver_flowcontrol_priority_level_seat_count_samples_count{instance="10.10.239.101:6443"}

查询结果

apiserver_flowcontrol_priority_level_seat_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="catch-all"}	14705604703
apiserver_flowcontrol_priority_level_seat_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="global-default"}	14705602958
apiserver_flowcontrol_priority_level_seat_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="leader-election"}	14705608846
apiserver_flowcontrol_priority_level_seat_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="node-high"}	14705607927
apiserver_flowcontrol_priority_level_seat_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="system"}	14705605687
apiserver_flowcontrol_priority_level_seat_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="workload-high"}	14705605278
apiserver_flowcontrol_priority_level_seat_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="workload-low"}	14705609422

说明：
- seat：表示席位，是用于表示每个请求消耗的资源量的单位
  - 简单请求：如读取一个小的 ConfigMap 可能只需要 1 个席位。
  - 复杂请求：如查询大量 Pod 的信息或大规模更新操作，可能需要多个席位。

3.2.8 `apiserver_flowcontrol_priority_level_seat_count_samples_bucket`

作用：该指标统计了每个优先级的席位数量样本数，并放入桶中

示例：

查询语句

apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",priority_level="catch-all"}

如下一个完整的bucket

apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="+Inf",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0",phase="executing",priority_level="catch-all"}	0
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.1",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.2",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.3",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.4",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.5",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.6",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.7",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.8",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.9",phase="executing",priority_level="catch-all"}	14705725191
apiserver_flowcontrol_priority_level_seat_count_samples_bucket{instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1",phase="executing",priority_level="catch-all"}	14705725191

3.2.9 `apiserver_flowcontrol_priority_level_seat_count_samples_sum`

作用：该指标统计了每个优先级的席位执行的总时间。
示例：

apiserver_flowcontrol_priority_level_seat_count_samples_sum{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",priority_level="catch-all"}	1131218392.179275

3.2.12 `apiserver_flowcontrol_priority_level_seat_count_watermarks_count`

作用：该指标统计了每个优先级席位数量水位标记的样本数。

3.2.11 `apiserver_flowcontrol_priority_level_seat_count_watermarks_bucket`

作用：该指标统计了每个优先级席位数量水位标记的样本数，放入桶中

3.2.13 `apiserver_flowcontrol_priority_level_seat_count_watermarks_sum`

作用：该指标统计了每个优先级席位数量各水位标记的总时间

3.3 读写请求

3.3.1 `apiserver_flowcontrol_read_vs_write_request_count_samples_count`

作用：读取和写入请求的样本数。

示例：

查询语句

apiserver_flowcontrol_read_vs_write_request_count_samples_count{instance="10.10.239.101:6443"}

如下一个完整的bucket

apiserver_flowcontrol_read_vs_write_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",request_kind="mutating"}	14706135722
apiserver_flowcontrol_read_vs_write_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="executing",request_kind="readOnly"}	14706136608
apiserver_flowcontrol_read_vs_write_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="waiting",request_kind="mutating"}	14706136222
apiserver_flowcontrol_read_vs_write_request_count_samples_count{instance="10.10.239.101:6443",job="kubernetes-apiservers",phase="waiting",request_kind="readOnly"}	14706136608

说明：
- 状态有"executing"和"waiting"两种
- 每种状态有读写和只读两种情况

3.3.2 `apiserver_flowcontrol_read_vs_write_request_count_samples_bucket`

作用：该指标统计了读取和写入请求的样本数，放入桶中。

3.3.3 `apiserver_flowcontrol_read_vs_write_request_count_samples_sum`

作用：该指标统计了读取和写入请求的样本数的总时间

3.3.4 `apiserver_flowcontrol_read_vs_write_request_count_watermarks_count`

作用：读取和写入请求的样本数，和之前的情况一样，加入了水位标记

3.3.5 `apiserver_flowcontrol_read_vs_write_request_count_watermarks_bucket`

作用：读取和写入请求的样本数，加入和水位标签，放入桶中

3.3.6 `apiserver_flowcontrol_read_vs_write_request_count_watermarks_sum`

作用：读取和写入请求样本执行的总时间（按水位标签），单位为妙

3.4 request（请求）

3.4.1 `apiserver_flowcontrol_request_concurrency_in_use`

作用：该指标衡量了当前在 Kubernetes API 服务器中使用的请求并发数。

示例：

查询条件：

apiserver_flowcontrol_request_concurrency_in_use{instance="10.10.239.101:6443"}

查询结果

apiserver_flowcontrol_request_concurrency_in_use{flow_schema="catch-all",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="catch-all"}	0
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="global-default",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="global-default"}	0
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="kube-controller-manager",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	0
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="kube-scheduler",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	0
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="kube-system-service-accounts",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	0
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="service-accounts",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-low"}	1
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="system-leader-election",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="leader-election"}	0
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="system-node-high",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="node-high"}	0
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="system-nodes",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="system"}	0
apiserver_flowcontrol_request_concurrency_in_use{flow_schema="workload-leader-election",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="leader-election"}	0

说明：
- flow_schema 和 priority_level 说明见apiserver_flowcontrol_current_executing_requests的说明

3.4.2 `apiserver_flowcontrol_request_concurrency_limit`

作用：针对不同优先级级别（Priority Level）设置的并发请求限制

示例：

查询条件

apiserver_flowcontrol_request_concurrency_limit{instance="10.10.239.101:6443"}

结果

apiserver_flowcontrol_request_concurrency_limit{instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="catch-all"}	13
apiserver_flowcontrol_request_concurrency_limit{instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="global-default"}	49
apiserver_flowcontrol_request_concurrency_limit{instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="leader-election"}	25
apiserver_flowcontrol_request_concurrency_limit{instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="node-high"}	98
apiserver_flowcontrol_request_concurrency_limit{instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="system"}	74
apiserver_flowcontrol_request_concurrency_limit{instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	98
apiserver_flowcontrol_request_concurrency_limit{instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-low"}	245

3.4.3 `apiserver_flowcontrol_request_execution_seconds_count`

作用：某种类型的请求在 Kubernetes API 服务器中的执行次数

示例：

查询条件

apiserver_flowcontrol_request_execution_seconds_count{instance="10.10.239.101:6443"}

结果

apiserver_flowcontrol_request_execution_seconds_count{flow_schema="catch-all",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="catch-all"}	3
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="exempt",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="exempt"}	13061945
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="global-default",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="global-default"}	10055
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="kube-controller-manager",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	5675693
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="kube-scheduler",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	490725
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="kube-system-service-accounts",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	34527549
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="probes",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="exempt"}	16705489
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="service-accounts",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-low"}	25304940
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="system-leader-election",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="leader-election"}	19213809
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="system-node-high",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="node-high"}	14810942
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="system-nodes",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="system"}	13948763
apiserver_flowcontrol_request_execution_seconds_count{flow_schema="workload-leader-election",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="leader-election"}	2191

3.4.4 `apiserver_flowcontrol_request_execution_seconds_bucket`

作用：某种类型的请求在 Kubernetes API 服务器中的执行次数，并放入bucket

3.4.5 `apiserver_flowcontrol_request_execution_seconds_sum`

作用：某种类型的请求在 Kubernetes API 服务器中的执行的总时间，单位为秒。

3.4.6 `apiserver_flowcontrol_request_queue_length_after_enqueue_count`

作用：请求被加入队列后的累计计数（也就是说，并不是当前的队列长度）

示例：

查询语句：

apiserver_flowcontrol_request_queue_length_after_enqueue_count{instance="10.10.239.101:6443"}

查询结果

apiserver_flowcontrol_request_queue_length_after_enqueue_count{flow_schema="global-default",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="global-default"}	10055
apiserver_flowcontrol_request_queue_length_after_enqueue_count{flow_schema="kube-controller-manager",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	5676221
apiserver_flowcontrol_request_queue_length_after_enqueue_count{flow_schema="kube-scheduler",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	490736
apiserver_flowcontrol_request_queue_length_after_enqueue_count{flow_schema="kube-system-service-accounts",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-high"}	34528529
apiserver_flowcontrol_request_queue_length_after_enqueue_count{flow_schema="service-accounts",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="workload-low"}	25306133
apiserver_flowcontrol_request_queue_length_after_enqueue_count{flow_schema="system-leader-election",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="leader-election"}	19214320
apiserver_flowcontrol_request_queue_length_after_enqueue_count{flow_schema="system-node-high",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="node-high"}	14811335
apiserver_flowcontrol_request_queue_length_after_enqueue_count{flow_schema="system-nodes",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="system"}	13949218
apiserver_flowcontrol_request_queue_length_after_enqueue_count{flow_schema="workload-leader-election",instance="10.10.239.101:6443",job="kubernetes-apiservers",priority_level="leader-election"}	2191

3.4.7 `apiserver_flowcontrol_request_queue_length_after_enqueue_bucket`

作用：请求被加入队列后的累计计数，放入不同的桶中

3.4.8 `apiserver_flowcontrol_request_queue_length_after_enqueue_sum`

作用：请求被加入队列后的累计的时间总和，单位是秒

3.4.10 `apiserver_flowcontrol_request_wait_duration_seconds_count`

作用：该指标统计了请求等待的请求数

3.4.9 `apiserver_flowcontrol_request_wait_duration_seconds_bucket`

作用：该指标统计了请求等待的请求数，放入不同的桶中。

3.4.12 `apiserver_flowcontrol_request_wait_duration_seconds_sum`

作用：该指标统计了请求等待的请求时间的总和，单位为秒。

4. request 和 response

4.1 请求次数

4.1.1 `apiserver_request_duration_seconds_count`

说明：用于记录 API Server 处理特定请求的次数。

示例：

查询语句：

apiserver_request_duration_seconds_count{instance="10.10.239.101:6443",scope="namespace",resource="deployments"}

查询结果

apiserver_request_duration_seconds_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="LIST",version="v1"}	2089
apiserver_request_duration_seconds_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="WATCH",version="v1"}	148

说明：
- cluster：集群的标识符（如 k8s-dev）。
- component：表示组件来源，通常是 apiserver。
- group：请求目标资源所属的 API 组（如 apps）。
- resource：表示操作的 Kubernetes 资源类型（如 daemonsets 或 deployments）。
- scope：表示资源作用域
  - cluster 表示集群级别资源。
  - namespace 表示命名空间级别资源。
  - resource 表示具体的资源对象
- verb：表示 HTTP 操作类型（如 LIST、DELETE、WATCH 等）。
- version：表示资源 API 的版本（如 v1）。

4.1.2 `apiserver_request_duration_seconds_bucket`

作用：用于记录 API Server 处理特定请求的次数，按bucket分类。
说明：
- 查询语句：
将上边count的第一条的条件拿出，metric换成bucket的，我们来看一下，拼接完如下：
```
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="DELETE",version="v1"}
```
- 查询结果
如下可见，bucket将其count的结果分在了不同的桶中

apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="+Inf",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.05",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.1",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.2",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.4",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.6",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="0.8",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1.25",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1.5",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="10",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="15",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="2",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="20",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="3",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="30",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="4",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="45",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="5",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="6",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="60",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_duration_seconds_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="8",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974

4.1.3 `apiserver_request_duration_seconds_sum`

作用：该指标记录了所有处理请求的总时长，单位为秒。

4.1.4 `apiserver_current_inflight_requests`

作用：统计当前正在处理的请求数（inflight飞行中的，即正在处理的）
示例：

apiserver_current_inflight_requests{instance="10.10.239.101:6443",job="kubernetes-apiservers",request_kind="mutating"}	1
apiserver_current_inflight_requests{instance="10.10.239.101:6443",job="kubernetes-apiservers",request_kind="readOnly"}	1
apiserver_current_inflight_requests{instance="10.10.239.102:6443",job="kubernetes-apiservers",request_kind="mutating"}	0
apiserver_current_inflight_requests{instance="10.10.239.102:6443",job="kubernetes-apiservers",request_kind="readOnly"}	1
apiserver_current_inflight_requests{instance="10.10.239.103:6443",job="kubernetes-apiservers",request_kind="mutating"}	1
apiserver_current_inflight_requests{instance="10.10.239.103:6443",job="kubernetes-apiservers",request_kind="readOnly"}	1

4.1.5 `apiserver_current_inqueue_requests`

作用：当前队列中的请求数
示例：

apiserver_current_inqueue_requests{instance="10.10.239.101:6443",job="kubernetes-apiservers",request_kind="mutating"}	0
apiserver_current_inqueue_requests{instance="10.10.239.101:6443",job="kubernetes-apiservers",request_kind="readOnly"}	1
apiserver_current_inqueue_requests{instance="10.10.239.102:6443",job="kubernetes-apiservers",request_kind="mutating"}	0
apiserver_current_inqueue_requests{instance="10.10.239.102:6443",job="kubernetes-apiservers",request_kind="readOnly"}	0
apiserver_current_inqueue_requests{instance="10.10.239.103:6443",job="kubernetes-apiservers",request_kind="mutating"}	0
apiserver_current_inqueue_requests{instance="10.10.239.103:6443",job="kubernetes-apiservers",request_kind="readOnly"}	1

4.2 过滤器

4.2.1 `apiserver_request_filter_duration_seconds_count`

作用：统计处理指定过滤器的请求的次数

说明：

查询语句：

apiserver_request_filter_duration_seconds_count{instance="10.10.239.101:6443"}

查询结果

apiserver_request_filter_duration_seconds_count{filter="audit",instance="10.10.239.101:6443",job="kubernetes-apiservers"}	145457038
apiserver_request_filter_duration_seconds_count{filter="authentication",instance="10.10.239.101:6443",job="kubernetes-apiservers"}	145457282
apiserver_request_filter_duration_seconds_count{filter="authorization",instance="10.10.239.101:6443",job="kubernetes-apiservers"}	145450750
apiserver_request_filter_duration_seconds_count{filter="impersonation",instance="10.10.239.101:6443",job="kubernetes-apiservers"}	145457038
apiserver_request_filter_duration_seconds_count{filter="priorityandfairness",instance="10.10.239.101:6443",job="kubernetes-apiservers"}	145457038

说明：
- filter：指定过滤器
  - audit：审计过滤器，上文表示审计过滤器处理了145448729次请求。
  - authentication：身份验证过滤器
  - authorization：授权过滤器。
  - impersonation：模拟过滤器。用于模拟其他用户身份处理请求，通常在一些特殊权限请求时使用。
  - priorityandfairness：实现流量的优先级和公平性，确保请求按照定义的优先级策略进行处理。

4.2.2 `apiserver_request_filter_duration_seconds_bucket`

作用：统计处理指定过滤器的请求的次数，放入桶中

4.2.3 `apiserver_request_filter_duration_seconds_sum`

作用：统计处理指定过滤器的请求的总时间

4.3 SLO（服务级别目标）

4.3.1 `apiserver_request_slo_duration_seconds_count`

作用：该指标统计了满足服务级别目标（SLO）条件的请求数量。

在apiserver_request_duration_seconds_count的基础上添加了resource，scope，verb，version等条件

说明：

查询语句：

apiserver_request_slo_duration_seconds_count{instance="10.10.239.101:6443",scope="namespace",resource="deployments"}

查询结果

apiserver_request_slo_duration_seconds_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="DELETE",version="v1"}	974
apiserver_request_slo_duration_seconds_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="LIST",version="v1"}	2101
apiserver_request_slo_duration_seconds_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="WATCH",version="v1"}	148

说明：参加apiserver_request_duration_seconds_count

4.3.2 `apiserver_request_slo_duration_seconds_bucket`

作用：该指标统计了满足服务级别目标（SLO）条件的请求数量，放入桶中

4.3.3 `apiserver_request_slo_duration_seconds_sum`

作用：该指标统计了满足服务级别目标（SLO）条件的请求总时长，单位为秒。

4.4 其他

4.4.1 `apiserver_requested_deprecated_apis`

作用：统计请求调用了某个已弃用的api的请求数量
说明：
- 查询语句：
将上边count的第一条的条件拿出，metric换成bucket的，我们来看一下，拼接完如下：
```
apiserver_requested_deprecated_apis{instance="10.10.239.101:6443"}
```
- 查询结果

apiserver_requested_deprecated_apis{group="autoscaling",instance="10.10.239.101:6443",job="kubernetes-apiservers",removed_release="1.25",resource="horizontalpodautoscalers",version="v2beta1"}	1
apiserver_requested_deprecated_apis{group="policy",instance="10.10.239.101:6443",job="kubernetes-apiservers",removed_release="1.25",resource="podsecuritypolicies",version="v1beta1"}	1
apiserver_requested_deprecated_apis{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="componentstatuses",version="v1"}	1

4.4.2 `apiserver_request_post_timeout_total`

作用：该指标记录了由于超时而导致的 Kubernetes API 请求的 POST 请求数量。

说明：

查询语句：

apiserver_request_post_timeout_total{instance="10.10.239.101:6443"}

查询结果

apiserver_request_post_timeout_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",source="timeout-handler",status="ok"}	3

4.4.3 `apiserver_request_terminations_total`

作用：该指标记录了由于超时而导致的 Kubernetes API 请求的 POST 请求数量。

说明：

查询语句：

apiserver_request_terminations_total{instance="10.10.239.101:6443"}

查询结果

apiserver_request_terminations_total{code="503",component="aggregator",instance="10.10.239.101:6443",job="kubernetes-apiservers",subresource="/apis/metrics.k8s.io/v1beta1",verb="GET"}	787
apiserver_request_terminations_total{code="504",component="apiserver",group="coordination.k8s.io",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="leases",scope="resource",verb="GET",version="v1"}	3

说明：
- 如第一条表示：通过聚合器（aggregator），GET 方式访问 instance="10.10.239.101:6443"的接口subresource="/apis/metrics.k8s.io/v1beta1"，返回 503错误。

4.4.4 `apiserver_request_total`

作用：该指标统计了 Kubernetes API 服务器处理的所有请求的总数。

说明：

查询语句：

apiserver_request_total{instance="10.10.239.101:6443",resource="deployments",scope="resource"}

查询结果

apiserver_request_total{code="200",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",subresource="scale",verb="PATCH",version="v1"}	10
apiserver_request_total{code="200",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",subresource="status",verb="PUT",version="v1"}	10687
apiserver_request_total{code="200",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="DELETE",version="v1"}	127
apiserver_request_total{code="200",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="GET",version="v1"}	56473
apiserver_request_total{code="200",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="PATCH",version="v1"}	268
apiserver_request_total{code="200",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="PUT",version="v1"}	141
apiserver_request_total{code="201",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="POST",version="v1"}	166
apiserver_request_total{code="400",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="POST",version="v1"}	6
apiserver_request_total{code="404",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="DELETE",version="v1"}	21
apiserver_request_total{code="404",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="GET",version="v1"}	179
apiserver_request_total{code="404",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="POST",version="v1"}	1
apiserver_request_total{code="409",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",subresource="status",verb="PUT",version="v1"}	892
apiserver_request_total{code="409",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="POST",version="v1"}	4
apiserver_request_total{code="409",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="PUT",version="v1"}	5
apiserver_request_total{code="422",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="PATCH",version="v1"}	7
apiserver_request_total{code="422",component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="POST",version="v1"}	4

4.5 response

4.5.2 `apiserver_response_sizes_count`

作用：响应大小统计次数的统计

说明：

查询语句：

apiserver_response_sizes_count{instance="10.10.239.101:6443",resource="deployments"}

查询结果

apiserver_response_sizes_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="cluster",verb="LIST",version="v1"}	206
apiserver_response_sizes_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="cluster",verb="WATCH",version="v1"}	72767
apiserver_response_sizes_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="LIST",version="v1"}	2106
apiserver_response_sizes_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="WATCH",version="v1"}	148
apiserver_response_sizes_count{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="resource",verb="GET",version="v1"}	56652

4.5.2 `apiserver_response_sizes_bucket`

作用：响应大小统计，放入bucket中

说明：

查询语句：

apiserver_response_sizes_bucket{instance="10.10.239.101:6443",resource="deployments",scope="namespace",verb="LIST"}

查询结果

apiserver_response_sizes_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="+Inf",resource="deployments",scope="namespace",verb="LIST",version="v1"}	2106
apiserver_response_sizes_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1000",resource="deployments",scope="namespace",verb="LIST",version="v1"}	1514
apiserver_response_sizes_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="10000",resource="deployments",scope="namespace",verb="LIST",version="v1"}	1696
apiserver_response_sizes_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="100000",resource="deployments",scope="namespace",verb="LIST",version="v1"}	2102
apiserver_response_sizes_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1e+06",resource="deployments",scope="namespace",verb="LIST",version="v1"}	2106
apiserver_response_sizes_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1e+07",resource="deployments",scope="namespace",verb="LIST",version="v1"}	2106
apiserver_response_sizes_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1e+08",resource="deployments",scope="namespace",verb="LIST",version="v1"}	2106
apiserver_response_sizes_bucket{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",le="1e+09",resource="deployments",scope="namespace",verb="LIST",version="v1"}	2106

4.5.3 `apiserver_response_sizes_sum`

作用：该指标跟踪了 Kubernetes API 服务器响应大小的总和，单位为字节。

5. storage

5.1 data key（秘钥）

5.1.1 `apiserver_storage_data_key_generation_duration_seconds_count`

作用：API Server 为存储生成数据加密密钥的操作次数
示例

apiserver_storage_data_key_generation_duration_seconds_count{instance="10.10.239.102:6443",job="kubernetes-apiservers"}	0

说明：
- 值为0，表示到目前为止，API Server 还没有进行任何密钥生成操作。
- 如果 Kubernetes 配置了静态数据加密（Encryption at Rest），API Server 在写入 etcd 数据时会对数据进行加密。

5.1.2 `apiserver_storage_data_key_generation_duration_seconds_bucket`

作用：API Server 为存储生成数据加密密钥的操作次数，放入bucket

5.1.3 `apiserver_storage_data_key_generation_duration_seconds_sum`

作用：API Server 为存储生成数据加密密钥的操作的总时长，单位为秒。

5.1.4 `apiserver_storage_data_key_generation_failures_total`

作用：该指标记录了存储数据密钥生成失败的总数

5.2 list

5.2.1 `apiserver_storage_list_evaluated_objects_total`

作用：该指标记录了从存储列表中被扫描或评估的对象总数

说明：

查询语句：

apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443"}

查询结果

apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="apiServerIPInfo"}	4428758
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="apiservices.apiregistration.k8s.io"}	481
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="apiservices.management.cattle.io"}	0
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="apps.catalog.cattle.io"}	11
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="authconfigs.management.cattle.io"}	183
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="certificatesigningrequests.certificates.k8s.io"}	0
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="clusterregistrationtokens.management.cattle.io"}	0
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="clusterrepos.catalog.cattle.io"}	6
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="clusterrolebindings.rbac.authorization.k8s.io"}	1665
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="clusterroles.rbac.authorization.k8s.io"}	1074
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="clusters.management.cattle.io"}	160
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="configmaps"}	265
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="controllerrevisions.apps"}	161
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="cronjobs.batch"}	0
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="csidrivers.storage.k8s.io"}	0
apiserver_storage_list_evaluated_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="csinodes.storage.k8s.io"}	165
……

5.2.2 `apiserver_storage_list_fetched_objects_total`

作用：该指标记录了从存储列表中实际提取的准备返回给客户端的对象总数

说明：

查询语句：

apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443"}

查询结果

apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="apiServerIPInfo"}	4429106
apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="apiservices.apiregistration.k8s.io"}	481
apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="apiservices.management.cattle.io"}	0
apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="apps.catalog.cattle.io"}	11
apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="authconfigs.management.cattle.io"}	183
apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="certificatesigningrequests.certificates.k8s.io"}	0
apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="clusterregistrationtokens.management.cattle.io"}	0
apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="clusterrepos.catalog.cattle.io"}	6
apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="clusterrolebindings.rbac.authorization.k8s.io"}	1665
apiserver_storage_list_fetched_objects_total{instance="10.10.239.102:6443",job="kubernetes-apiservers",resource="clusterroles.rbac.authorization.k8s.io"}	1074

5.2.3 `apiserver_storage_list_returned_objects_total`

作用：该指标记录了从存储列表中返回，并且被用户接收的对象总数。
以上三个metric的比较

指标名称	含义	对象数量	场景
`evaluated_objects_total`	存储中被扫描或评估的对象总数	大于或等于提取的数量	包括所有被扫描的对象，即使最终未返回给客户端。
`fetched_objects_total`	实际提取并返回给客户端的对象总数	小于或等于评估的数量	仅包含符合查询条件的对象，实际返回给客户端或请求方。
`returned_objects_total`	客户端最终接收到的对象总数	小于或等于提取的数量	进一步经过服务端过滤后，实际传输到客户端的对象数量。

5.2.4 `apiserver_storage_list_total`

作用：存储列表操作的总数
说明：
- 改metric 执行一次列表操作（比如kubectl get pod -A）结果增加1。
- 前边三个metric执行一次列表操作结果增加列表中成员的数量。

5.3 watch

5.3.1 `apiserver_watch_events_sizes_count`

作用：统计了Watch事件

说明：

查询语句：

apiserver_watch_events_sizes_count{instance="10.10.239.101:6443"}

查询结果

apiserver_watch_events_sizes_count{group="admissionregistration.k8s.io",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="MutatingWebhookConfiguration",version="v1"}	407811
apiserver_watch_events_sizes_count{group="admissionregistration.k8s.io",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="ValidatingWebhookConfiguration",version="v1"}	388693
apiserver_watch_events_sizes_count{group="apiextensions.k8s.io",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="CustomResourceDefinition",version="v1"}	380298
apiserver_watch_events_sizes_count{group="apiregistration.k8s.io",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="APIService",version="v1"}	380505
apiserver_watch_events_sizes_count{group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="ControllerRevision",version="v1"}	176101
apiserver_watch_events_sizes_count{group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="DaemonSet",version="v1"}	282464
apiserver_watch_events_sizes_count{group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="Deployment",version="v1"}	332079
apiserver_watch_events_sizes_count{group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="ReplicaSet",version="v1"}	464521
apiserver_watch_events_sizes_count{group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="StatefulSet",version="v1"}	388390
apiserver_watch_events_sizes_count{group="autoscaling",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="HorizontalPodAutoscaler",version="v2"}	223354

说明：如，执行 kubectl get pods -n crust-clf --watch命令，prometheus查询apiserver_watch_events_sizes_count{instance="10.10.239.101:6443",kind="Pod"}的结果会增加一

5.3.2 `apiserver_watch_events_sizes_bucket`

作用：统计了Watch事件，放入桶中（此处桶界限的单位是Byte，表示每个事件的数据大小）

5.3.3 `apiserver_watch_events_sizes_sum`

作用：统计了各种Watch事件的总大小，单位是Byte。

5.3.4 `apiserver_watch_events_total`

作用：统计了Watch事件的总数

查询语句：

apiserver_watch_events_total{instance="10.10.239.101:6443",kind="Pod"}

查询结果

apiserver_watch_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="Pod",version="v1"}	1489384

说明

 [root@k8s-dev-m01 ~]# kubectl get pods -A|grep -v NAMESPACE|wc -l
 77

5.4.2 `apiserver_storage_envelope_transformation_cache_misses_total`

作用：存储层中，密钥封装 (Envelope Encryption) 转换过程中缓存未命中的总次数。

在启用数据加密功能时，API Server 会通过 Envelope Encryption 对存储的敏感数据（如密钥、密码）进行加解密操作。为了提高性能，API Server 使用缓存来存储加解密密钥。

6. 其他

6.1 cache list

6.1.1 `apiserver_cache_list_total`

作用：表示缓存中与不同资源相关的对象的总数

说明：

查询语句：

apiserver_cache_list_total{instance="10.10.239.101:6443"}

查询结果

apiserver_cache_list_total{index="f:spec.nodeName",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/pods"}	14
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/apiextensions.k8s.io/customresourcedefinitions"}	10
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/apiregistration.k8s.io/apiservices"}	10
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/catalog.cattle.io/apps"}	7
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/catalog.cattle.io/clusterrepos"}	7
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/catalog.cattle.io/operations"}	7
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/certificatesigningrequests"}	8
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/clusterrolebindings"}	11
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/clusterroles"}	10
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/configmaps"}	12
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/controllerrevisions"}	6
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/controllers"}	11
apiserver_cache_list_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource_prefix="/cronjobs"}	9

6.1.2 `apiserver_cache_list_fetched_objects_total`

作用：表示从缓存中实际提取并返回给客户端的不同资源相关对象的总数

6.1.3 `apiserver_cache_list_returned_objects_total`

作用：表示从缓存中返回给客户端的不同资源相关对象的总数

6.2 client

6.2.1 `apiserver_client_certificate_expiration_seconds_count`

作用：客户端证书过期时间检查次数的计数

注意：不是服务器的证书，而是和服务器链接的客户端证书。包括比如开发电脑远程链接

说明：

查询语句：

rate(apiserver_client_certificate_expiration_seconds_count[10m])

查询结果

{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux"}	0.12374644444444445
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m02",kubernetes_io_os="linux"}	0.12567777777777778
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m03",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m03",kubernetes_io_os="linux"}	0.12477644444444445
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-n01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n01",kubernetes_io_os="linux"}	0.12544933333333333
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-n02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n02",kubernetes_io_os="linux"}	0.1262991111111111
{instance="10.10.239.101:6443",job="kubernetes-apiservers"}	4.586828611111111
{instance="10.10.239.102:6443",job="kubernetes-apiservers"}	1.2532182962962963
{instance="10.10.239.103:6443",job="kubernetes-apiservers"}	0.6262733333333332

说明：以最后一条为例，10分钟内，链接10.10.239.103:6443的客户端过期事件为每秒0.6件。

6.2.2 `apiserver_client_certificate_expiration_seconds_bucket`

作用：客户端证书数量累计值，放到不同的桶中

示例

查询语句

rate(apiserver_client_certificate_expiration_seconds_bucket{instance="10.10.239.103:6443",job="kubernetes-apiservers"}[10m])

结果

{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="+Inf"}	0.6204834834834836
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="0"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="1.5552e+07"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="172800"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="1800"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="2.592e+06"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="21600"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="3.1104e+07"}	0.6204834834834836
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="345600"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="3600"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="43200"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="604800"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="7.776e+06"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="7200"}	0
{instance="10.10.239.103:6443",job="kubernetes-apiservers",le="86400"}	0

说明

过期时间范围（秒）	标签（le）	每秒新增速率值
<= 0	`le="0"`	0
<= 1,800（30 分钟）	`le="1800"`	0
<= 3,600（1 小时）	`le="3600"`	0
<= 7,200（2 小时）	`le="7200"`	0
<= 21,600（6 小时）	`le="21600"`	0
<= 43,200（12 小时）	`le="43200"`	0
<= 86,400（1 天）	`le="86400"`	0
<= 172,800（2 天）	`le="172800"`	0
<= 345,600（4 天）	`le="345600"`	0
<= 604,800（7 天）	`le="604800"`	0
<= 2,592,000（30 天）	`le="2.592e+06"`	0
<= 7,776,000（90 天）	`le="7.776e+06"`	0
<= 15,552,000（180 天）	`le="1.5552e+07"`	0
<= 31,104,000（1 年）	`le="3.1104e+07"`	0.6204834834834836
无限制（+∞）	`le="+Inf"`	0.6204834834834836

27. `apiserver_client_certificate_expiration_seconds_sum`

作用：所有客户端证书过期时间和的累计值

其实不理解这个指标有什么意义，虽然它的rate值除以 count的rate值是证书平均过期时间，但是这种平均过期时间有什么用呢？乞丐和马云的平均值是个富翁，就可以防止乞丐饿死么？

6.3 x509 SAN字段

6.3.1 `apiserver_webhooks_x509_missing_san_total`

作用：处理的 Webhook 请求中，证书缺少 SAN（Subject Alternative Name）字段的次数

SAN 是 X.509 证书的重要字段，通常包含了证书的有效主机名或 IP 地址。如果缺少 SAN，可能会导致证书校验失败或不符合标准

示例：

apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m01",kubernetes_io_os="linux"}	0
apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m02",kubernetes_io_os="linux"}	0
apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-m03",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-m03",kubernetes_io_os="linux"}	0
apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-n01",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n01",kubernetes_io_os="linux"}	0
apiserver_webhooks_x509_missing_san_total{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="k8s-dev-n02",job="kubernetes-nodes",kubernetes_io_arch="amd64",kubernetes_io_hostname="k8s-dev-n02",kubernetes_io_os="linux"}	0
apiserver_webhooks_x509_missing_san_total{instance="10.10.239.101:6443",job="kubernetes-apiservers"}	0
apiserver_webhooks_x509_missing_san_total{instance="10.10.239.102:6443",job="kubernetes-apiservers"}	0
apiserver_webhooks_x509_missing_san_total{instance="10.10.239.103:6443",job="kubernetes-apiservers"}	0

6.3.2 `apiserver_kube_aggregator_x509_missing_san_total`

作用：通过聚合器的请求中，证书缺少 SAN（Subject Alternative Name）字段的次数

6.4 其他

6.4.1 `apiserver_envelope_encryption_dek_cache_fill_percent`

作用：用于信封加密的数据加密密钥（DEK）缓存的填充百分比

6.4.2 `apiserver_init_events_total`

作用：统计了初始化事件的总数
示例：

apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*coordination.Lease"}	422
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.ConfigMap"}	23
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Endpoints"}	183
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Namespace"}	1646
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Node"}	3197
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.PersistentVolume"}	1
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.PersistentVolumeClaim"}	2
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Pod"}	1110
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*core.Secret"}	31
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*discovery.EndpointSlice"}	20
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*storage.StorageClass"}	1
apiserver_init_events_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="*unstructured.Unstructured"}	108

6.4.3 `apiserver_longrunning_gauge`

作用：衡量长时间运行的请求的数量。
示例：

apiserver_longrunning_gauge{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="cluster",verb="WATCH",version="v1"}	4
apiserver_longrunning_gauge{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="WATCH",version="v1"}	0

6.4.4 `apiserver_longrunning_requests`

作用：统计了正在处理的长时间运行的请求总数。
示例：

apiserver_longrunning_requests{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="cluster",verb="WATCH",version="v1"}	4
apiserver_longrunning_requests{component="apiserver",group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="deployments",scope="namespace",verb="WATCH",version="v1"}	0

6.4.5 `apiserver_registered_watchers`

作用：有多少个客户端正在通过 watch 功能注册并监听指定种类对象的资源的变化
示例：

apiserver_registered_watchers{group="apps",instance="10.10.239.101:6443",job="kubernetes-apiservers",kind="Deployment",version="v1"}	4

6.4.6 `apiserver_selfrequest_total`

作用：记录发出的自请求的总数
示例：

apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="namespaces",verb="GET"}	2218318
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="namespaces",verb="LIST"}	2
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="namespaces",verb="WATCH"}	32921
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="nodes",verb="LIST"}	2
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="nodes",verb="WATCH"}	32892
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="persistentvolumes",verb="LIST"}	2
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="persistentvolumes",verb="WATCH"}	32903
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="poddisruptionbudgets",verb="LIST"}	1
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="pods",verb="LIST"}	2
apiserver_selfrequest_total{instance="10.10.239.101:6443",job="kubernetes-apiservers",resource="pods",verb="WATCH"}	32876

6.4.7 `apiserver_tls_handshake_errors_total`

作用：统计在 TLS 握手过程中发生的错误总数

prometheus监控k8s的metric详解-01（apiserver部分）

文章目录

1. apiserver admission（准入）和audit（审计）

1.1 准入控制器调用次数

1.1.1 apiserver_admission_controller_admission_duration_seconds_count

1.1.2 apiserver_admission_controller_admission_duration_seconds_bucket

1.1.3 apiserver_admission_controller_admission_duration_seconds_sum

1.2 step（准入步骤）

1.2.1 apiserver_admission_step_admission_duration_seconds_count

1.2.2 apiserver_admission_step_admission_duration_seconds_bucket

1.2.3 apiserver_admission_step_admission_duration_seconds_sum

1.3 summary（摘要）

1.3.1 apiserver_admission_step_admission_duration_seconds_summary

1.3.2 apiserver_admission_step_admission_duration_seconds_summary_count

1.3.3 apiserver_admission_step_admission_duration_seconds_summary_sum

1.4 webhook

1.4.1 apiserver_admission_webhook_admission_duration_seconds_count

1.4.2 apiserver_admission_webhook_admission_duration_seconds_bucket

1.4.5 apiserver_admission_webhook_admission_duration_seconds_sum

1.4.6 apiserver_admission_webhook_rejection_count

1.4.7 apiserver_admission_webhook_request_total

1.5 审计

1.5.1 apiserver_audit_event_total

1.5.2 apiserver_audit_requests_rejected_total

2. apiserver delegated（委托）

2.1 delegated authn（委托认证）

2.1.1 apiserver_delegated_authn_request_duration_seconds_count

2.1.2 apiserver_delegated_authn_request_duration_seconds_bucket

2.1.3 apiserver_delegated_authn_request_duration_seconds_sum

2.1.4 apiserver_delegated_authn_request_total

2.2 delegated authz（委托授权）

2.2.1 apiserver_delegated_authz_request_duration_seconds_count

2.2.2 apiserver_delegated_authz_request_duration_seconds_bucket

2.2.3 apiserver_delegated_authz_request_duration_seconds_sum

2.2.5 apiserver_delegated_authz_request_total

3. apiserver flowcontrol

3.1 请求和速率（待整理）

3.1.1 apiserver_flowcontrol_current_executing_requests

3.1.2 apiserver_flowcontrol_current_inqueue_requests

3.1.3 apiserver_flowcontrol_current_r（未验证）

3.1.4 apiserver_flowcontrol_dispatched_requests_total

3.1.5 apiserver_flowcontrol_dispatch_r（未验证）

3.1.6 apiserver_flowcontrol_latest_s

3.1.7 apiserver_flowcontrol_next_discounted_s_bounds

3.1.8 apiserver_flowcontrol_next_s_bounds

3.2 priority level（优先级别）

3.2.2 apiserver_flowcontrol_priority_level_request_count_samples_count

3.2.1 apiserver_flowcontrol_priority_level_request_count_samples_bucket

3.2.3 apiserver_flowcontrol_priority_level_request_count_samples_sum

3.2.4 apiserver_flowcontrol_priority_level_request_count_watermarks_count

3.2.5 apiserver_flowcontrol_priority_level_request_count_watermarks_bucket

3.2.6 apiserver_flowcontrol_priority_level_request_count_watermarks_sum

3.2.7 apiserver_flowcontrol_priority_level_seat_count_samples_count

3.2.8 apiserver_flowcontrol_priority_level_seat_count_samples_bucket

3.2.9 apiserver_flowcontrol_priority_level_seat_count_samples_sum

3.2.12 apiserver_flowcontrol_priority_level_seat_count_watermarks_count

3.2.11 apiserver_flowcontrol_priority_level_seat_count_watermarks_bucket

3.2.13 apiserver_flowcontrol_priority_level_seat_count_watermarks_sum

3.3 读写请求

3.3.1 apiserver_flowcontrol_read_vs_write_request_count_samples_count

3.3.2 apiserver_flowcontrol_read_vs_write_request_count_samples_bucket

3.3.3 apiserver_flowcontrol_read_vs_write_request_count_samples_sum

3.3.4 apiserver_flowcontrol_read_vs_write_request_count_watermarks_count

3.3.5 apiserver_flowcontrol_read_vs_write_request_count_watermarks_bucket

3.3.6 apiserver_flowcontrol_read_vs_write_request_count_watermarks_sum

3.4 request（请求）

3.4.1 apiserver_flowcontrol_request_concurrency_in_use

3.4.2 apiserver_flowcontrol_request_concurrency_limit

3.4.3 apiserver_flowcontrol_request_execution_seconds_count

3.4.4 apiserver_flowcontrol_request_execution_seconds_bucket

3.4.5 apiserver_flowcontrol_request_execution_seconds_sum

3.4.6 apiserver_flowcontrol_request_queue_length_after_enqueue_count

3.4.7 apiserver_flowcontrol_request_queue_length_after_enqueue_bucket

3.4.8 apiserver_flowcontrol_request_queue_length_after_enqueue_sum

3.4.10 apiserver_flowcontrol_request_wait_duration_seconds_count

3.4.9 apiserver_flowcontrol_request_wait_duration_seconds_bucket

3.4.12 apiserver_flowcontrol_request_wait_duration_seconds_sum

4. request 和 response

4.1 请求次数

4.1.1 apiserver_request_duration_seconds_count

1.1.1 `apiserver_admission_controller_admission_duration_seconds_count`

1.1.2 `apiserver_admission_controller_admission_duration_seconds_bucket`

1.1.3 `apiserver_admission_controller_admission_duration_seconds_sum`

1.2.1 `apiserver_admission_step_admission_duration_seconds_count`

1.2.2 `apiserver_admission_step_admission_duration_seconds_bucket`

1.2.3 `apiserver_admission_step_admission_duration_seconds_sum`

1.3.1 `apiserver_admission_step_admission_duration_seconds_summary`

1.3.2 `apiserver_admission_step_admission_duration_seconds_summary_count`

1.3.3 `apiserver_admission_step_admission_duration_seconds_summary_sum`

1.4.1 `apiserver_admission_webhook_admission_duration_seconds_count`

1.4.2 `apiserver_admission_webhook_admission_duration_seconds_bucket`

1.4.5 `apiserver_admission_webhook_admission_duration_seconds_sum`

1.4.6 `apiserver_admission_webhook_rejection_count`

1.4.7 `apiserver_admission_webhook_request_total`

1.5.1 `apiserver_audit_event_total`

1.5.2 `apiserver_audit_requests_rejected_total`

2.1.1 `apiserver_delegated_authn_request_duration_seconds_count`

2.1.2 `apiserver_delegated_authn_request_duration_seconds_bucket`

2.1.3 `apiserver_delegated_authn_request_duration_seconds_sum`

2.1.4 `apiserver_delegated_authn_request_total`

2.2.1 `apiserver_delegated_authz_request_duration_seconds_count`

2.2.2 `apiserver_delegated_authz_request_duration_seconds_bucket`

2.2.3 `apiserver_delegated_authz_request_duration_seconds_sum`

2.2.5 `apiserver_delegated_authz_request_total`

3.1.1 `apiserver_flowcontrol_current_executing_requests`

3.1.2 `apiserver_flowcontrol_current_inqueue_requests`

3.1.3 `apiserver_flowcontrol_current_r`（未验证）

3.1.4 `apiserver_flowcontrol_dispatched_requests_total`

3.1.5 `apiserver_flowcontrol_dispatch_r`（未验证）

3.1.6 `apiserver_flowcontrol_latest_s`

3.1.7 `apiserver_flowcontrol_next_discounted_s_bounds`

3.1.8 `apiserver_flowcontrol_next_s_bounds`

3.2.2 `apiserver_flowcontrol_priority_level_request_count_samples_count`

3.2.1 `apiserver_flowcontrol_priority_level_request_count_samples_bucket`

3.2.3 `apiserver_flowcontrol_priority_level_request_count_samples_sum`

3.2.4 `apiserver_flowcontrol_priority_level_request_count_watermarks_count`

3.2.5 `apiserver_flowcontrol_priority_level_request_count_watermarks_bucket`

3.2.6 `apiserver_flowcontrol_priority_level_request_count_watermarks_sum`

3.2.7 `apiserver_flowcontrol_priority_level_seat_count_samples_count`

3.2.8 `apiserver_flowcontrol_priority_level_seat_count_samples_bucket`

3.2.9 `apiserver_flowcontrol_priority_level_seat_count_samples_sum`

3.2.12 `apiserver_flowcontrol_priority_level_seat_count_watermarks_count`

3.2.11 `apiserver_flowcontrol_priority_level_seat_count_watermarks_bucket`

3.2.13 `apiserver_flowcontrol_priority_level_seat_count_watermarks_sum`

3.3.1 `apiserver_flowcontrol_read_vs_write_request_count_samples_count`

3.3.2 `apiserver_flowcontrol_read_vs_write_request_count_samples_bucket`

3.3.3 `apiserver_flowcontrol_read_vs_write_request_count_samples_sum`

3.3.4 `apiserver_flowcontrol_read_vs_write_request_count_watermarks_count`

3.3.5 `apiserver_flowcontrol_read_vs_write_request_count_watermarks_bucket`

3.3.6 `apiserver_flowcontrol_read_vs_write_request_count_watermarks_sum`

3.4.1 `apiserver_flowcontrol_request_concurrency_in_use`

3.4.2 `apiserver_flowcontrol_request_concurrency_limit`

3.4.3 `apiserver_flowcontrol_request_execution_seconds_count`

3.4.4 `apiserver_flowcontrol_request_execution_seconds_bucket`

3.4.5 `apiserver_flowcontrol_request_execution_seconds_sum`

3.4.6 `apiserver_flowcontrol_request_queue_length_after_enqueue_count`

3.4.7 `apiserver_flowcontrol_request_queue_length_after_enqueue_bucket`

3.4.8 `apiserver_flowcontrol_request_queue_length_after_enqueue_sum`

3.4.10 `apiserver_flowcontrol_request_wait_duration_seconds_count`

3.4.9 `apiserver_flowcontrol_request_wait_duration_seconds_bucket`

3.4.12 `apiserver_flowcontrol_request_wait_duration_seconds_sum`

4.1.1 `apiserver_request_duration_seconds_count`

4.1.2 `apiserver_request_duration_seconds_bucket`

4.1.3 `apiserver_request_duration_seconds_sum`

4.1.4 `apiserver_current_inflight_requests`

4.1.5 `apiserver_current_inqueue_requests`

4.2.1 `apiserver_request_filter_duration_seconds_count`

4.2.2 `apiserver_request_filter_duration_seconds_bucket`

4.2.3 `apiserver_request_filter_duration_seconds_sum`

4.3.1 `apiserver_request_slo_duration_seconds_count`

4.3.2 `apiserver_request_slo_duration_seconds_bucket`

4.3.3 `apiserver_request_slo_duration_seconds_sum`

4.4.1 `apiserver_requested_deprecated_apis`

4.4.2 `apiserver_request_post_timeout_total`

4.4.3 `apiserver_request_terminations_total`

4.4.4 `apiserver_request_total`

4.5.2 `apiserver_response_sizes_count`

4.5.2 `apiserver_response_sizes_bucket`

4.5.3 `apiserver_response_sizes_sum`

5.1.1 `apiserver_storage_data_key_generation_duration_seconds_count`