{ "componentChunkName": "component---src-templates-issues-tsx", "path": "/issues/324", "result": {"data":{"issuesJson":{"id":"65e3007f-38c5-550f-96d0-7cabd6d1a705","title":"postfinance/kubenurse: 监控 K8S 集群中的所有网络连接、测量请求持续时间、记录错误。以 Prometheus 格式导出这些指标","number":324,"bodyHTML":"

\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Repos	postfinance/kubenurse
Views	$\"views\"$
Stars	$\"stars\"$
Forks	$\"forks\"$
License	$\"license\"$
UpdatedAt	$\"last-commit\"$
CreatedAt	$\"create-at\"$

\n\n

\n

\n

\n

$\"CI\"$
\n $\"Coverage$
\n $\"GitHub$

\n\n\n\n

Kubenurse\n
- Grafana dashboard
- Metrics
- Deployment\n
  - Helm deployment\n
    - Helm parameters
    \n
  - Configuration
  \n
- HTTP Endpoints
- Health Checks\n
  - API Server Direct
  - API Server DNS
  - Me Ingress
  - Me Service
  - Neighbourhood
  \n
- Neighbourhood filtering\n
  - Neighbourhood incoming checks metric
  \n
\n

\n\n

Kubenurse

\n

Kubenurse is a little service that monitors all network connections in a
\nKubernetes cluster. Kubenurse measures request durations, records errors and
\nexports those metrics in Prometheus format.

\n

Here's an overview of the checks performed by kubenurse, which are exposed as
\nlabels for the various duration/error prometheus metrics.

\n

\n

Grafana dashboard

\n

Once the kubenurse pods are up and running and scraped by your metrics agent,
\nyou can import the example dashboard to start
\nscrutinizing network latencies and errors.

\n

\n

Metrics

\n

All performed checks expose metrics which can be used to monitor/alert:

\n

node-to-node network latencies and errors
pod-to-apiserver communication
Ingress roundtrip latencies and errors
Service roundtrip latencies and errors (kube-proxy / your CNI)
Major kube-apiserver issues
kube-dns (or CoreDNS) errors
External DNS resolution errors (ingress URL resolution)

\n

At /metrics you will find the following metrics:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

metric name	labels	description
~~`kubenurse request duration`~~	`type`	(deprecated since v1.13.0) latency histogram for request duration, replaced with the metric below.
`kubenurse httpclient request duration seconds`	`type`	latency histogram for request duration, partitioned by request type
`kubenurse httpclient trace request duration seconds`	`type, event`	latency histogram for httpclient trace metric instrumentation, partitioned by request type and httptrace connection events
`kubenurse httpclient requests total`	`type, code, method`	counter for the total number of http requests, partitioned by HTTP code, method, and request type
`kubenurse errors total`	`type, event`	error counter, partitioned by httptrace event and request type
`kubenurse neighbourhood incoming checks`	n\\a	gauge which reports how many unique neighbours have queried the current pod in the last minute

\n

For metrics partitioned with a type label, it is possible to precisely know
\nwhich request type increased an error counter, or to compare the latencies of
\nmultiple request types, for example compare how your service and ingress
\nlatencies differ.

\n

Some event labels include dns_start, got_conn, tls_handshake_done, and
\nmore. the details can be seen in the
\nhttptrace.go
\nfile.

\n

Deployment

\n

You can get the Docker image from Docker Hub.
\nThe examples directory
\ncontains manifests which can be used to deploy kubenurse to the kube-system namespace of your cluster.

\n

Helm deployment

\n

You can also deploy kubenurse with Helm, the Chart can be found in repository https://postfinance.github.io/kubenurse/ or directory ./helm/kubenurse/.
\nThe following command can be used to install kubenurse with Helm: helm upgrade [RELEASE_NAME] --install --repo https://postfinance.github.io/kubenurse/ kubenurse.

\n

Helm parameters

\n

\n

helm parameters list

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Setting	Description	Default
daemonset.image.repository	The repository name	`postfinance/kubenurse`
daemonset.image.tag	The tag/ version of the image	`v1.4.0`
daemonset.podLabels	Additional labels to be added to the pods of the daemonset	`[]`
daemonset.podAnnotations	Additional annotations to be added to the pods of the daemonset	`[]`
daemonset.podSecurityContext	The security context of the daemonset	`{}`
daemonset.priorityClassName	The priority class name for the daemonset pods	`\"\"`
daemonset.containerSecurityContext	The security context of the containers within the pods of the daemonset	`{}`
daemonset.containerResources	The container resources of the containers within the pods of the daemonset	`{}`
daemonset.containerImagePullPolicy	The container image pull policy the pods of the daemonset	`IfNotPresent`
daemonset.tolerations	The tolerations of the daemonset	See Default tolerations below
daemonset.dnsConfig	Specifies the DNS parameters of the pods in the daemonset	`{}`
daemonset.volumeMounts	Additional volumeMounts to be added to the pods of the daemonset	`[]`
daemonset.volumes	Additional volumes to be added to the daemonset	`[]`
daemonset.rollingUpdate.maxUnavailable	The maximum number of DaemonSet pods that can be unavailable during the update	`34%`
daemonset.rollingUpdate.maxSurge	The maximum number of nodes with an existing available DaemonSet pod that can have an updated pod during an update
serviceMonitor.enabled	Adds a ServiceMonitor for use with Prometheus-operator	`false`
serviceMonitor.labels	Additional labels to be added to the ServiceMonitor	`{}`
serviceMonitor.relabelings	Additional relabelings to be added to the endpoint of the ServiceMonitor	`[]`
serviceAccount.name	The name of the service account which is used	`Release.Name`
service.name	The name of service which exposes the kubenurse application	`8080-8080`
service.port	The port number of the service	`8080`
service.labels	Additional labels to be added to the Service
ingress.enabled	Enable/ Disable the ingress	`true`
ingress.className	The classname of the ingress controller (e.g. the nginx ingress controller)	`nginx`
ingress.url	The url of the ingress; e.g. kubenurse.westeurope.cloudapp.example.com	`dummy-kubenurse.example.com`
insecure	Set `KUBENURSE_INSECURE` environment variable	`true`
allow_unschedulable	Sets `KUBENURSE_ALLOW_UNSCHEDULABLE` environment variable	`false`
neighbour_filter	Sets `KUBENURSE_NEIGHBOUR_FILTER` environment variable	`app.kubernetes.io/name=kubenurse`
neighbour_limit	Sets `KUBENURSE_NEIGHBOUR_LIMIT` environment variable	`10`
histogram_buckets	Sets `KUBENURSE_HISTOGRAM_BUCKETS` environment variable
extra_ca	Sets `KUBENURSE_EXTRA_CA` environment variable
extra_checks	Sets `KUBENURSE_EXTRA_CHECKS` environment variable
kubernetes_service_dns	Sets `KUBERNETES_SERVICE_DNS` environment variable
check_api_server_direct	Sets `KUBENURSE_CHECK_API_SERVER_DIRECT` environment variable	`true`
check_api_server_dns	Sets `KUBENURSE_CHECK_API_SERVER_DNS` environment variable	`true`
check_me_ingress	Sets `KUBENURSE_CHECK_ME_INGRESS` environment variable	`true`
check_me_service	Sets `KUBENURSE_CHECK_ME_SERVICE` environment variable	`true`
check_neighbourhood	Sets `KUBENURSE_CHECK_NEIGHBOURHOOD` environment variable	`true`
check_interval	Sets `KUBENURSE_CHECK_INTERVAL` environment variable	`5s`
reuse_connections	Sets `KUBENURSE_REUSE_CONNECTIONS` environment variable	`false`
use_tls	Sets `KUBENURSE_USE_TLS` environment variable	`false`
cert_file	Sets `KUBENURSE_CERT_FILE` environment variable
cert_key	Sets `KUBENURSE_CERT_KEY` environment variable

\n

\n

Configuration

\n

\n

kubenurse environment variables list

\n

KUBENURSE_INGRESS_URL: An URL to the kubenurse in order to check the ingress
KUBENURSE_SERVICE_URL: An URL to the kubenurse in order to check the Kubernetes service
KUBENURSE_INSECURE: If \"true\", TLS connections will not validate the certificate
KUBENURSE_EXTRA_CA: Additional CA cert path for TLS connections
KUBENURSE_EXTRA_CHECKS: Additional checks, specified as a list (separated by a vertical bar |) where each entry of the list has the format: <metric_name>:<url_to_check>. For example google:https://www.google.ch/|cloudflare:https://www.cloudflare.com/
KUBENURSE_NAMESPACE: Namespace in which to look for the neighbour kubenurses
KUBENURSE_NEIGHBOUR_FILTER: A Kubernetes label selector (eg. app=kubenurse) to filter neighbour kubenurses
KUBENURSE_NEIGHBOUR_LIMIT: The maximum number of neighbours each kubenurse will query
KUBENURSE_ALLOW_UNSCHEDULABLE: If this is \"true\", path checks to neighbouring kubenurses are made even if they are running on unschedulable nodes.
KUBENURSE_CHECK_API_SERVER_DIRECT: If this is \"true\" kubenurse will perform the check [API Server Direct](#API Server Direct). default is \"true\"
KUBENURSE_CHECK_API_SERVER_DNS: If this is \"true\", kubenurse will perform the check [API Server DNS](#API Server DNS). default is \"true\"
KUBENURSE_CHECK_ME_INGRESS: If this is \"true\", kubenurse will perform the check [Me Ingress](#Me Ingress). default is \"true\"
KUBENURSE_CHECK_ME_SERVICE: If this is \"true\", kubenurse will perform the check [Me Service](#Me Service). default is \"true\"
KUBENURSE_CHECK_NEIGHBOURHOOD: If this is \"true\", kubenurse will perform the check Neighbourhood. default is \"true\"
KUBENURSE_CHECK_INTERVAL: the frequency to perform kubenurse checks. the string should be formatted for time.ParseDuration. defaults to 5s
KUBENURSE_REUSE_CONNECTIONS: whether to reuse connections or not for all checks. default is \"false\"
KUBENURSE_HISTOGRAM_BUCKETS: optional comma-separated list of float64, used in place of the default prometheus histogram buckets
KUBENURSE_USE_TLS: If this is \"true\", enable TLS endpoint on port 8443
KUBENURSE_CERT_FILE: Certificate to use with TLS endpoint
KUBENURSE_CERT_KEY: Key to use with TLS endpoint

\n

Following variables are injected to the Pod by Kubernetes and should not be defined manually:

\n

KUBERNETES_SERVICE_HOST: Host to communicate to the kube-apiserver
KUBERNETES_SERVICE_PORT: Port to communicate to the kube-apiserver

\n

The DNS name of the API server can be configured with the
\nKUBERNETES_SERVICE_DNS environment variable, and it defaults to
\nkubernetes.default.svc.cluster.local.

\n

\n

HTTP Endpoints

\n

The kubenurse service listens for http requests on port 8080 (optionally https on port 8443) and exposes endpoints:

\n

/: Redirects to /alive
/alive: Returns a pretty printed JSON with the check results, described below
/alwayshappy: Returns http-200 which is used for testing itself
/metrics: Exposes Prometheus metrics

\n

The /alive endpoint returns a JSON like this with status code 200 if everything is OK else 500:

\n

{\n  \"api_server_direct\": \"ok\",\n  \"api_server_dns\": \"ok\",\n  \"me_ingress\": \"ok\",\n  \"me_service\": \"ok\",\n  \"hostname\": \"kubenurse-1234-x2bwx\",\n  \"neighbourhood_state\": \"ok\",\n  \"neighbourhood\": [\n   {\n    \"PodName\": \"kubenurse-1234-8fh2x\",\n    \"PodIP\": \"10.10.10.67\",\n    \"HostIP\": \"10.12.12.66\",\n    \"NodeName\": \"k8s-66.example.com\",\n    \"Phase\": \"Running\"\n   },\n   {\n    \"PodName\": \"kubenurse-1234-ffjbs\",\n    \"PodIP\": \"10.10.10.138\",\n    \"HostIP\": \"10.12.12.89\",\n    \"NodeName\": \"k8s-89.example.com\",\n    \"Phase\": \"Running\"\n   }\n  ],\n  \"headers\": {\n   \"Accept\": [\n    \"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\"\n   ],\n   \"Accept-Encoding\": [\n    \"gzip, deflate, br\"\n   ],\n   ...\n  }\n}

\n

Health Checks

\n

Every five seconds, the checks described below are run.

\n

API Server Direct

\n

Checks the /version endpoint of the Kubernetes API Server through
\nthe direct link (KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT).

\n

Metric type: api_server_direct

\n

API Server DNS

\n

Checks the /version endpoint of the Kubernetes API Server through
\nthe Cluster DNS URL https://kubernetes.default.svc:$KUBERNETES_SERVICE_PORT.
\nThis also verifies a working kube-dns deployment.

\n

Metric type: api_server_dns

\n

Me Ingress

\n

Checks if the kubenurse is reachable at the /alwayshappy endpoint behind the ingress.
\nThis address is provided by the environment variable KUBENURSE_INGRESS_URL that
\ncould look like https://kubenurse.example.com.
\nThis also verifies a correct upstream DNS resolution.

\n

Metric type: me_ingress

\n

Me Service

\n

Checks if the kubenurse is reachable at the /alwayshappy endpoint through the Kubernetes service.
\nThe address is provided by the environment variable KUBENURSE_SERVICE_URL that
\ncould look like http://kubenurse.mynamespace.default.svc:8080.
\nThis also verifies a working kube-proxy setup.

\n

Metric type: me_service

\n

Neighbourhood

\n

Checks if every neighbour kubenurse is reachable at the /alwayshappy endpoint.
\nNeighbours are discovered by querying the kube-apiserver for every Pod in the
\nKUBENURSE_NAMESPACE with label KUBENURSE_NEIGHBOUR_FILTER.
\nThe request is done directly to the Pod-IP (port 8080, or 8443 if TLS is enabled) and the metric types contains the prefix
\npath_ and the hostname of the kubelet on which the neighbour kubenurse should run.
\nOnly kubenurses on nodes that are schedulable are considered as neighbours,
\nthis can be changed by setting KUBENURSE_ALLOW_UNSCHEDULABLE=\"true\".

\n

Metric type: path_$KUBELET_HOSTNAME

\n

Neighbourhood filtering

\n

The number of checks for the neighbourhood used to grow as $O(N^2)$, which
\nrendered kubenurse impractical on large clusters, as documented in issue
\n#55.
\nTo combat this, a node filtering feature was implemented, which works as follows

\n

kubenurse computes the sha256 checksums for all neighbours' node names
it sorts those checksums (this is actually implemented with a max-heap)
it computes its own node name checksum, and queries the next 10 (per default)
\nnodes in the sorted checksums list

\n

Here's an example with 6 nodes, where each node queries the next 3 nodes:

\n

\n

Thanks to this, every node is making queries to the same 10 nodes, unless one
\nof those nodes disappears, in which case kubenurse will pick the next node in
\nthe sorted checksums list. This comes with several advantages:

\n

because of the way we first hash the node names, the checks are randomly
\ndistributed, independant of the node names. if we only picked the 10 next
\nnodes in a sorted list of the node names, then we might have biased the
\nresults in environments where node names are sequential
metrics-wise, a kubenurse pod should typically only have entries for ca. 10
\nother neighbouring nodes worth of checks, which greatly reduces the load on
\nyour monitoring infrastructure
because we use a deterministic algorithm to choose which nodes to query, the
\nmetrics churn rate stays minimal. (that is, if we randomly picked 10 nodes
\nfor every check, then in the end there would be one prometheus bucket for
\nevery node on the cluster, which would put useless load on the monitoring
\ninfrastructure)

\n

Per default, the neighbourhood filtering is set to 10 nodes, which means that
\non cluster with more than 10 nodes, each kubenurse will query exactly 10 nodes,
\nas described above.

\n

Neighbourhood incoming checks metric

\n

It is possible to check that each node receives the proper number of
\nneighbourhood queries with the kubenurse_neighbourhood_incoming_checks
\nmetric. If you have the neighbourhood limit set to e.g. 10, then this
\nmetric should be equal to 10 on all nodes, with some variations during a
\nrollout restart.

\n

To bypass the node filtering feature, you simply need to set the
\nKUBENURSE_NEIGHBOUR_LIMIT environment variable to 0.

","updatedAt":"2025-05-21T16:08:29Z","upvoteCount":null,"author":{"login":"eryajf","avatarUrl":"https://avatars.githubusercontent.com/u/33259379?u=e4a4090a38ac2473aaed4ef9945233636776c6c3&v=4","url":"https://github.com/eryajf"},"category":null,"labels":{"edges":[{"node":{"name":"更多","color":"25B472"}},{"node":{"name":"K8S-Tools","color":"2d411a"}},{"node":{"name":"postfinance","color":"3b7fa2"}}]},"comments":{"edges":[]}}},"pageContext":{"number":324,"previous":{"title":"noovertime7/kubemanage: 一个简单易用的 K8s 管理平台，前端使用 Vue3 ，后端使用 Gin+Gorm ，未来将支持多集群接入","number":323},"next":{"title":"teamcode-inc/kubeorbit: 轻量高效的微服务本地联调测试工具, 轻松将集群内流量转发到本地开发环境","number":325}}}, "staticQueryHashes": ["151096407","2861350382"]}