{ "componentChunkName": "component---src-templates-issues-tsx", "path": "/issues/329", "result": {"data":{"issuesJson":{"id":"e2efd05b-5185-52dd-8d66-9c0024f4273a","title":"resmoio/kubernetes-event-exporter: 将 K8s 的事件同步到其他平台中，比如 ES，Kafka，file，Webhook 等","number":329,"bodyHTML":"

\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Repos	resmoio/kubernetes-event-exporter
Views	$\"views\"$
Stars	$\"stars\"$
Forks	$\"forks\"$
License	$\"license\"$
UpdatedAt	$\"last-commit\"$
CreatedAt	$\"create-at\"$

\n\n

\n

\n

\n

kubernetes-event-exporter

\n

\n
Note: This is an active fork of Opsgenie Kubernetes Event Exporter
\nsince that is not maintained since November 2021. Development is sponsored by Resmo.
\n

\n

\n
This tool is presented at KubeCon 2019 San Diego.
\n

\n

This tool allows exporting the often missed Kubernetes events to various outputs so that they can be used for
\nobservability or alerting purposes. You won't believe what you are missing.

\n

Deployment

\n

Head on to deploy/ folder and apply the YAMLs in the given filename order. Do not forget to modify the
\ndeploy/01-config.yaml file to your configuration needs. The additional information for configuration is as follows:

\n

Kustomize

\n

Deploy with Kustomize by Git ref (i.e., commit sha, tag, or branch).

\n

Create namespace before deployment:

\n

kubectl create namespace monitoring

\n

apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\nresources:\n  - https://github.com/resmoio/kubernetes-event-exporter?ref=master

\n

Helm

\n

Please use Bitnami Chart which is
\ncomprehensive.

\n

Configuration

\n

Configuration is done via a YAML file, when run in Kubernetes, ConfigMap. The tool watches all the events and
\nuser has to option to filter out some events, according to their properties. Critical events can be routed to alerting
\ntools such as Opsgenie, or all events can be dumped to an Elasticsearch instance. You can use namespaces, labels on the
\nrelated object to route some Pod related events to owners via Slack. The final routing is a tree which allows
\nflexibility. It generally looks like following:

\n

route:\n  # Main route\n  routes:\n    # This route allows dumping all events because it has no fields to match and no drop rules.\n    - match:\n        - receiver: dump\n    # This starts another route, drops all the events in *test* namespaces and Normal events\n    # for capturing critical events\n    - drop:\n        - namespace: \"*test*\"\n        - type: \"Normal\"\n      match:\n        - receiver: \"critical-events-queue\"\n    # This a final route for user messages\n    - match:\n        - kind: \"Pod|Deployment|ReplicaSet\"\n          labels:\n            version: \"dev\"\n          receiver: \"slack\"\nreceivers:\n# See below for configuring the receivers

\n

A match rule is exclusive, all conditions must be matched to the event.
During processing a route, drop rules are executed first to filter out events.
The match rules in a route are independent of each other. If an event matches a rule, it goes down it's subtree.
If all the match rules are matched, the event is passed to the receiver.
A route can have many sub-routes, forming a tree.
Routing starts from the root route.

\n

Using Secrets

\n

In your config file, you can refer to environment variables as ${API_KEY} therefore you can use ConfigMap or Secrets
\nto keep the config file clean of secrets.

\n

Troubleshoot \"Events Discarded\" warning:

\n

If there are client-side throttling warnings in the event-exporter log:
\nAdjust the following values in configuration:\n
```
kubeQPS: 100\nkubeBurst: 500\n
```
\n
\n
Burst to roughly match your events per minute
\nQPS to be 1/5 of the burst
\n
\n
If there is no request throttling, but events are still dropped:
\nConsider increasing events cut off age\n
```
maxEventAgeSeconds: 60\n
```
\n

\n

Opsgenie

\n

Opsgenie is an alerting and on-call management tool. kubernetes-event-exporter can push to
\nevents to Opsgenie so that you can notify the on-call when something critical happens. Alerting should be precise and
\nactionable, so you should carefully design what kind of alerts you would like in Opsgenie. A good starting point might
\nbe filtering out Normal type of events, while some additional filtering can help. Below is an example configuration.

\n

# ...\nreceivers:\n  - name: \"alerts\"\n    opsgenie:\n      apiKey: xxx\n      priority: \"P3\"\n      message: \"Event {{ .Reason }} for {{ .InvolvedObject.Namespace }}/{{ .InvolvedObject.Name }} on K8s cluster\"\n      alias: \"{{ .UID }}\"\n      description: \"<pre>{{ toPrettyJson . }}</pre>\"\n      tags:\n        - \"event\"\n        - \"{{ .Reason }}\"\n        - \"{{ .InvolvedObject.Kind }}\"\n        - \"{{ .InvolvedObject.Name }}\"

\n

Webhooks/HTTP

\n

Webhooks are the easiest way of integrating this tool to external systems. It allows templating & custom headers which
\nallows you to push events to many possible sources out there. See [Customizing Payload] for more information.

\n

# ...\nreceivers:\n  - name: \"alerts\"\n    webhook:\n      endpoint: \"https://my-super-secret-service.com\"\n      headers:\n        X-API-KEY: \"123\"\n        User-Agent: kube-event-exporter 1.0\n      layout: # Optional

\n

Elasticsearch

\n

Elasticsearch is a full-text, distributed search engine which can also do powerful
\naggregations. You may decide to push all events to Elasticsearch and do some interesting queries over time to find out
\nwhich images are pulled, how often pod schedules happen etc. You
\ncan watch the presentation
\nin Kubecon to see what else you can do with aggregation and reporting.

\n

# ...\nreceivers:\n  - name: \"dump\"\n    elasticsearch:\n      hosts:\n        - http://localhost:9200\n      index: kube-events\n      # Ca be used optionally for time based indices, accepts Go time formatting directives\n      indexFormat: \"kube-events-{2006-01-02}\"\n      username: # optional\n      password: # optional\n      cloudID: # optional\n      apiKey: # optional\n      headers: # optional，Can be used to append the additional key/value pairs into the request headers\n      # If set to true, it allows updating the same document in ES (might be useful handling count)\n      useEventID: true|false\n      # Type should be only used for clusters Version 6 and lower.\n      # type: kube-event\n      # If set to true, all dots in labels and annotation keys are replaced by underscores. Defaults false\n      deDot: true|false\n      layout: # Optional\n      tls: # optional, advanced options for tls\n        insecureSkipVerify: true|false # optional, if set to true, the tls cert won't be verified\n        serverName: # optional, the domain, the certificate was issued for, in case it doesn't match the hostname used for the connection\n        caFile: # optional, path to the CA file of the trusted authority the cert was signed with

\n

OpenSearch

\n

OpenSearch is a community-driven, open source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2.
\nOpenSearch enables people to easily ingest, secure, search, aggregate, view, and analyze data. These capabilities are popular for use cases such as application search, log analytics, and more.
\nYou may decide to push all events to OpenSearch and do some interesting queries over time to find out
\nwhich images are pulled, how often pod schedules happen etc.

\n

# ...\nreceivers:\n  - name: \"dump\"\n    opensearch:\n      hosts:\n        - http://localhost:9200\n      index: kube-events\n      # Ca be used optionally for time based indices, accepts Go time formatting directives\n      indexFormat: \"kube-events-{2006-01-02}\"\n      username: # optional\n      password: # optional\n      # If set to true, it allows updating the same document in ES (might be useful handling count)\n      useEventID: true|false\n      # Type should be only used for clusters Version 6 and lower.\n      # type: kube-event\n      # If set to true, all dots in labels and annotation keys are replaced by underscores. Defaults false\n      deDot: true|false\n      layout: # Optional\n      tls: # optional, advanced options for tls\n        insecureSkipVerify: true|false # optional, if set to true, the tls cert won't be verified\n        serverName: # optional, the domain, the certificate was issued for, in case it doesn't match the hostname used for the connection\n        caFile: # optional, path to the CA file of the trusted authority the cert was signed with

\n

Slack

\n

Slack is a cloud-based instant messaging platform where many people use it for integrations and getting notified by
\nsoftware such as Jira, Opsgenie, Google Calendar etc. and even some implement ChatOps on it. This tool also allows
\nexporting events to Slack channels or direct messages to persons. If your objects in Kubernetes, such as Pods,
\nDeployments have real owners, you can opt-in to notify them via important events by using the labels of the objects. If
\na Pod sandbox changes and it's restarted, or it cannot find the Docker image, you can immediately notify the owner.

\n

# ...\nreceivers:\n  - name: \"slack\"\n    slack:\n      token: YOUR-API-TOKEN-HERE\n      channel: \"@{{ .InvolvedObject.Labels.owner }}\"\n      message: \"{{ .Message }}\"\n      color: # optional\n      title: # optional\n      author_name: # optional\n      footer: # optional\n      fields:\n        namespace: \"{{ .Namespace }}\"\n        reason: \"{{ .Reason }}\"\n        object: \"{{ .Namespace }}\"\n

\n

Kinesis

\n

Kinesis is an AWS service allows to collect high throughput messages and allow it to be used in stream processing.

\n

# ...\nreceivers:\n  - name: \"kinesis\"\n    kinesis:\n      streamName: \"events-pipeline\"\n      region: us-west-2\n      layout: # Optional

\n

Firehose

\n

Firehose is an AWS service providing high throughput message collection for use in stream processing.

\n

# ...\nreceivers:\n  - name: \"firehose\"\n    firehose:\n      deliveryStreamName: \"events-pipeline\"\n      region: us-west-2\n      layout: # Optional

\n

SNS

\n

SNS is an AWS service for highly durable pub/sub messaging system.

\n

# ...\nreceivers:\n  - name: \"sns\"\n    sns:\n      topicARN: \"arn:aws:sns:us-east-1:1234567890123456:mytopic\"\n      region: \"us-west-2\"\n      layout: # Optional

\n

SQS

\n

SQS is an AWS service for message queuing that allows high throughput messaging.

\n

# ...\nreceivers:\n  - name: \"sqs\"\n    sqs:\n      queueName: \"/tmp/dump\"\n      region: us-west-2\n      layout: # Optional

\n

File

\n

For some debugging purposes, you might want to push the events to files. Or you can already have a logging tool that can
\ningest these files and it might be a good idea to just use plain old school files as an integration point.

\n

# ...\nreceivers:\n  - name: \"file\"\n    file:\n      path: \"/tmp/dump\"\n      layout: # Optional

\n

Stdout

\n

Standard out is also another file in Linux. logLevel refers to the application logging severity - available levels
\ntrace, debug, info, warn, error, fatal and panic. When not specified, default level is set to info. You
\ncan use the following configuration as an example.

\n

logLevel: error\nlogFormat: json\nmaxEventAgeSeconds: 5\nroute:\n  routes:\n    - match:\n        - receiver: \"dump\"\nreceivers:\n  - name: \"dump\"\n    stdout:\n      deDot: true|false

\n

Kafka

\n

Kafka is a popular tool used for real-time data pipelines. You can combine it with other tools for further analysis.

\n

receivers:\n  - name: \"kafka\"\n    kafka:\n      clientId: \"kubernetes\"\n      topic: \"kube-event\"\n      brokers:\n        - \"localhost:9092\"\n      compressionCodec: \"snappy\"\n      tls:\n        enable: true\n        certFile: \"kafka-client.crt\"\n        keyFile: \"kafka-client.key\"\n        caFile: \"kafka-ca.crt\"\n      sasl:\n        enable: true\n        username: \"kube-event-producer\"\n        password: \"kube-event-producer-password\"\n        mechanism: \"sha512\"\n      layout: #optional\n        kind: \"{{ .InvolvedObject.Kind }}\"\n        namespace: \"{{ .InvolvedObject.Namespace }}\"\n        name: \"{{ .InvolvedObject.Name }}\"\n        reason: \"{{ .Reason }}\"\n        message: \"{{ .Message }}\"\n        type: \"{{ .Type }}\"\n        createdAt: \"{{ .GetTimestampISO8601 }}\"

\n

OpsCenter

\n

OpsCenter provides a central location
\nwhere operations engineers and IT professionals can view, investigate, and resolve operational work items (OpsItems)
\nrelated to AWS resources. OpsCenter is designed to reduce mean time to resolution for issues impacting AWS resources.
\nThis Systems Manager capability aggregates and standardizes OpsItems across services while providing contextual
\ninvestigation data about each OpsItem, related OpsItems, and related resources. OpsCenter also provides Systems Manager
\nAutomation documents (runbooks) that you can use to quickly resolve issues. You can specify searchable, custom data for
\neach OpsItem. You can also view automatically-generated summary reports about OpsItems by status and source.

\n

# ...\nreceivers:\n  - name: \"alerts\"\n    opscenter:\n    title: \"{{ .Message }}\",\n    category: \"{{ .Reason }}\", # Optional\n    description: \"Event {{ .Reason }} for {{ .InvolvedObject.Namespace }}/{{ .InvolvedObject.Name }} on K8s cluster\",\n    notifications: # Optional: SNS ARN\n      - \"sns1\"\n      - \"sns2\"\n  operationalData: # Optional\n    - Reason: \"\"{ { .Reason } }\"}\"\n  priority: \"6\", # Optional\n  region: \"us-east1\",\n  relatedOpsItems: # Optional: OpsItems ARN\n    - \"ops1\"\n    - \"ops2\"\n    severity: \"6\" # Optional\n    source: \"production\"\n  tags: # Optional\n    - ENV: \"{{ .InvolvedObject.Namespace }}\"

\n

Customizing Payload

\n

Some receivers allow customizing the payload. This can be useful to integrate it to external systems that require the
\ndata be in some format. It is designed to reduce the need for code writing. It allows mapping an event using Go
\ntemplates, with sprig library additions. It supports a recursive map definition, so that
\nyou can create virtually any kind of JSON to be pushed to a webhook, a Kinesis stream, SQS queue etc.

\n

# ...\nreceivers:\n  - name: pipe\n    kinesis:\n      region: us-west-2\n      streamName: event-pipeline\n      layout:\n        region: \"us-west-2\"\n        eventType: \"kubernetes-event\"\n        createdAt: \"{{ .GetTimestampMs }}\"\n        details:\n          message: \"{{ .Message }}\"\n          reason: \"{{ .Reason }}\"\n          type: \"{{ .Type }}\"\n          count: \"{{ .Count }}\"\n          kind: \"{{ .InvolvedObject.Kind }}\"\n          name: \"{{ .InvolvedObject.Name }}\"\n          namespace: \"{{ .Namespace }}\"\n          component: \"{{ .Source.Component }}\"\n          host: \"{{ .Source.Host }}\"\n          labels: \"{{ toJson .InvolvedObject.Labels}}\"

\n

Pubsub

\n

Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between independent
\napplications.

\n

receivers:\n  - name: \"pubsub\"\n    pubsub:\n      gcloud_project_id: \"my-project\"\n      topic: \"kube-event\"\n      create_topic: False

\n

Teams

\n

Microsoft Teams is your hub for teamwork in Office 365. All your team conversations, files, meetings, and apps live
\ntogether in a single shared workspace, and you can take it with you on your favorite mobile device.

\n

# ...\nreceivers:\n  - name: \"ms_teams\"\n    teams:\n      endpoint: \"https://outlook.office.com/webhook/...\"\n      layout: # Optional

\n

Syslog

\n

Syslog sink support enables to write k8s-events to syslog daemon server over tcp/udp. This can also be consumed by
\nrsyslog.

\n

# ...\nreceivers:\n  - name: \"syslog\"\n    syslog:\n      network: \"tcp\"\n      address: \"127.0.0.1:11514\"\n      tag: \"k8s.event\"\n

\n

BigQuery

\n

Google's query thing

\n

receivers:\n  - name: \"my-big-query\"\n    bigquery:\n      location:\n      project:\n      dataset:\n      table:\n      credentials_path:\n      batch_size:\n      max_retries:\n      interval_seconds:\n      timeout_seconds:

\n

Pipe

\n

pipe output directly into some file descriptor

\n

receivers:\n  - name: \"my_pipe\"\n    pipe:\n      path: \"/dev/stdout\"\n      deDot: true|false

\n

AWS EventBridge

\n

receivers:\n - name: \"eventbridge\"\n   eventbridge:\n     detailType: \"deployment\"\n     source: \"cd\"\n     eventBusName: \"default\"\n     region: \"ap-southeast-1\"\n     details:\n       message: \"{{ .Message }}\"\n       namespace: \"{{ .Namespace }}\"\n       reason: \"{{ .Reason }}\"\n       object: \"{{ .Namespace }}\"\n

\n

Loki

\n

receivers:\n  - name: \"loki\"\n    loki:\n      headers: # optional\n        X-Scope-OrgID: tennantID\n      streamLabels:\n        foo: bar\n      url: http://127.0.0.1:3100/loki/api/v1/push

","updatedAt":"2025-05-21T16:08:48Z","upvoteCount":null,"author":{"login":"eryajf","avatarUrl":"https://avatars.githubusercontent.com/u/33259379?u=e4a4090a38ac2473aaed4ef9945233636776c6c3&v=4","url":"https://github.com/eryajf"},"category":null,"labels":{"edges":[{"node":{"name":"更多","color":"25B472"}},{"node":{"name":"K8S-Tools","color":"2d411a"}},{"node":{"name":"resmoio","color":"90374f"}}]},"comments":{"edges":[]}}},"pageContext":{"number":329,"previous":{"title":"jonmosco/kube-ps1: Kubernetes Bash/Zsh提示符插件。","number":328},"next":{"title":"duyanghao/kubernetes-reading-notes: Kubernetes源码阅读笔记，专注云原生、边缘计算、程序员综合素质以及自我修养","number":330}}}, "staticQueryHashes": ["151096407","2861350382"]}