{ "componentChunkName": "component---src-templates-issues-tsx", "path": "/issues/356", "result": {"data":{"issuesJson":{"id":"f98970a1-677a-53ef-b133-c3fc4cc609a7","title":"loggie-io/loggie: 一个基于 Golang 的轻量级、高性能、云原生日志采集 Agent 和中转处理 Aggregator，支持多 Pipeline 和组件热插拔 ","number":356,"bodyHTML":"

\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Repos	loggie-io/loggie
Views	$\"views\"$
Stars	$\"stars\"$
Forks	$\"forks\"$
License	$\"license\"$
UpdatedAt	$\"last-commit\"$
CreatedAt	$\"create-at\"$

\n\n

\n

\n

\n

\n

$\"go.dev$
\n $\"CII$
\n $\"dockerHub\"$

\n

\n
English | 中文
\n

\n

Loggie is a lightweight, high-performance, cloud-native agent and aggregator based on Golang.

\n

Supports multiple pipeline and pluggable components, including data transfer, filtering, parsing, and alerting.
Uses native Kubernetes CRD for operation and management.
Offers a range of observability, reliability, and automation features suitable for production environments.

\n

Based on Loggie, we can build a cloud-native scalable log data platform.

\n

\n

Features

\n

Next-generation cloud-native log collection and transmission

\n

Building pipelines based on CRD

\n

Loggie includes LogConfig/ClusterLogConfig/Interceptor/Sink CRDs, allowing for the creation of data collection, transfer, processing, and sending pipelines through simple YAML file creation.

\n

eg:

\n

apiVersion: loggie.io/v1beta1\nkind: LogConfig\nmetadata:\n  name: tomcat\n  namespace: default\nspec:\n  selector:\n    type: pod\n    labelSelector:\n      app: tomcat\n  pipeline:\n    sources: |\n      - type: file\n        name: common\n        paths:\n          - stdout\n          - /usr/local/tomcat/logs/*.log\n    sinkRef: default\n    interceptorRef: default

\n

$\"crd-usage\"$

\n

Multiple architectures

\n

\n
Agent: Deployed via DaemonSet, Loggie can collect log files without the need for containers to mount volumes.
\n
\n
Sidecar: Supports non-intrusive auto-injection of Loggie sidecars, without the need to manually add them to the Deployment/StatefulSet templates.
\n
\n
Aggregator: Supports deployment as an independent intermediate machine, which can receive aggregated data sent by Loggie Agent and can also be used to consume and process various data sources.
\n

\n

But regardless of the deployment architecture, Loggie still maintains a simple and intuitive internal design.

\n

\n

High Performance

\n

Benchmark

\n

Configure Filebeat and Loggie to collect logs, and send them to a Kafka topic without using client compression, with the Kafka topic partition configured as 3.

\n

With sufficient resources for the Agent specification, modify the number of files collected, the concurrency of the sending client (configure Filebeat worker and Loggie parallelism), and observe their respective CPU, memory, and pod network card transmission rates.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Agent	File Size	File Count	Sink Concurrency	CPU	MEM (rss)	Transmission Rates
Filebeat	3.2G	1	3	7.5~8.5c	63.8MiB	75.9MiB/s
Filebeat	3.2G	1	8	10c	65MiB	70MiB/s
Filebeat	3.2G	10	8	11c	65MiB	80MiB/s

Loggie	3.2G	1	3	2.1c	60MiB	120MiB/s
Loggie	3.2G	1	8	2.4c	68.7MiB	120MiB/s
Loggie	3.2G	10	8	3.5c	70MiB	210MiB/s

\n

Adaptive Sink Concurrency

\n

With sink concurrency configuration enabled, Loggie can:

\n

Automatically adjust the downstream data sending parallelism based on the actual downstream data response, making full use of the downstream server's performance without affecting its performance.
Adjust the downstream data sending speed appropriately when upstream data collection is blocked to relieve upstream blocking.

\n

Lightweight Streaming Data Analysis and Monitoring

\n

Logs are a universal data type and are not related to platforms or systems. How to better utilize this data is the core capability that Loggie focuses on and develops.

\n

\n

Real-time parsing and transformation

\n

With the configuration of transformer interceptors and the configuration of functional actions, Loggie can achieve:

\n

Parsing of various data formats (json, grok, regex, split, etc.)
Conversion of various fields (add, copy, move, set, del, fmt, etc.)
Support for conditional judgment and processing logic (if, else, return, dropEvent, ignoreError, etc.)

\n

eg:

\n

interceptors:\n  - type: transformer\n    actions:\n      - action: regex(body)\n        pattern: (?<ip>\\S+) (?<id>\\S+) (?<u>\\S+) (?<time>\\[.*?\\]) (?<url>\\\".*?\\\") (?<status>\\S+) (?<size>\\S+)\n      - if: equal(status, 404)\n        then:\n          - action: add(topic, not_found)\n          - action: return()\n      - if: equal(status, 500)\n        then:\n          - action: dropEvent()

\n

Detection, recognition, and alerting

\n

Helps you quickly detect potential problems and anomalies in the data and issue timely alerts. Support custom webhooks to connect to various alert channels.

\n

Supports matching methods such as:

\n

No data: no log data generated within the configured time period.
Fuzzy matching
Regular expression matching
Conditional judgment\n
- Field comparison: equal/less/greater...
\n

\n

Log data aggregation and monitoring

\n

Often, metric data is not only exposed through prometheus exporters, but log data itself can also provide a source of metrics. For example, by counting the access logs of a gateway, you can calculate the number of 5xx or 4xx status codes within a certain time interval, aggregate the qps of a certain interface, and calculate the total amount of body data, etc.

\n

eg:

\n

- type: aggregator\n  interval: 1m\n  select:\n    # operator：COUNT/COUNT-DISTINCT/SUM/AVG/MAX/MIN\n    - {key: amount, operator: SUM, as: amount_total}\n    - {key: quantity, operator: SUM, as: qty_total}\n  groupBy: [\"city\"]\n  calculate:\n    - {expression: \" ${amount_total} / ${qty_total} \", as: avg_amount}

\n

Observability and fast troubleshooting

\n

\n
Loggie provides configurable and rich metrics, and dashboards that can be imported into Grafana with one click.
\n\n
\n
Quickly troubleshoot Loggie itself and any problems in data transmission by Loggie terminal.
\n\n

\n

FAQs

\n

Loggie vs Filebeat/Fluentd/Logstash/Flume

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

	Loggie	Filebeat	Fluentd	Logstash	Flume
Language	Golang	Golang	Ruby	JRuby	Java
Multiple Pipelines	✓	single queue	single queue	✓	✓
Multiple output	✓	one output	copy	✓	✓
Aggregator	✓	✓	✓	✓	✓
Log Alarm	✓
Kubernetes container log collection	support container stdout and logs files in container	stdout	stdout
Configuration delivery	through CRD	manual	manual	manual	manual
Monitoring	support Prometheus metrics，and can be configured to output indicator log files separately, sending metrics, etc.		prometheus metrics	need exporter	need exporter
Resource Usage	low	low	average	high	high

\n

Documentation

\n

Setup

\n

Quickstart
Installation (Kubernetes, Node)

\n

User Guide

\n

\n

Reference

\n

Args
System
Pipelines\n
- Source: file, kafka, kubeEvent, grpc, elasticsearch, prometheusExporter..
- Sink: elasticsearch, kafka, grpc, loki, zinc, alertWebhook, dev..
- Interceptor: transformer, schema, limit, logAlert, maxbytes..
\n
CRD: LogConfig, ClusterLogConfig, Sink, Interceptor

\n

RoadMap

\n

\n

License

\n

\n

Contributions

\n

Pull requests, comments and suggestions are welcome.

\n

Please see CONTRIBUTING.md for more information.

","updatedAt":"2025-05-21T16:10:34Z","upvoteCount":null,"author":{"login":"eryajf","avatarUrl":"https://avatars.githubusercontent.com/u/33259379?u=e4a4090a38ac2473aaed4ef9945233636776c6c3&v=4","url":"https://github.com/eryajf"},"category":null,"labels":{"edges":[{"node":{"name":"更多","color":"25B472"}},{"node":{"name":"数据管道","color":"fa3097"}},{"node":{"name":"loggie-io","color":"5b40ba"}}]},"comments":{"edges":[]}}},"pageContext":{"number":356,"previous":{"title":"localsend/localsend: 一个免费、开源的应用程序，允许你在本地网络上安全地与附近设备共享文件和消息，无需互联网连接","number":355},"next":{"title":"logto-io/logto: 专为现代应用程序和 SaaS 产品设计的 Auth0 替代方案。它提供无缝的开发人员体验，非常适合个人和成长型公司","number":357}}}, "staticQueryHashes": ["151096407","2861350382"]}