{ "componentChunkName": "component---src-templates-issues-tsx", "path": "/issues/201", "result": {"data":{"issuesJson":{"id":"07d6bdf2-06b8-5740-ab07-0c99861d2ef6","title":"go-mysql-org/go-mysql-elasticsearch: 将 MySQL 数据自动同步到 ES 的工具。它首先用 mysqldump 获取原始数据，然后使用 binlog 增量同步数据","number":201,"bodyHTML":"
\n



\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Reposgo-mysql-org/go-mysql-elasticsearch
Views\"views\"
Stars\"stars\"
Forks\"forks\"
License\"license\"
UpdatedAt\"last-commit\"
CreatedAt\"create-at\"
\n\n

\"\"

\n\n
\n

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically.

\n

It uses mysqldump to fetch the origin data at first, then syncs data incrementally with binlog.

\n

Call for Committer/Maintainer

\n

Sorry that I have no enough time to maintain this project wholly, if you like this project and want to help me improve it continuously, please contact me through email (siddontang@gmail.com).

\n

Requirement: In the email, you should list somethings(including but not limited to below) to make me believe we can work together.

\n

Your GitHub ID
\nThe contributions to go-mysql-elasticsearch before, including PRs or Issues.
\nThe reason why you can improve go-mysql-elasticsearch.

\n

Install

\n\n

How to use?

\n\n

Notice

\n\n

Source

\n

In go-mysql-elasticsearch, you must decide which tables you want to sync into elasticsearch in the source config.

\n

The format in config file is below:

\n
[[source]]\nschema = \"test\"\ntables = [\"t1\", t2]\n\n[[source]]\nschema = \"test_1\"\ntables = [\"t3\", t4]\n
\n

schema is the database name, and tables includes the table need to be synced.

\n

If you want to sync all table in database, you can use asterisk(*).

\n
[[source]]\nschema = \"test\"\ntables = [\"*\"]\n\n# When using an asterisk, it is not allowed to sync multiple tables\n# tables = [\"*\", \"table\"]\n
\n

Rule

\n

By default, go-mysql-elasticsearch will use MySQL table name as the Elasticserach's index and type name, use MySQL table field name as the Elasticserach's field name.
\ne.g, if a table named blog, the default index and type in Elasticserach are both named blog, if the table field named title,
\nthe default field name is also named title.

\n

Notice: go-mysql-elasticsearch will use the lower-case name for the ES index and type. E.g, if your table named BLOG, the ES index and type are both named blog.

\n

Rule can let you change this name mapping. Rule format in config file is below:

\n
[[rule]]\nschema = \"test\"\ntable = \"t1\"\nindex = \"t\"\ntype = \"t\"\nparent = \"parent_id\"\nid = [\"id\"]\n\n    [rule.field]\n    mysql = \"title\"\n    elastic = \"my_title\"\n
\n

In the example above, we will use a new index and type both named \"t\" instead of default \"t1\", and use \"my_title\" instead of field name \"title\".

\n

Rule field types

\n

In order to map a mysql column on different elasticsearch types you can define the field type as follows:

\n
[[rule]]\nschema = \"test\"\ntable = \"t1\"\nindex = \"t\"\ntype = \"t\"\n\n    [rule.field]\n    // This will map column title to elastic search my_title\n    title=\"my_title\"\n\n    // This will map column title to elastic search my_title and use array type\n    title=\"my_title,list\"\n\n    // This will map column title to elastic search title and use array type\n    title=\",list\"\n\n    // If the created_time field type is \"int\", and you want to convert it to \"date\" type in es, you can do it as below\n    created_time=\",date\"\n
\n

Modifier \"list\" will translates a mysql string field like \"a,b,c\" on an elastic array type '{\"a\", \"b\", \"c\"}' this is specially useful if you need to use those fields on filtering on elasticsearch.

\n

Wildcard table

\n

go-mysql-elasticsearch only allows you determind which table to be synced, but sometimes, if you split a big table into multi sub tables, like 1024, table_0000, table_0001, ... table_1023, it is very hard to write rules for every table.

\n

go-mysql-elasticserach supports using wildcard table, e.g:

\n
[[source]]\nschema = \"test\"\ntables = [\"test_river_[0-9]{4}\"]\n\n[[rule]]\nschema = \"test\"\ntable = \"test_river_[0-9]{4}\"\nindex = \"river\"\ntype = \"river\"\n
\n

\"test_river_[0-9]{4}\" is a wildcard table definition, which represents \"test_river_0000\" to \"test_river_9999\", at the same time, the table in the rule must be same as it.

\n

At the above example, if you have 1024 sub tables, all tables will be synced into Elasticsearch with index \"river\" and type \"river\".

\n

Parent-Child Relationship

\n

One-to-many join ( parent-child relationship in Elasticsearch ) is supported. Simply specify the field name for parent property.

\n
[[rule]]\nschema = \"test\"\ntable = \"t1\"\nindex = \"t\"\ntype = \"t\"\nparent = \"parent_id\"\n
\n

Note: you should setup relationship with creating the mapping manually.

\n

Filter fields

\n

You can use filter to sync specified fields, like:

\n
[[rule]]\nschema = \"test\"\ntable = \"tfilter\"\nindex = \"test\"\ntype = \"tfilter\"\n\n# Only sync following columns\nfilter = [\"id\", \"name\"]\n
\n

In the above example, we will only sync MySQL table tfiler's columns id and name to Elasticsearch.

\n

Ignore table without a primary key

\n

When you sync table without a primary key, you can see below error message.

\n
schema.table must have a PK for a column\n
\n

You can ignore these tables in the configuration like:

\n
# Ignore table without a primary key\nskip_no_pk_table = true\n
\n

Elasticsearch Pipeline

\n

You can use Ingest Node Pipeline to pre-process documents before indexing, like JSON string decode, merge fileds and more.

\n
[[rule]]\nschema = \"test\"\ntable = \"t1\"\nindex = \"t\"\ntype = \"_doc\"\n\n# pipeline id\npipeline = \"my-pipeline-id\"\n
\n

Node: you should create pipeline manually and Elasticsearch >= 5.0.

\n

Why not other rivers?

\n

Although there are some other MySQL rivers for Elasticsearch, like elasticsearch-river-jdbc, elasticsearch-river-mysql, I still want to build a new one with Go, why?

\n\n

Todo

\n\n

Donate

\n

If you like the project and want to buy me a cola, you can through:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
PayPal微信
\"\"[\"\"
\n

Feedback

\n

go-mysql-elasticsearch is still in development, and we will try to use it in production later. Any feedback is very welcome.

\n

Email: siddontang@gmail.com

","updatedAt":"2025-05-21T16:00:03Z","upvoteCount":null,"author":{"login":"eryajf","avatarUrl":"https://avatars.githubusercontent.com/u/33259379?u=e4a4090a38ac2473aaed4ef9945233636776c6c3&v=4","url":"https://github.com/eryajf"},"category":null,"labels":{"edges":[{"node":{"name":"更多","color":"25B472"}},{"node":{"name":"数据管道","color":"fa3097"}},{"node":{"name":"go-mysql-org","color":"8a4930"}}]},"comments":{"edges":[]}}},"pageContext":{"number":201,"previous":{"title":"eryajf/go-ldap-admin: 🌉 基于 Golang+Vue 实现的 OpenLDAP 后台管理项目","number":200},"next":{"title":"GopeedLab/gopeed: 一款由Golang+Flutter开发的高速下载器，支持（HTTP、BitTorrent、Magnet）协议下载，并且支持全平台使用。除了基本的下载功能外，Gopeed 还是一款高度可定制化的下载器，支持通过对接APIs或者安装和开发扩展来实现更多的功能","number":202}}}, "staticQueryHashes": ["151096407","2861350382"]}