{ "componentChunkName": "component---src-templates-issues-tsx", "path": "/issues/159", "result": {"data":{"issuesJson":{"id":"9205cca0-0400-56be-af90-916c2e99ee2b","title":"medcl/esm: ES 索引迁移工具，支持跨版本迁移","number":159,"bodyHTML":"
\n



\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Reposmedcl/esm
Views\"views\"
Stars\"stars\"
Forks\"forks\"
License\"license\"
UpdatedAt\"last-commit\"
CreatedAt\"create-at\"
\n\n

\"\"

\n\n
\n

An Elasticsearch Migration Tool

\n

Elasticsearch cross version data migration.

\n

Links:

\n\n

Features:

\n\n

ESM is fast!

\n

A 3 nodes cluster(3 * c5d.4xlarge， 16C，32GB，10Gbps)

\n
root@ip-172-31-13-181:/tmp# ./esm -s https://localhost:8000 -d https://localhost:8000 -x logs1kw -y logs122 -m elastic:medcl123 -n elastic:medcl123 -w 40 --sliced_scroll_size=60 -b 5 --buffer_count=2000000  --regenerate_id\n[12-19 06:31:20] [INF] [main.go:506,main] start data migration..\nScroll 10064570 / 10064570 [=================================================] 100.00% 55s\nBulk 10062602 / 10064570 [==================================================]  99.98% 55s\n[12-19 06:32:15] [INF] [main.go:537,main] data migration finished.\n
\n

Migrated 10,000,000 documents within a minute, Nginx log generated from kibana_sample_data_logs.

\n

Before ESM

\n

Before running the esm, please manually prepare the target index with mapping and optimized settings to improve the speed, for example:

\n
PUT your-new-index\n{\n  \"settings\": {\n    \"index.translog.durability\": \"async\", \n    \"refresh_interval\": \"-1\", \n    \"number_of_shards\": 10,\n    \"number_of_replicas\": 0\n  }\n}\n
\n

Example:

\n

copy index index_name from 192.168.1.x to 192.168.1.y:9200

\n
./bin/esm  -s http://192.168.1.x:9200   -d http://192.168.1.y:9200 -x index_name  -w=5 -b=10 -c 10000\n
\n

copy index src_index from 192.168.1.x to 192.168.1.y:9200 and save with dest_index

\n
./bin/esm -s http://localhost:9200 -d http://localhost:9200 -x src_index -y dest_index -w=5 -b=100\n
\n

use sync feature for incremental update index src_index from 192.168.1.x to 192.168.1.y:9200

\n
./bin/esm --sync -s http://localhost:9200 -d http://localhost:9200 -x src_index -y dest_index\n
\n

support Basic-Auth

\n
./bin/esm -s http://localhost:9200 -x \"src_index\" -y \"dest_index\"  -d http://localhost:9201 -n admin:111111\n
\n

copy settings and override shard size

\n
./bin/esm -s http://localhost:9200 -x \"src_index\" -y \"dest_index\"  -d http://localhost:9201 -m admin:111111 -c 10000 --shards=50  --copy_settings\n\n
\n

copy settings and mapping, recreate target index, add query to source fetch, refresh after migration

\n
./bin/esm -s http://localhost:9200 -x \"src_index\" -q=query:phone -y \"dest_index\"  -d http://localhost:9201  -c 10000 --shards=5  --copy_settings --copy_mappings --force  --refresh\n\n
\n

dump elasticsearch documents into local file

\n
./bin/esm -s http://localhost:9200 -x \"src_index\"  -m admin:111111 -c 5000 -q=query:mixer  --refresh -o=dump.bin \n
\n

dump source and target index to local file and compare them, so can find the difference quickly

\n
./bin/esm --sort=_id -s http://localhost:9200 -x \"src_index\" --truncate_output --skip=_index -o=src.json\n./bin/esm --sort=_id -s http://localhost:9200 -x \"dst_index\" --truncate_output --skip=_index -o=dst.json\ndiff -W 200 -ry --suppress-common-lines src.json dst.json\n
\n

loading data from dump files, bulk insert to another es instance

\n
./bin/esm -d http://localhost:9200 -y \"dest_index\"   -n admin:111111 -c 5000 -b 5 --refresh -i=dump.bin\n
\n

support proxy

\n
 ./bin/esm -d http://123345.ap-northeast-1.aws.found.io:9200 -y \"dest_index\"   -n admin:111111  -c 5000 -b 1 --refresh  -i dump.bin  --dest_proxy=http://127.0.0.1:9743\n
\n

use sliced scroll(only available in elasticsearch v5) to speed scroll, and update shard number

\n
 ./bin/esm -s=http://192.168.3.206:9200 -d=http://localhost:9200 -n=elastic:changeme -f --copy_settings --copy_mappings -x=bestbuykaggle  --sliced_scroll_size=5 --shards=50 --refresh\n
\n

migrate 5.x to 6.x and unify all the types to doc

\n
./esm -s http://source_es:9200 -x \"source_index*\"  -u \"doc\" -w 10 -b 10 - -t \"10m\" -d https://target_es:9200 -m elastic:passwd -n elastic:passwd -c 5000 \n\n
\n

to migrate version 7.x and you may need to rename _type to _doc

\n
./esm -s http://localhost:9201 -x \"source\" -y \"target\"  -d https://localhost:9200 --rename=\"_type:type,age:myage\"  -u\"_doc\"\n\n
\n

filter migration with range query

\n
./esm -s https://192.168.3.98:9200 -m elastic:password -o json.out -x kibana_sample_data_ecommerce -q \"order_date:[2020-02-01T21:59:02+00:00 TO 2020-03-01T21:59:02+00:00]\"\n\n
\n

range query, keyword type and escape

\n
./esm -s https://192.168.3.98:9200 -m test:123 -o 1.txt -x test1  -q \"@timestamp.keyword:[\\\"2021-01-17 03:41:20\\\" TO \\\"2021-03-17 03:41:20\\\"]\"\n
\n

generate testing data, if input.json contains 10 documents, the follow command will ingest 100 documents, good for testing

\n
./bin/esm -i input.json -d  http://localhost:9201 -y target-index1  --regenerate_id  --repeat_times=10 \n
\n

select source fields

\n
 ./bin/esm -s http://localhost:9201 -x my_index -o dump.json --fields=author,title\n
\n

rename fields while do bulk indexing

\n
./bin/esm -i dump.json -d  http://localhost:9201 -y target-index41  --rename=title:newtitle\n
\n

user buffer_count to control memory used by ESM， and use gzip to compress network traffic

\n
./esm -s https://localhost:8000 -d https://localhost:8000 -x logs1kw -y logs122 -m elastic:medcl123 -n elastic:medcl123 --regenerate_id -w 20 --sliced_scroll_size=60 -b 5 --buffer_count=1000000 --compress false \n
\n

Download

\n

https://github.com/medcl/esm/releases

\n

Compile:

\n

if download version is not fill you environment,you may try to compile it yourself. go required.

\n

make build

\n\n

Options

\n
Usage:\n  esm [OPTIONS]\n\nApplication Options:\n  -s, --source=                    source elasticsearch instance, ie: http://localhost:9200\n  -q, --query=                     query against source elasticsearch instance, filter data before migrate, ie: name:medcl\n      --sort=                      sort field when scroll, ie: _id (default: _id)\n  -d, --dest=                      destination elasticsearch instance, ie: http://localhost:9201\n  -m, --source_auth=               basic auth of source elasticsearch instance, ie: user:pass\n  -n, --dest_auth=                 basic auth of target elasticsearch instance, ie: user:pass\n  -c, --count=                     number of documents at a time: ie \"size\" in the scroll request (10000)\n      --buffer_count=              number of buffered documents in memory (100000)\n  -w, --workers=                   concurrency number for bulk workers (1)\n  -b, --bulk_size=                 bulk size in MB (5)\n  -t, --time=                      scroll time (1m)\n      --sliced_scroll_size=        size of sliced scroll, to make it work, the size should be > 1 (1)\n  -f, --force                      delete destination index before copying\n  -a, --all                        copy indexes starting with . and _\n      --copy_settings              copy index settings from source\n      --copy_mappings              copy index mappings from source\n      --shards=                    set a number of shards on newly created indexes\n  -x, --src_indexes=               indexes name to copy,support regex and comma separated list (_all)\n  -y, --dest_index=                indexes name to save, allow only one indexname, original indexname will be used if not specified\n  -u, --type_override=             override type name\n      --green                      wait for both hosts cluster status to be green before dump. otherwise yellow is okay\n  -v, --log=                       setting log level,options:trace,debug,info,warn,error (INFO)\n  -o, --output_file=               output documents of source index into local file\n      --truncate_output=           truncate before dump to output file\n  -i, --input_file=                indexing from local dump file\n      --input_file_type=           the data type of input file, options: dump, json_line, json_array, log_line (dump)\n      --source_proxy=              set proxy to source http connections, ie: http://127.0.0.1:8080\n      --dest_proxy=                set proxy to target http connections, ie: http://127.0.0.1:8080\n      --refresh                    refresh after migration finished\n      --sync=                      sync will use scroll for both source and target index, compare the data and sync(index/update/delete)\n      --fields=                    filter source fields(white list), comma separated, ie: col1,col2,col3,...\n      --skip=                      skip source fields(black list), comma separated, ie: col1,col2,col3,...\n      --rename=                    rename source fields, comma separated, ie: _type:type, name:myname\n  -l, --logstash_endpoint=         target logstash tcp endpoint, ie: 127.0.0.1:5055\n      --secured_logstash_endpoint  target logstash tcp endpoint was secured by TLS\n      --repeat_times=              repeat the data from source N times to dest output, use align with parameter regenerate_id to amplify the data size\n  -r, --regenerate_id              regenerate id for documents, this will override the exist document id in data source\n      --compress                   use gzip to compress traffic\n  -p, --sleep=                     sleep N seconds after finished a bulk request (-1)\n\nHelp Options:\n  -h, --help                       Show this help message\n\n\n
\n

FAQ

\n\n
http.max_header_size: 16k\nhttp.max_initial_line_length: 8k\n
\n

Versions

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
FromTo
1.x1.x
1.x2.x
1.x5.x
1.x6.x
1.x7.x
2.x1.x
2.x2.x
2.x5.x
2.x6.x
2.x7.x
5.x1.x
5.x2.x
5.x5.x
5.x6.x
5.x7.x
6.x1.x
6.x2.x
6.x5.0
6.x6.x
6.x7.x
7.x1.x
7.x2.x
7.x5.x
7.x6.x
7.x7.x
","updatedAt":"2025-05-21T15:57:10Z","upvoteCount":null,"author":{"login":"eryajf","avatarUrl":"https://avatars.githubusercontent.com/u/33259379?u=e4a4090a38ac2473aaed4ef9945233636776c6c3&v=4","url":"https://github.com/eryajf"},"category":null,"labels":{"edges":[{"node":{"name":"更多","color":"25B472"}},{"node":{"name":"ElasticSearch-Manage","color":"6b7852"}},{"node":{"name":"medcl","color":"8c6f56"}}]},"comments":{"edges":[]}}},"pageContext":{"number":159,"previous":{"title":"Bronya0/ES-King: 一个现代、实用、轻量的ES GUI客户端，支持多平台。","number":158},"next":{"title":"gtamas/etcdmanager: Etcd 客户端，支持 Linux、Windows、MacOS","number":160}}}, "staticQueryHashes": ["151096407","2861350382"]}