如何将mysql数据迁移到ElasticSearch实时

本文介绍了如何将mysql数据迁移到ElasticSearch实时的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述 我有一个mysql数据库与几个表，我想将mysql数据迁移到ElasticSearch。通过批处理作业将整个数据库迁移到ES很容易。但是我应该如何从mysql实时更新ES。即如果在mysql中有更新操作，那么我应该在ES中执行相同的操作。 我研究了可以反映mysql中任何更改的mysql binLog。但是我必须将binLog解析为ES语法，我觉得这真的很痛苦。谢谢！ （与Solr相同的情况）

解决方案

有一个现有的项目需要您的binlog，转换和运送您可以通过以下方式查看： github/siddontang/go- mysql-elasticsearch

另一个是这样的： github/noplay/python-mysql-replication 。

然而，请注意，无论选择哪一个，在索引binlog之前预先创建索引和映射是一个很好的做法。这样可以更好地控制您的数据。

I have a mysql database with couple tables, I wanna migrate the mysql data to ElasticSearch. It's easy to migrate the whole database to ES via a batch job. But how should I update ES from mysql realtime. i.e if there was a update operation in mysql then I should do the same operation in ES. I researched mysql binLog which can reflect any changes from mysql. But I have to parse binLog to ES syntax, I think it's really painful. Thanks! (the same case with Solr)

解决方案

There is an existing project which takes your binlog, transforms it and ships it to Elasticsearch, You can check it out at: github/siddontang/go-mysql-elasticsearch

Another one would be this one: github/noplay/python-mysql-replication.

Note, however, that whichever you pick, it's a good practice to pre-create your index and mappings before indexing your binlog. That gives you more control over your data.