Crate数据库

Crate数据库 - 分片与分区和节点之间的关系(Crate database - relation between shards and partitions and nodes)

我是crate数据库的新手，并试图理解分片，分区和节点之间的关系。

一个分片对应多少个分区？ 我可以配置将特定分区的数据作为特定分片的一部分发送吗？ 我可以配置将特定分片发送到特定节点吗？ 我可以将碎片从一个节点移动到crate集群中的另一个节点，因为它可以在elasticsearch中完成吗？ 我可以为每个碎片设置不同数量的副本吗？

Usecase是将最新数据保存在我的一些性能最佳的节点上，其中包含更多副本和旧数据，而不是那么好的硬件，只有0或只有1个副本。

I am new to crate database and trying to understand relation between shards, partitions and nodes.

How many partitions corresponds to one shard? Can I configure to send data of a particular partition to be part of particular shard? Can I configure to send a particular shard to particular node ? Can I move a shard from one node to another in crate cluster as it can be done in elasticsearch? Can I have different number of replicas per shard ?

Usecase is to keep the latest data on few of my best performing nodes with more replicas and older data on not so good hardware with 0 or just 1 replica.

最满意答案

碎片是Crate最小的“数据单元”，表格应该具有适当的数量（我不知道是一种精确的科学），以便在整个群集中均匀分布数据（以及工作负载范围），因为这是在碎片。 目前，没有直接控制放置（在哪个节点上）和特定分片的复制。

一个分片对应多少个分区？

实际上它是另一种方式：分区有一些分片，因为分区被视为“子表”，其中包含数据子集。 从原始CREATE TABLE语句创建分区（它用作模板），因此甚至可以具有与其他分区不同的分片计数。

我可以配置将特定分区的数据作为特定分片的一部分发送吗？

好吧，不 - 不明确。 分片管理在后台通过魔术算法处理:)控制行所在的分区就像更新分区列的值一样简单。

我可以配置将特定分片发送到特定节点吗？

不是。配置中有一些旋钮可以控制节点上的分片数量： https ： //crate.io/docs/reference/configuration.html#allocation但是除非你确切知道什么，否则不建议更改这些设置你在做;）

我可以将碎片从一个节点移动到crate集群中的另一个节点，因为它可以在elasticsearch中完成吗？

不，不明确。

我可以为每个碎片设置不同数量的副本吗？

不，副本是每表设置，整个表将受到影响。

Usecase是将最新数据保存在我的一些性能最佳的节点上，其中包含更多副本和旧数据，而不是那么好的硬件，只有0或只有1个副本。

对于这个用例，我建议使用第二个表（您无法控制存储数据的机器），或者 - 如果您不需要查询数据 - 使用旧机器存储快照： https： //crate.io/a/backing-up-and-restoring-crate/并在需要时恢复它。

干杯，克劳斯

Shards are the smallest "unit of data" Crate has and a table should have an appropriate amount (not an exact science, I know) to distribute the data (and by extent the workload) evenly across the cluster, since this is done within those shards. Currently there is no direct control over placement (on which node) and replication of particular shards.

How many partitions corresponds to one shard?

Actually it's the other way around: A partition has a few shards, since a partition is treated like a "sub-table" with a subset of data in it. A partition is created from the original CREATE TABLE statement (it is used as a template) and can therefore even have a different shard count than other partitions.

Can I configure to send data of a particular partition to be part of particular shard?

Well, no - not explicitly. The shard management is handled in the background by a magic algorithm :) Controlling the partition a row resides in is as simple as updating the partition column's value.

Can I configure to send a particular shard to particular node ?

No. There are knobs in the configuration to control the # of shards on a node in general: https://crate.io/docs/reference/configuration.html#allocation but it's not recommended to change these setting unless you know exactly what you are doing ;)

Can I move a shard from one node to another in crate cluster as it can be done in elasticsearch?

No, not explicitly.

Can I have different number of replicas per shard ?

No, replicas are a per-table setting and the whole table will be affected.

Usecase is to keep the latest data on few of my best performing nodes with more replicas and older data on not so good hardware with 0 or just 1 replica.

For this use case I would recommend using either a second table (you cannot control on what machine the data is stored though), or - if you don't need to query the data - use your old machines to store the snapshots: https://crate.io/a/backing-up-and-restoring-crate/ and restore it when needed.

Cheers, Claus