是否应该在消息类型上拆分事件中心?

本文介绍了是否应该在消息类型上拆分事件中心?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我正在考虑将Azure事件中心用于当前正在处理的项目.今天，我们将服务总线队列用于命令，在这里，每种消息类型使用一个队列.

拥有多个事件中心是否有意义，还是对多个消息类型使用一个中心更好?

解决方案

这是一个需要权衡的问题，需要对您现在和将来要构建的系统以及它们如何使用不同的事件类型进行判断./p>

以下摘录自一些指南 Jay Kreps已经给出了在Apache Kafka之上设计系统的建议，该系统也同样适用于Event Hubs(主要例外是保留期短和对消费者群体数量的限制).

让我们从纯事件数据开始-这些活动是在公司内部进行的.在网络公司中，这些可能是点击，印象和各种用户操作.联邦快递可能会提供包裹运送，包裹提取，司机位置，通知，转移等.

这些事件类型可以用每种动作类型的单个逻辑流表示.为简单起见，我建议将Avro模式和主题命名为同一事物，例如PageViewEvent.如果事件具有自然主键，则可以使用该主键对Kafka中的数据进行分区，否则Kafka客户端将自动为您加载平衡数据.

...

我们在不同时间进行了实验，将多个事件混合在一个主题中，发现这通常会导致不必要的复杂性.取而代之的是，为每个事件分配自己的主题，消费者可以随时订阅多个这样的主题，以在需要时获得混合供稿.

我通常同意此建议(如果您要在Event Hubs/Kafka/Kinesis上设计系统，则应该阅读整个博客文章).需要忽略他们不感​​兴趣的消息的订阅者不仅烦人，而且如果其中一种事件类型开始主导组合流，这将成为问题.

但是拥有多个流并将它们组合在一起确实会产生成本，并且在做出决策时需要权衡它们.我列出了一些我想到的东西.

除非您花费精力将其添加回去，否则您将失去来自同一来源的不同类型事件之间的排序.

如果您希望将不同主题的进度一起提交，则需要对其进行管理.

如果要在主题之间共享的主键上划分事件流，并且希望每个主题中的分区一起传播，则不能使用 EventProcessorHost ，因为分区最终可能会自动平衡为不同的分区程序.

每个分区只有一个线程的使用者最终将所需的线程数乘以主题数.除非您拥有无法共享的昂贵结构，否则可能不是问题.

在我自己的部署中，即使我们当前使用相同的代码来处理所有事件，我们仍将不同的事件中心用于不同的事件类型.这仅仅是因为我希望添加仅关心某些事件类型的新组件.我希望这会有所帮助，并且最糟糕的是，我告诉你去看一下Kafka的指南，因为该原理是相同的，而且使用时间更长.

I am considering using Azure event Hub for a project I am currently working on. We are using Service Bus Queues for commands today and here we are using one queue per messagetype.

Would it make sense to have several Event Hubs or is it better to use one hub for several message types?

解决方案

This is a question full of tradeoffs and exercising judgement about what systems you expect to build now and in the future and how they might use the different event types.

Below is an excerpt from some of the guidance Jay Kreps has given for designing systems on top of Apache Kafka which applies well to Event Hubs as well (with the major exception of the limitations imposed by short retention periods and limitations on number of consumer groups).

Let’s begin with pure event data—the activities taking place inside the company. In a web company these might be clicks, impression, and various user actions. FedEx might have package deliveries, package pick ups, driver positions, notifications, transfers and so on.

These type of events can be represented with a single logical stream per action type. For simplicity I recommend naming the Avro schema and the topic the same thing, e.g. PageViewEvent. If the event has a natural primary key you can use that to partition data in Kafka, otherwise the Kafka client will automatically load balance data for you.

...

We experimented at various times with mixing multiple events in a single topic and found this generally lead to undue complexity. Instead give each event it’s own topic and consumers can always subscribe to multiple such topics to get a mixed feed when they want that.

I generally agree with this advice (and you should read that entire blog post if you're designing a system on Event Hubs/Kafka/Kinesis). Subscribers needing to ignore messages they aren't interested in is not only annoying, it becomes problematic later if one of the event types starts to dominate the combined stream.

But having multiple streams and combining them together does have costs, and they need to be weighed in making a decision. I've listed some that come to mind.

You lose ordering between events of different type from the same source unless you spend the effort to add it back.

If you want to commit progress to the different topics together then you need to manage them.

If you are partitioning the event streams on a primary key shared between the topics and want the partitions in each topic to travel together, you can't use the high level clients like EventProcessorHost as partitions can end up autobalanced to different processes.

A consumer with one thread per partition ends up multiplying the needed number of threads by the number of topics. Probably not an issue unless you have expensive structures that can't be shared.

In my own deployment we use different event hubs for different event types even though we currently use the same code to process them all. This is simply because I expect to add new components that only care about certain event types. I hope this helps, and at worst I've told you to go look at the guidance for Kafka since the principle's the same and it's been around longer.