什么决定了Kafka消费者偏移量?

本文介绍了什么决定了Kafka消费者偏移量?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

限时送ChatGPT账号..

我对 Kafka 比较陌生.我已经对它进行了一些试验，但是关于消费者抵消，我还不清楚一些事情.据我目前所知，当消费者启动时，它将开始读取的偏移量由配置设置 auto.offset.reset 决定(如果我错了，请纠正我).

现在假设主题中有 10 条消息(偏移量 0 到 9)，并且消费者碰巧在它关闭之前(或在我杀死消费者之前)消费了其中的 5 条消息.然后说我重新启动该消费者进程.我的问题是:

如果 auto.offset.reset 设置为 earliest，它是否总是从偏移量 0 开始消费?

如果 auto.offset.reset 设置为 latest，它会从偏移量 5 开始消费吗?

关于这种场景的行为总是确定性的吗?

如果我的问题中有任何不清楚的地方，请随时发表评论.

解决方案

它比你描述的要复杂一些.
auto.offset.reset config 仅当您的消费者组没有在某处提交有效偏移量时才会启动(现在支持的 2 个偏移存储是 Kafka 和 Zookeeper)，这也取决于您使用的消费者类型.>

如果您使用高级 Java 使用者，请想象以下场景:

您在消费者组 group1 中有一个消费者，该消费者已消费 5 条消息并死亡.下次启动这个消费者时，它甚至不会使用那个 auto.offset.reset 配置，而是从它死的地方继续，因为它只会从偏移存储(Kafka 或 ZK)中获取存储的偏移正如我所提到的).

您在一个主题中有消息(如您​​所描述的)，并且您在一个新的消费者组 group2 中启动了一个消费者.任何地方都没有存储偏移量，这次 auto.offset.reset 配置将决定是从主题的开头(earliest)还是从主题的结尾开始主题(最新)

影响与earliest 和latest 配置相对应的偏移值的另一件事是日志保留策略.假设您有一个保留配置为 1 小时的主题.您发布了 5 条消息，然后一小时后又发布了 5 条消息.latest 偏移量仍将与前一个示例中的相同，但 earliest 将不能为 0，因为 Kafka 已经删除了这些消息以及最早的可用偏移量将是 5.

上面提到的所有内容都与 SimpleConsumer 无关，每次运行它时，它都会决定从哪里开始使用 auto.offset.reset 配置.>

如果你使用0.9之前的Kafka版本，你必须用smallest,largest替换earliest,latest>.

I am relatively new to Kafka. I have done a bit of experimenting with it, but a few things are unclear to me regarding consumer offset. From what I have understood so far, when a consumer starts, the offset it will start reading from is determined by the configuration setting auto.offset.reset (correct me if I am wrong).

Now say for example that there are 10 messages (offsets 0 to 9) in the topic, and a consumer happened to consume 5 of them before it went down (or before I killed the consumer). Then say I restart that consumer process. My questions are:

If the auto.offset.reset is set to earliest, is it always going to start consuming from offset 0 ?

If the auto.offset.reset is set to latest, is it going to start consuming from offset 5 ?

Is the behaviour regarding this kind of scenario always deterministic ?

Please don't hesitate to comment if anything in my question is unclear.

解决方案

It is a bit more complex than you described.
The auto.offset.reset config kicks in ONLY if your consumer group does not have a valid offset committed somewhere (2 supported offset storages now are Kafka and Zookeeper), and it also depends on what sort of consumer you use.

If you use a high-level java consumer then imagine following scenarios:

You have a consumer in a consumer group group1 that has consumed 5 messages and died. Next time you start this consumer it won't even use that auto.offset.reset config and will continue from the place it died because it will just fetch the stored offset from the offset storage (Kafka or ZK as I mentioned).

You have messages in a topic (like you described) and you start a consumer in a new consumer group group2. There is no offset stored anywhere and this time the auto.offset.reset config will decide whether to start from the beginning of the topic (earliest) or from the end of the topic (latest)

One more thing that affects what offset value will correspond to earliest and latest configs is log retention policy. Imagine you have a topic with retention configured to 1 hour. You produce 5 messages, and then an hour later you post 5 more messages. The latest offset will still remain the same as in previous example but the earliest one won't be able to be 0 because Kafka will already remove these messages and thus the earliest available offset will be 5.

Everything mentioned above is not related to SimpleConsumer and every time you run it, it will decide where to start from using the auto.offset.reset config.

If you use Kafka version older than 0.9, you have to replace earliest, latest with smallest,largest.

这篇关于什么决定了Kafka消费者偏移量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！