Kafka Cluster

本文介绍了Kafka Cluster-生产者的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我对卡夫卡有几个问题。如果有人可以通过回答其中一个来帮助我，我将非常感激。

I have several questions about Kafka. If someone can help me by responding to one of them, i will be very thankful.

在此先感谢您：）

Q1）我知道分区是在Kafka Broker中拆分的。但是拆分基于什么？例如，如果我有3个代理和6个分区，那么如何确保每个代理将有2个分区？ Qf）当前生产者发送新消息时，消息的默认格式是什么？该消息在Kafka中是如何产生的？

Q1) I know that partitions are split across Kafka Broker. But the split is based on what ?. For instance, if I have 3 brokers and 6 partitions, how to ensure that each broker will have 2 partitions ? How this split is currently made in Kafka ?

Q2） Avro格式？如何将这种默认格式更改为更适合的另一种格式？

Q2) When a producer send a new message, what id the default format of the message ? Avro format ? How can I change this default format to another format which may be more suitable for example ?

Q3）我知道可以配置文件（日志段）的最大大小在分区中，我必须更改以下配置：log.segment.bytes（默认为1G）。但是我必须更改哪个配置参数来增加/减少目录（即分区）的最大大小？

Q3) I know that to configure the maximum size of a file (log segment) within a partition, I have to change the following configuration : log.segment.bytes (1G by default). But which configuration parameter, I have to change to increase/decrease the maximum size of a directory (i.e a partition) ?

Q4）如果一个分区被认为是引导者，死了，跟随者分区之一将带头。选举新领导人要采取什么步骤？ （i.e）如何选举新的领导者？

Q4) If a partition consider as the leader is dead, one of the follower partition will take the lead. What is the step, to elect the new leader ? (i.e) How the election of a new leader is made of ?

Q5）什么是配置参数，可让我更改两次磁盘保留之间的时间？ （在磁盘上保留数据-顺序写入）

Q5) What is the configuration parameter, that allow me to change the time between 2 persist on disk ? (persist data on disk - sequential write)

Q6）如何将消息从硬盘上的Kafka经纪人头发送给Kafka使用者？ Kafka Broker和Zookeeper之间有什么相互作用？ 是Zookeeper将消息发送给消费者或Kafka经纪人吗？

Q6) How the message is sent from the hard disk Head of a Kafka broker to a Kafka consumer ? What is the interaction between Kafka Broker and Zookeeper ? Is it Zookeeper which send the message to the consumer or Kafka Broker ?

谢谢您，

推荐答案

Q1：请参见如何进行分区

Q2：经纪人不了解消息格式-他们将消息视为纯字节数组。因此，它可以处理您想要的任何消息格式。格式由您自己的代码确定-选择所需的内容，然后仅向生产者/消费者提供相应的解序列器。

Q2: Brokers are agnostic to the message format -- they treat messages a plain byte arrays. Thus, it can handle any message format you want to have. The format is determined in your own code -- choose whatever you want and just provide the corresponding de/serializer to the producer/consumer.

Q3：主题和分区是或者在经过可配置的保留时间（ log.retention.ms ）后被截断，或者如果它们超过了 log.retention.bytes 。此外，可以压缩主题以避免无限增长。 （请参阅 log.cleanup.policy ）

Q3: Topics and thus partitions are either truncated after a configurable retention time passed (log.retention.ms) or if they grow beyond log.retention.bytes. Furthermore, topics can be compacted to avoid infinite growth. (cf. log.cleanup.policy)

Q4：要进行领导人选举，使用Apache Zookeeper。

Q4: For leader election Apache Zookeeper is used.

第5季度：不明白这个问题。

Q5: Don't understand the question.

第6季度：ZK仅用于维护元数据（确实存在哪些主题）例如）。 ZK不参与客户端-代理通信的任何实际数据传输。 Kafka使用自己的网络协议。有关更多详细信息，请参见Kafka Wiki： cwiki.apache/confluence/ display / KAFKA / Index

Q6: ZK is only used to maintain metadata (which topics do exists for example). ZK is not involved in any actual data transfer of client-broker communication. Kafka uses its own network protocol. See the Kafka Wiki for more details: cwiki.apache/confluence/display/KAFKA/Index