feat(producer): rack-aware partitioner

[zornhsu]

Description

• Inspired by KIP-1123, this PR adds the support for rack-aware partitioning to all Sarama partitioners. (In contrast to KIP-1123 targets only Java BuiltInPartitioner)
• It comes with the new config Producer.PartitionerRackAware. Together with the existing RackID config, users are free to enable this new behavior.
• It also introduced -rackid to kafka-console-producer.go to facilitate manual testing.

Why

See KIP-1123 Motivation

Test Plan

Manually tested on 3 node cluster with 3 racks and 3 partitions

➜ ./kafka-console-producer -rackid use1-az2 -brokers test_broker:9092 -topic test_topic -value test_value
topic=test_topic partition=0     offset=31
➜ ./kafka-console-producer -rackid use1-az4 -brokers test_broker:9092 -topic test_topic -value test_value
topic=test_topic partition=1     offset=41
➜ ./kafka-console-producer -rackid use1-az6 -brokers test_broker:9092 -topic test_topic -value test_value
topic=test_topic partition=2     offset=51

rackid mismatch would fall back to random partitioning

➜ ./kafka-console-producer -rackid use1-azX -brokers test_broker:9092 -topic test_topic -value test_value
topic=test_topic partition=1     offset=42
➜ ./kafka-console-producer -rackid use1-azX -brokers test_broker:9092 -topic test_topic -value test_value
topic=test_topic partition=0     offset=32

Backward compatible with no rackid to random partitioning

➜ ./kafka-console-producer                  -brokers test_broker:9092 -topic test_topic -value test_value
topic=test_topic partition=0     offset=33
➜ ./kafka-console-producer                  -brokers test_broker:9092 -topic test_topic -value test_value
topic=test_topic partition=2     offset=52

[puellanivis]

I might additionally reorder some things to:

clientRack := client::RackID

var partitionsInRack []int32
for _, p := range partitions {
	…
}
if len(partitionsInRack) > 0 {
	…
}

This would put the clientRack declaration distinctly at the top of the code block, so its loop invariance is more pronounced. As well, all the code dealing with the partitionInRack is grouped more tightly together.

[puellanivis]

🤔 Do we have any Java-side Kafka config value to inform this config name? Does Java’s Kafka even support this?

[zornhsu]

Yes, the Java version is partitioner.rack.aware, where the Java PR to support this is pending review.

[puellanivis]

👍

[zornhsu]

Hello @dnwe , would you mind taking a look at this PR when you are available?

[dnwe]

@zornhsu don't worry, we will get to it in time. I had partially been waiting to see what feedback the upstream apache/kafka PR received, but that has still not yet had any review

[zornhsu]

Thank you for the feedback, @dnwe! To help me plan my work, when do you think you would be able to review this PR?

To enhance Sarama feature offerings, I would propose that we independently review and merge the PR in the absence of upstream reviews. The primary dependency appears to be the config name. If the final Java config name is changed, I am willing to submit a follow-up pull request to deprecate the current config and align it with the Java version config.

[dnwe]

@ijuma (seeing as I have you nearby) do you have any thoughts on this? The KIP and this PR both some like they provide a reasonable feature and the motivation is valid, but this is an unusual situation where upstream haven’t yet had much discussion nor decided on it, but we have a proposal to add the capability to Sarama – I’m not sure what’s the best course of action here

[ijuma]

@dnwe Since this is purely a client-side KIP, you have the option of not waiting for the upstream KIP to land. That said, there is a risk for the upstream KIP to evolve and then you may end up with different behavior. For example, I just commented on the KIP thread asking why we need two configs (which is different from KIP-392 for the consumer).

[github-actions]

Thank you for your contribution! However, this pull request has not had any activity in the past 90 days and will be closed in 30 days if no updates occur.
If you believe the changes are still valid then please verify your branch has no conflicts with main and rebase if needed. If you are awaiting a (re-)review then please let us know.

[zornhsu]

Pending @~showuon review the Java version of this PR