Multizone rack awareness

[solsson]

Important when you have zones > 1 and kafka replicas > zones.

• http://kafka.apache.org/documentation/#basic_ops_racks
• https://kubernetes.io/docs/admin/multiple-zones/#nodes-are-labeled
• https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#interlude-built-in-node-labels
• https://stackoverflow.com/questions/36690446/inject-node-labels-into-kubernetes-pod
  • https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#capabilities-of-the-downward-api
  • Add spec.nodeName and spec.serviceAccountName to downward env var kubernetes/kubernetes#27880 (comment)

[solsson]

I think it's time to introduce an init container (or "sidecar") that replaces the ever growing command: in the kafka StatefulSet yaml. Node label lookup won't be as easy as --override broker.id=${HOSTNAME##*-}.

[solsson]

Explains why node name but no other node attributes are available via downward API, and thus why we need API calls: kubernetes/kubernetes#25957

[solsson]

I picked the zone lookup kubectl command from #56 but avoided the RBAC. The RBAC part needs more scrutiny so I'll add it as an addon branch.

[solsson]

Verified on GKE. Some error cases verified on minikube (which naturally lacks the label).

https://github.com/wurstmeister/kafka-docker#broker-rack has a strategy for AWS but I'm assuming because of https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#interlude-built-in-node-labels that the label is set there too.

[solsson]

Got RBAC policy from 13520a6