[server] Coordinator Server Supports High-Available

[zcoo]

Purpose

Coordinator Server Supports High-Available.

Linked issue: close #188

Brief change log

Tests

see : com.alibaba.fluss.server.coordinator.CoordinatorServerElectionTest

API and Format

Documentation

Currently I make it by using an active (or leader) coordinator server and several standby (or alive) coordinator server. When several coordinator servers start up at the same time, only one can successfully preempt the ZK node and win the election and become active coordinator server leader. Once it failovers, the standby servers will take over it with an leader election process.

[zcoo]

not ready for review

[michaelkoepf]

@zcoo if it is not ready for review, you can click on Convert to draft in the right side bar, and, optionally, add [WIP] as prefix in the title of the PR.

[zcoo]

@michaelkoepf i get it, thank you

[zcoo]

Ready for review now! @wuchong @swuferhong

[LB-Yu]

I have a question here. Do we need to properly handle coordinatorEpoch and coordinatorEpochZkVersion in this PR, just like Kafka does? In my opinion:

• coordinatorEpoch prevents TabletServers from processing requests from zombie CoordinatorServers in the event of a split-brain scenario among CoordinatorServers.
• coordinatorEpochZkVersion(KAFKA-6082) prevents zombie CoordinatorServers from modifying the ZK state during a split-brain scenario.

Should we take these two concerns into consideration in the same way as Kafka?

[zcoo]

I have a question here. Do we need to properly handle coordinatorEpoch and coordinatorEpochZkVersion in this PR, just like Kafka does? In my opinion:

• coordinatorEpoch prevents TabletServers from processing requests from zombie CoordinatorServers in the event of a split-brain scenario among CoordinatorServers.
• coordinatorEpochZkVersion(KAFKA-6082) prevents zombie CoordinatorServers from modifying the ZK state during a split-brain scenario.

Should we take these two concerns into consideration in the same way as Kafka?

Thanks for reminder. I considered coordinatorEpoch and coordinatorEpochZkVersion , but ultimately did not implement them in this PR to simplify its complexity. I think we can handle it later.

Do you have any suggestions?
@LB-Yu @wuchong @swuferhong

[LB-Yu]

I have a question here. Do we need to properly handle coordinatorEpoch and coordinatorEpochZkVersion in this PR, just like Kafka does? In my opinion:

• coordinatorEpoch prevents TabletServers from processing requests from zombie CoordinatorServers in the event of a split-brain scenario among CoordinatorServers.
• coordinatorEpochZkVersion(KAFKA-6082) prevents zombie CoordinatorServers from modifying the ZK state during a split-brain scenario.

Should we take these two concerns into consideration in the same way as Kafka?

Thanks for reminder. I considered coordinatorEpoch and coordinatorEpochZkVersion , but ultimately did not implement them in this PR to simplify its complexity. I think we can handle it later.

Do you have any suggestions? @LB-Yu @wuchong @swuferhong

IMO, we can split them into different PRs, but they should be merged together? Otherwise, we might introduce other metadata inconsistency issues when introducing HA. What's your opinion?

[LB-Yu]

Why we don't use LeaderLatch here if we need to hold the leadership?

[zcoo]

Very good suggestion to me! It seems LeaderLatch is more suitable in this scene and I will try to use it instead