[Feature] CoordinatorServer Supports High-Available

[wuchong]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

The current implementation of the CoordinatorServer in Fluss lacks high availability support, which poses a risk to system stability and reliability. Achieving high availability is crucial for ensuring that the CoordinatorServer can handle failures gracefully and continue to provide uninterrupted service to clients.

Solution

We can deploy a standby CoordinatorServer alongside the primary one. In the event of a failure in the primary CoordinatorServer, the standby server will take over the control through a leader election process orchestrated by Zookeeper. Since the CoordinatorServer is stateless, this transition is seamless and straightforward.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[Vipamp]

hi, @wuchong i left some questions about this issue.

1. In this issue content，CoordinatorServer is stateless，but there are many Map objects in class CoordinatorContext，in my opinion， those maps need to be persistent and can be recovered while standby CoordinateServer becoming active，is it?
   2.In near future，zookeeper will be removed，so supporting HA by zookeeper is necessary？What is the plan for this.

[luoyuxia]

@Vipamp Thanks for your attention. I'd try to answer question 1. The CoordinatorContext can be restored from zookepper persistent node like /tabletservers/tables/[tableId]/ for assignment so that we won't need to store the CoordinatorContext.

[wuchong]

Hi @Vipamp , yes, currently CoordinatorServer is stateless, and all the Map objects can be restored from ZK (the metadata has been persisted in ZK). In the future, when ZK is removed, Fluss itself should persist the metadata using KvStore for metadata storage and Raft for cluster coordination and ensuring consistency.

[wang454]

I will try , can u assign to me? Thanks.

[wuchong]

@wang454 sure