Awarno/haproxy

[AWarno]

Enabling Multi-Instance Deployment with HAProxy

Why HAProxy

HAProxy is a lightweight, reliable, and widely used load balancer. It generalizes well to all server types. Using an external load balancer is officially recommended in the vLLM documentation (see vLLM Data Parallel Deployment); the documentation provides an example using NGINX, but HAProxy should work similarly.

Alternative Solutions

• Ray
  This is useful for multi-node deployments when a model is too large for a single node. It can also be used for multi-instance setups, but it requires knowing how to launch and manage each server type individually (vLLM, SGLang may have different CLI arguments for this). It does not generalize as well as using an external load balancer. However, we may want to provide an example of how to use it for multi-node large model deployment.

• LiteLLM
  Offers backend orchestration but is generally overkill for simple load balancing. The project evolves quickly, which may affect stability.

• NGINX
  Very similar to HAProxy for this use case and officially recommended in the vLLM documentation:
  vLLM Data Parallel Deployment
  HAProxy, however, is slightly simpler/nicer to use in practice (based on my experience).

Literature

• vLLM Data Parallel Deployment
• vLLM Multi-Node Serving
• vLLM Parallelism and Scaling
• SGLang Worker (Ray)

TODO

• Run on longer tasks to validate stability and performance. (I have checked ifeval so far)
• Check if the HAProxy template is correctly included in the pip wheel (consider renaming it)
• Documentation
• dataclass in types fix!!!!

Next Steps

• Add a multi-node deployment example using Ray server. This will likely just require creating one example configuration file under examples/.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[AWarno]

/ok to test 06e6a85