-
Notifications
You must be signed in to change notification settings - Fork 1.9k
feat(vector sink): Add connection_ttl_secs config for load balancing #24034
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
Hi @Shell32-Natsu, can you add a changelog fragment since this is a user facing change? Thanks! |
|
Will do! |
|
Added |
|
Hi @pront Could you take a look? Thanks! |
| #[configurable(metadata(docs::examples = 300))] | ||
| #[configurable(metadata(docs::examples = 600))] | ||
| #[serde(default)] | ||
| pub(in crate::sinks::vector) connection_ttl_secs: Option<u64>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| pub(in crate::sinks::vector) connection_ttl_secs: Option<u64>, | |
| pub connection_ttl_secs: Option<u64>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One question for my understanding: What happens to active requests if TTL expires?
| } | ||
| } | ||
|
|
||
| fn check_and_recreate_client(&mut self) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This silently ignores errors when rebuilding the proxy connector. If client recreation fails, it continues with the old client without logging or alerting.
Also, worth considering emitting internal metrics tracking recreations.
| } | ||
|
|
||
| impl std::fmt::Debug for VectorService { | ||
| fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can at least add self.client.created_at here but also can we just derive debug here? Is it too verbose?
Summary
This PR adds a
connection_ttl_secsconfiguration option to thevectorsink that allows connections to be periodically closed and re-established. This addresses load balancing issues where long-lived HTTP/2 connections prevent proper distribution of traffic across backend instances.When
connection_ttl_secsis set, the sink will track the age of each connection and automatically recreate the underlying HTTP/2 client after the specified duration. This forces a reconnection through the load balancer, allowing traffic to be distributed to different backend instances.Vector configuration
Without the option (default behavior - unlimited connection reuse):
How did you test this PR?
The implementation tracks connection creation time and checks before each request whether the TTL has been exceeded. When expired, it recreates the hyper HTTP/2 client and gRPC client transparently.
Change Type
[ ] Bug fix
[x] New feature
[ ] Non-functional (chore, refactoring, docs)
[ ] Performance
Is this a breaking change?
[ ]Yes
[x] No
Does this PR include user facing changes?
[x] Yes. Please add a changelog fragment based on our guidelines.
[ ] No. A maintainer will apply the no-changelog label to this PR.
References
This addresses the common issue where HTTP/2 connection multiplexing and keep-alive prevent load balancers from distributing traffic across multiple backend instances, as HTTP/2 connections are designed to be long-lived and reused.