Skip to content

Conversation

@Shell32-Natsu
Copy link

Summary

This PR adds a connection_ttl_secs configuration option to the vector sink that allows connections to be periodically closed and re-established. This addresses load balancing issues where long-lived HTTP/2 connections prevent proper distribution of traffic across backend instances.

When connection_ttl_secs is set, the sink will track the age of each connection and automatically recreate the underlying HTTP/2 client after the specified duration. This forces a reconnection through the load balancer, allowing traffic to be distributed to different backend instances.

Vector configuration

[sinks.my_vector_sink]
type = "vector"
address = "http://load-balancer:6000"
compression = true
connection_ttl_secs = 300  # Reconnect every 5 minutes

Without the option (default behavior - unlimited connection reuse):

[sinks.my_vector_sink]
type = "vector"
address = "http://load-balancer:6000"

How did you test this PR?

  1. Unit tests: Added test_connection_ttl_config to verify configuration parsing with and without the TTL option
  2. Build verification: Confirmed the implementation compiles successfully with cargo build --package vector --lib --features "sinks-vector"
  3. Test suite: All existing vector sink tests pass

The implementation tracks connection creation time and checks before each request whether the TTL has been exceeded. When expired, it recreates the hyper HTTP/2 client and gRPC client transparently.

Change Type

[ ] Bug fix
[x] New feature
[ ] Non-functional (chore, refactoring, docs)
[ ] Performance

Is this a breaking change?

[ ]Yes
[x] No

Does this PR include user facing changes?

[x] Yes. Please add a changelog fragment based on our guidelines.
[ ] No. A maintainer will apply the no-changelog label to this PR.

References

This addresses the common issue where HTTP/2 connection multiplexing and keep-alive prevent load balancers from distributing traffic across multiple backend instances, as HTTP/2 connections are designed to be long-lived and reused.

@Shell32-Natsu Shell32-Natsu requested a review from a team as a code owner October 21, 2025 01:38
@github-actions github-actions bot added the domain: sinks Anything related to the Vector's sinks label Oct 21, 2025
@pront pront added the sink: vector Anything `vector` sink related label Oct 21, 2025
@pront pront self-assigned this Oct 21, 2025
@pront
Copy link
Member

pront commented Oct 21, 2025

Hi @Shell32-Natsu, can you add a changelog fragment since this is a user facing change? Thanks!

@Shell32-Natsu
Copy link
Author

Will do!

@Shell32-Natsu
Copy link
Author

Added

@Shell32-Natsu
Copy link
Author

Hi @pront Could you take a look? Thanks!

#[configurable(metadata(docs::examples = 300))]
#[configurable(metadata(docs::examples = 600))]
#[serde(default)]
pub(in crate::sinks::vector) connection_ttl_secs: Option<u64>,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pub(in crate::sinks::vector) connection_ttl_secs: Option<u64>,
pub connection_ttl_secs: Option<u64>,

Copy link
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question for my understanding: What happens to active requests if TTL expires?

}
}

fn check_and_recreate_client(&mut self) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This silently ignores errors when rebuilding the proxy connector. If client recreation fails, it continues with the old client without logging or alerting.

Also, worth considering emitting internal metrics tracking recreations.

}

impl std::fmt::Debug for VectorService {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can at least add self.client.created_at here but also can we just derive debug here? Is it too verbose?

@thomasqueirozb thomasqueirozb added the meta: awaiting author Pull requests that are awaiting their author. label Oct 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: sinks Anything related to the Vector's sinks meta: awaiting author Pull requests that are awaiting their author. sink: vector Anything `vector` sink related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants