VPA: prune stale container aggregates, split recommendations over true number of containers #6745

jkyros · 2024-04-22T15:39:57Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

Previously we weren't cleaning up "stale" aggregates when container names changed (because of renames, removals) and that was resulting in:

VPAs showing recommendations for containers which no longer exist
Resources being split across containers which no longer exist (resulting in some containers ending up with resource limits too small for them to effectively live)
There was also a corner case where during a rollout after a container was renamed/removed from a deployment, we were counting the number of unique container names and not the actual number of containers in each pod, so we were splitting resources that shouldn't have been split.

This PR is an attempt to clean up those stale aggregates without incurring too much overhead, and make sure that the resources get spread across the correct number of containers during a rollout.

Which issue(s) this PR fixes:

Fixes #6744

Special notes for your reviewer:

There are probably a lot of different ways we can do the pruning of stale aggregates for missing containers:

I went with explicitly marking and sweeping them because it saved us an additional loop through all the pods and containers
We could also just as easily just have a PruneAggregates() that runs after LoadPods() that goes through everything and removes them (or do this work as part of LoadPods() but that seems...deceptive?)
We could probably also tweak the existing garbageCollectAggregateCollectionStates and run it immediately after LoadPods() every time but that might be expensive.

I'm not super-attached to any particular approach, I'd just like to fix this, so I can retool it if necessary.

If I am being ignorant, and there are corner cases I'm missing, absolutely let me know
it probably need some tests/cleanup and I'll change the names of things to...whatever you want them to be. 😄

Does this PR introduce a user-facing change?

Added pruning of container aggregates and changed container math so resources will no longer be split across the wrong number of containers

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2024-04-22T15:40:07Z

Hi @jkyros. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kwiesmueller · 2024-04-29T17:53:57Z

vertical-pod-autoscaler/pkg/recommender/input/cluster_feeder.go

+	// TODO(jkyros): This only removes the container state from the VPA's aggregate states, there
+	// is still a reference to them in feeder.clusterState.aggregateStateMap, and those get
+	// garbage collected eventually by the rate limited aggregate garbage collector later.
+	// Maybe we should clean those up here too since we know which ones are stale?


Is it a lot of extra work to do that? Do you see any risks doing it here?

No, I don't think it's a lot of extra work, it should be reasonably cheap to clean them up here since it's just deletions from the other maps if the keys exist, I just didn't know all the history.

It seemed possible at least that we were intentionally waiting to clean up the aggregates so if there was an unexpected hiccup we didn't just immediately blow away all that aggregate history we worked so hard to get? (Like maybe someone oopses, deletes their deployment, then puts it back? Right now we don't have to start over -- the pods come back in, find their container aggregates, and resume ? But if I clean them up here, we have to start over...)

Cleaning the map is cheap, so I don't mind handling it here. However, in cases like the one mentioned above (where someone deletes a deployment and recreates it immediately), I think it's better to leave this logic to be handled by gc.
@adrianmoisey, any thoughts on this?

Leaving it for the gc feels like the right option to me, for what it's worth

kwiesmueller · 2024-04-29T17:58:31Z

vertical-pod-autoscaler/pkg/recommender/model/cluster.go

+// the correct number and not just the number of aggregates that have *ever* been present. (We don't want minimum resources
+// to erroneously shrink, either)
+func (cluster *ClusterState) setVPAContainersPerPod(pod *PodState) {
+	for _, vpa := range cluster.Vpas {


I'm wondering if there is already a place where this logic could go so we don't have to loop over all VPAs for every pod again here.
In large clusters with a VPA to Pod ratio that's closer to 1 this could be a little wasteful.

Hmm, yeah, I struggled with finding a less expensive way without making too much of a mess. Unless I'm missing something (and I might be) we don't seem to have a VPA <--> Pod map -- probably because we didn't need one until now? At the very least I think I should gate this to only run if the number of containers in the pod is > 1.

Like, I think our options are:

update the VPA as the pods roll through (which requires me to find the VPA for each pod like I did here) or

count the containers as we load the VPAs (but we load the VPAs before we load the pods, so we'd have to go through the pods again, so that doesn't help us)

have the VPA actually track the pods it's managing, something like this: jkyros@6ddc208 (could also just be an array of PodIDs and we could look up the state so we could save the memory cost of the PodState pointer, but you know what I mean)

I put it where I did (option 1) because at least LoadPods() was already looping through all the pods so we could freeload off the "outer" pod loop and I figured we didn't want to spend the memory on option 3. If we'd entertain option 3 and are okay with the memory usage, I can totally do that?

I believe option 3 is the best approach. We can implement this in a follow-up PR.

k8s-triage-robot · 2024-07-31T20:11:33Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-08-30T20:12:28Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

sreber84 · 2024-09-20T09:24:07Z

/remove-lifecycle rotten

adrianmoisey · 2024-12-03T11:41:10Z

/ok-to-test
/assign

I want to see if I can help get this merged

maxcao13 · 2024-12-03T21:33:48Z

Hi everyone, so John has take a hiatus and has left me with this PR, so after catching up, I guess we are still waiting for those conversations to resolve on which way do we want to go with those design decisions. The two commits I just put up are just a improvement on the existing implementation (assuming we will go with that, we don't have to), and some e2e tests to prove this works.

maxcao13 · 2025-01-24T07:05:51Z

In order, the most recent commits are:

3d4f751 is a commit for e2e tests.
24c4eba is a commit which reworks a bit of how the pruning works, in that if an aggregate container still exists, but is considered stale, it will get pruned unless there is a container with the same name that still exists. The grace period of the combined aggregateByContainerName will follow the aggregate with the most recent updateTime. I think this makes more sense as it follows how the VPA usually works with contributive container names.
b8d78c6 is a commit with unit tests.

Previously we were dividing the resources per pod by the number of container aggregates, but in a situation where we're doing a rollout and the container names are changing (either a rename, or a removal) we're splitting resources across the wrong number of containers, resulting in smaller values than we should actually have. This collects a count of containers in the model when the pods are loaded, and uses the "high water mark value", so in the event we are doing something like adding a container during a rollout, we favor the pod that has the additional container. There are probably better ways to do this plumbing, but this was my initial attempt, and it does fix the issue.

Previously we were only cleaning checkpoints after something happened to the VPA or the targetRef, and so when a container got renamed the checkpoint would stick around forever. Since we're trying to clean up the aggregates immediately now, we need to force the checkpoint garbage collection to clean up any checkpoints that don't have matching aggregates. If the checkpoints did get loaded back in after a restart, PruneContainers() would take the aggregates back out, but we probably shouldn't leave the checkpoints out there. Signed-off-by: Max Cao <[email protected]>

Previously we were letting the rate limited garbage collector clean up the aggregate states, and that works really well in most cases, but when the list of containers in a pod changes, either due to the removal or rename of a container, the aggregates for the old containers stick around forever and cause problems. To get around this, this marks all existing aggregates/initial aggregates in the list for each VPA as "not under a VPA" every time before we LoadPods(), and then LoadPods() will re-mark the aggregates as "under a VPA" for all the ones that are still there, which lets us easily prune the stale container aggregates that are still marked as "not under a VPA" but are still wrongly in the VPA's list. This does leave the ultimate garbage collection to the rate limited garbage collector, which should be fine, we just needed the stale entries to get removed from the per-VPA lists so they didn't affect VPA behavior.

Signed-off-by: Max Cao <[email protected]>

…the updater prunes aggregateContainerStates/recommendations due to non-existent containers Signed-off-by: Max Cao <[email protected]>

…for non-breaking opt-in change Signed-off-by: Max Cao <[email protected]>

…d of containerPolicy Signed-off-by: Max Cao <[email protected]>

…to be linked to new containers We need to set VPAContainersPerPod for a VPA correctly so we can split resources correctly on its first run through the recommendation loop. So I opted to explicitly set it after updating the pod's containers. This also allows old aggregateContainerStates that were previously created from a removed Pod's container, to be reused by a new Pod's container that shares the same VPA and targetRef. This allows recommendations to be updated correctly when aggregates are pruned or created. Signed-off-by: Max Cao <[email protected]>

Also sets the VPAContainersPerPod correctly when initializing VPA from history provider. Signed-off-by: Max Cao <[email protected]>

Signed-off-by: Max Cao <[email protected]>

Changes so that the recommender starts to prune stale aggregates only when all aggregates by containerName no longer belong to an active container. The aggregate of all containers of the same name are considered stale based on the "newest" aggregate state. i.e. the aggregate with the most recent update time. Signed-off-by: Max Cao <[email protected]>

Signed-off-by: Max Cao <[email protected]>

k8s-ci-robot · 2025-04-14T18:22:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jkyros
Once this PR has been reviewed and has the lgtm label, please ask for approval from adrianmoisey. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

vertical-pod-autoscaler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Max Cao <[email protected]>

…Pod field Signed-off-by: Max Cao <[email protected]>

k8s-ci-robot · 2025-05-22T01:56:41Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jkyros · 2025-07-07T14:37:15Z

This was discussed in the 7/7/2025 sig-autoscaling meeting re: whether this was still wanted, sounded like the "momentum shift" was maybe due to in-place VPA taking precedence, and if we get this updated/rebased we can potentially keep this one alive 😄

k8s-ci-robot · 2025-10-04T05:04:36Z

@jkyros: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-autoscaling-e2e-vpa-full	`0f97598`	link	true	`/test pull-autoscaling-e2e-vpa-full`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/bug Categorizes issue or PR as related to a bug. labels Apr 22, 2024

k8s-ci-robot requested review from krzysied and voelzmo April 22, 2024 15:40

k8s-ci-robot added area/vertical-pod-autoscaler cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 22, 2024

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 22, 2024

jkyros marked this pull request as ready for review April 22, 2024 17:56

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 22, 2024

k8s-ci-robot requested a review from kgolab April 22, 2024 17:57

kwiesmueller reviewed Apr 29, 2024

View reviewed changes

voelzmo mentioned this pull request Jun 3, 2024

VPA updater constantly fails to match the container that doesn't even exists #6215

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 31, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 30, 2024

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Sep 20, 2024

voelzmo mentioned this pull request Dec 3, 2024

Pass the whole VPA into cappingRecommendationProcessor.Apply() #7527

Merged

k8s-ci-robot assigned adrianmoisey Dec 3, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 3, 2024

maxcao13 force-pushed the vpa-aggregates-fix-mark-sweep branch 2 times, most recently from c3a1f0e to c84075b Compare December 3, 2024 22:29

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 22, 2025

maxcao13 force-pushed the vpa-aggregates-fix-mark-sweep branch 2 times, most recently from 9a40688 to b8d78c6 Compare January 24, 2025 06:59

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 5, 2025

jkyros and others added 12 commits April 14, 2025 10:33

VPA: Slightly improve runtime on splitting aggregates

c096853

Signed-off-by: Max Cao <[email protected]>

VPA: Introduce pruningGracePeriod which allows a grace period before …

80aa24d

…the updater prunes aggregateContainerStates/recommendations due to non-existent containers Signed-off-by: Max Cao <[email protected]>

VPA: Add globalPruningGracePeriod flag which defaults to a long time …

d1628dd

…for non-breaking opt-in change Signed-off-by: Max Cao <[email protected]>

VPA: refactor pruning grace period as vpa apiObject annotation instea…

13f741a

…d of containerPolicy Signed-off-by: Max Cao <[email protected]>

VPA: resolve various TODOs and review comments

a279159

Also sets the VPAContainersPerPod correctly when initializing VPA from history provider. Signed-off-by: Max Cao <[email protected]>

VPA: Add e2e tests for pruning grace period and stale recommendations

ece9bb8

Signed-off-by: Max Cao <[email protected]>

VPA: Add unit tests for pruning grace period and containersPerPod field

72906af

Signed-off-by: Max Cao <[email protected]>

maxcao13 force-pushed the vpa-aggregates-fix-mark-sweep branch from b8d78c6 to 72906af Compare April 14, 2025 18:21

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 14, 2025

maxcao13 added 2 commits April 14, 2025 11:44

fixup! VPA: re-design aggregate pruning

d07f664

Signed-off-by: Max Cao <[email protected]>

fixup! VPA: Add unit tests for pruning grace period and containersPer…

0f97598

…Pod field Signed-off-by: Max Cao <[email protected]>

raywainman modified the milestones: vertical-pod-autoscaler-1.4.0, vertical-pod-autoscaler-1.5.0 May 8, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 22, 2025

adrianmoisey mentioned this pull request Oct 11, 2025

AEP-7571: Pod-level resources support in VPA #8586

Open

Uh oh!

VPA: prune stale container aggregates, split recommendations over true number of containers #6745

Are you sure you want to change the base?

VPA: prune stale container aggregates, split recommendations over true number of containers #6745

Uh oh!

Conversation

jkyros commented Apr 22, 2024

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Apr 22, 2024

Uh oh!

kwiesmueller Apr 29, 2024

Choose a reason for hiding this comment

Uh oh!

jkyros May 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omerap12 Jan 13, 2025

Choose a reason for hiding this comment

Uh oh!

adrianmoisey Jan 19, 2025

Choose a reason for hiding this comment

Uh oh!

kwiesmueller Apr 29, 2024

Choose a reason for hiding this comment

Uh oh!

jkyros May 2, 2024

Choose a reason for hiding this comment

Uh oh!

omerap12 Jan 13, 2025

Choose a reason for hiding this comment

Uh oh!

k8s-triage-robot commented Jul 31, 2024

Uh oh!

k8s-triage-robot commented Aug 30, 2024

Uh oh!

sreber84 commented Sep 20, 2024

Uh oh!

adrianmoisey commented Dec 3, 2024

Uh oh!

maxcao13 commented Dec 3, 2024

Uh oh!

maxcao13 commented Jan 24, 2025

Uh oh!

k8s-ci-robot commented Apr 14, 2025

Uh oh!

k8s-ci-robot commented May 22, 2025

Uh oh!

jkyros commented Jul 7, 2025

Uh oh!

k8s-ci-robot commented Oct 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

jkyros May 2, 2024 •

edited

Loading