[PodLevelResources] Pod Level Hugepage Resources

[KevinTMtz]

What type of PR is this?

What this PR does / why we need it:

This PR implements Pod Level Hugepage Resources that require following changes:

1. Add hugepages to pod level supported resources
2. Default pod level hugepage limits
3. Containers with hugepage volume mounts with unset hugepage limits
4. Use pod level hugepage limits for cgroup when unset in container
5. Unit tests for pod level hugepage resources
6. End to end tests for pod level hugepage resources

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

- Changed the Pod API to support `hugepage resources` at `spec` level for pod-level resources.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [Other doc]: https://docs.google.com/document/d/1JaqE2eRmFAPlRayv8vsAWE4SmQCVXQLr9rFPhEaPlvQ/edit?usp=sharing

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

Hi @KevinTMtz. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[thockin]

Ping me for final approval when kubelet reviews are done.

[tallclair]

Does

kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container_linux.go

Line 146 in 648620d

lcr.HugepageLimits = GetHugepageLimitsFromResources(container.Resources)

need to be updated to fallback to pod-level resources?

[KevinTMtz]

Does

kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container_linux.go

Line 146 in 648620d

lcr.HugepageLimits = GetHugepageLimitsFromResources(container.Resources)

need to be updated to fallback to pod-level resources?

Currently we are not going to set the cgroup values for the containers that do not specify hugepage limits, because of the starving problem caused by not having hugepages isolation between the containers that specify limits and the ones that do not, when setting the HugeTLB cgroups to max (or max minus the aggregated limits) of the containers that did not specify.
If we do not set the hugepage cgroup to the container with unset request/limits, the behavior will depend of the containerd HugeTLB being enabled or not:

• Containerd HugeTLB Enabled

  • The containers with hugepage request/limit will not be starved, their limits will be respected (because only them will be able to use the hugepages, and the available hugepages will be the aggregated sum of their request/limit)
  • The containers without hugepage request/limit will not have access to any hugepages

• Containerd HugeTLB Disabled

  • The containers with hugepage request/limit set and unset will have the same behavior, all of them will be able to access the whole hugepage resources in the pod, all of them will be able to starve all the others

Initially, I added the commit Use pod level hugepage limits for cgroup when unset in container, however because of this situation, I reverted the change

@tallclair

[ndixita]

@KevinTMtz Is it allowed for aggregated huge pages limits to be greater than pod-level huge pages limits? Did we test this case?

[KevinTMtz]

@KevinTMtz Is it allowed for aggregated huge pages limits to be greater than pod-level huge pages limits? Did we test this case?

It is not possible, since hugepages can not be overcommitted.

If the user tries to do something like this:

apiVersion: v1
kind: Pod
metadata:
  name: pod-level-hugepages
spec:
  resources:
    limits:
      memory: 256Mi
      hugepages-2Mi: 512Mi
  containers:
  - name: hptest-1
    image: "hptest:0.1"
    resources:
      limits:
        cpu: 1
        hugepages-2Mi: 514Mi
    volumeMounts:
    - mountPath: /hugepages-2Mi
      name: hugepage-2mi
  volumes:
  - name: hugepage-2mi
    emptyDir:
      medium: HugePages-2Mi

This error will be obtained:

The Pod "pod-more-container" is invalid: 
* spec.resources.requests[hugepages-2Mi]: Invalid value: "512Mi": must be greater than or equal to aggregate container requests of 514Mi
* spec.resources.containers[0][hugepages-2Mi].limits: Invalid value: "514Mi": must be less than or equal to pod limits of 512Mi

@ndixita

[tallclair]

/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: dd85775c100da0e1d90446ced8156d5fc33c1ac6

[thockin]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: KevinTMtz, tallclair, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/apis/OWNERS [thockin]
• pkg/scheduler/OWNERS [thockin]
• pkg/volume/emptydir/OWNERS [thockin]
• staging/src/k8s.io/component-helpers/resource/OWNERS [thockin]
• test/e2e_node/OWNERS [tallclair,thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

a couple comments here:

1. this is effectively relaxing validation, which means we have to make sure it isn't active by default in the first release it is present in (1.33 will choke on pods this allows in with pod-level hugepages requests/limits)
2. this allowed things into the API which weren't actually implemented or tested yet (implementation is in [PodLevelResources] Propagate Pod level hugepage cgroup to containers #131089) ... we generally want to make sure implementation merges with API

both of those issues need to be addressed before pod-level resources enables by default

[liggitt]

oh, I just realized this merged in 1.33, not 1.34. whew

[tallclair]

Kevin also clarified that this PR did implement support. The implementation is implicit - pod level huge pages were already set on the pod cgroup using the Pod{Requests,Limits} helper which already factors in pod-level resources, so once we allow huge-pages to be set at the pod-level, the implementation is already there.