Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions content/en/docs/concepts/policy/node-resource-managers.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,11 +189,15 @@ The following policy options exist for the static CPU management policy:
`align-by-socket` (alpha, hidden by default)
: Align CPUs by physical package / socket boundary, rather than logical NUMA boundaries (available since Kubernetes v1.25)
`distribute-cpus-across-cores` (alpha, hidden by default)
: allocate virtual cores, sometimes called hardware threads, across different physical cores (available since Kubernetes v1.31)
: Allocate virtual cores, sometimes called hardware threads, across different physical cores (available since Kubernetes v1.31)
`distribute-cpus-across-numa` (alpha, hidden by default)
: spread CPUs across different NUMA domains, aiming for an even balance between the selected domains (available since Kubernetes v1.23)
: Spread CPUs across different NUMA domains, aiming for an even balance between the selected domains (available since Kubernetes v1.23)
Comment on lines +192 to +194
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is unrelated fix I asked @wongchar to pull in

`full-pcpus-only` (beta, visible by default)
: Always allocate full physical cores (available since Kubernetes v1.22)
`strict-cpu-reservation` (alpha, hidden by default)
: Prevent all the pods regardless of their Quality of Service class to run on reserved CPUs (available since Kubernetes v1.32)
Comment on lines +197 to +198
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, fixing a rebase artifact of my previous #48797

`prefer-align-cpus-by-uncorecache` (alpha, hidden by default)
: Align CPUs by uncore (Last-Level) cache boundary on a best-effort way (available since Kubernetes v1.32)

You can toggle groups of options on and off based upon their maturity level
using the following feature gates:
Expand Down Expand Up @@ -273,6 +277,24 @@ of `reservedSystemCPUs` and cause host OS services to starve in real life deploy
If the `strict-cpu-reservation` policy option is enabled, the static policy will not allow
any workload to use the CPU cores specified in `reservedSystemCPUs`.

##### `prefer-align-cpus-by-uncorecache`

If the `prefer-align-cpus-by-uncorecache` policy is specified, the static policy
will allocate CPU resources for individual containers such that all CPUs assigned
to a container share the same uncore cache block (also known as the Last-Level Cache
or LLC). By default, the `CPUManager` will tightly pack CPU assignments which can
result in containers being assigned CPUs from multiple uncore caches. This option
enables the `CPUManager` to allocate CPUs in a way that maximizes the efficient use
of the uncore cache. Allocation is performed on a best-effort basis, aiming to
affine as many CPUs as possible within the same uncore cache. If the container's
CPU requirement exceeds the CPU capacity of a single uncore cache, the `CPUManager`
minimizes the number of uncore caches used in order to maintain optimal uncore
cache alignment. Specific workloads can benefit in performance from the reduction
of inter-cache latency and noisy neighbors at the cache level. If the `CPUManager`
cannot align optimally while the node has sufficient resources, the container will
still be admitted using the default packed behavior.


## Memory Management Policies

{{< feature-state feature_gate_name="MemoryManager" >}}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
title: PreferAlignCpusByUncoreCache
content_type: feature_gate
_build:
list: never
render: false

stages:
- stage: alpha
defaultValue: false
fromVersion: "1.32"
---
When `PreferAlignCpusByUncoreCache` is enabled while the CPU Manager Policy is set to `static`, containers within a `Guaranteed` pod will individually be aligned to an uncore cache group at a best-effort policy. This feature can optimize performance for certain cache-sensitive workloads by minimizing the cpu allocation across uncore caches.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try this

Suggested change
When `PreferAlignCpusByUncoreCache` is enabled while the CPU Manager Policy is set to `static`, containers within a `Guaranteed` pod will individually be aligned to an uncore cache group at a best-effort policy. This feature can optimize performance for certain cache-sensitive workloads by minimizing the cpu allocation across uncore caches.
Enable support for aligning workloads to a cache group that spans multiple cores: an _uncore_ cache group.
To learn more, read [CPU management policies](/docs/tasks/administer-cluster/cpu-management-policies/).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should ideally link to a concept page instead; see #48340 (review)

Copy link
Contributor Author

@wongchar wongchar Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://kubernetes.io/docs/concepts/policy/ - can you add a very brief page into that section that explains the idea?

This feature is a CPU Manager Policy (under Node Resource Managers). Just want to confirm my understanding that you are asking for a standalone concept page for prefer-align-cpus-by-uncorecache that is separate from the existing CPU Manager Policies concept page?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just want to confirm my understanding that you are asking for a standalone concept page for prefer-align-cpus-by-uncorecache that is separate from the existing CPU Manager Policies concept page?

Not that, no.

I'm asking for this feature gate description to hyperlink to a concept page about CPU manager policies. The target of the hyperlink doesn't have to be specifically about prefer-align-cpus-by-uncorecache.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The https://kubernetes.io/docs/concepts/policy/node-resource-managers/ concept page is too basic and SIG Node should put some help towards making it more reader friendly. SIG Docs will help.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the last upload after rebase on top of #48797

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can get #48797 merged and merged into v1.32 docs, then the changes here can go there instead (which is what I recommend doing).

See #48340 (review)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm willing to help with #48797, but @tengqm raised some points about the feasibility in the 1.32 timeframe. I guess we can figure out during the review of #48797 ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something else we can do: merge this as-is, but track some work to do after the release to make the docs more readable.

Adding this extra detail harms the quality and readability of the existing task page, so we should plan that work, but we can accept that harm and not block adding this alpha feature on having it appropriately documented.

Ideally we do block beta graduation on getting the docs redone, though. Beta features should have docs that we think are GA quality (and we ship them to find out).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can and will work on getting #48797 ready anyway (https://kubernetes.slack.com/archives/C0BP8PW9G/p1732284959539839). If we decide to not merge for 1.32 due to the too strict timeline, I'll rebase once the other linked content landed, and once cpumanager content is settled I'll move on to memory manager and others:

  1. cpumanager
  2. memory manager
  3. topology manager
  4. (perhaps) device manager, this component doesn't have policies so it can use some refactoring but probably doesn't belong here.

Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ The following policy options exist for the static `CPUManager` policy:
* `align-by-socket` (alpha, hidden by default) (1.25 or higher)
* `distribute-cpus-across-cores` (alpha, hidden by default) (1.31 or higher)
* `strict-cpu-reservation` (alpha, hidden by default) (1.32 or higher)
* `prefer-align-cpus-by-uncorecache` (alpha, hidden by default) (1.32 or higher)

The `full-pcpus-only` option can be enabled by adding `full-pcpus-only=true` to
the CPUManager policy options.
Expand All @@ -176,5 +177,10 @@ options together at this moment.
The `strict-cpu-reservation` option can be enabled by adding `strict-cpu-reservation=true` to
the CPUManager policy options followed by removing the `/var/lib/kubelet/cpu_manager_state` file and restart kubelet.

The `prefer-align-cpus-by-uncorecache` option can be enabled by adding the
`prefer-align-cpus-by-uncorecache` to the `CPUManager` policy options. If
incompatible options are used, the kubelet will fail to start with the error
explained in the logs.

For mode detail about the behavior of the individual options you can configure, please refer to the
[Node ResourceManagers](/docs/concepts/policy/node-resource-managers) documentation.