Does cadvisor work with unified cgroup?

[keepclean]

Hello!

When I tried launch cadvisor on machine with kernel version 4.14.59 and kernel options
... cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1 ...

I found on cadvisor's logs messages like:

Nov 23 08:11:45 <redacted> cadvisor[124466]: I1123 08:11:45.689071 124466 storagedriver.go:50] Caching stats in memory for 2m0s Nov 23 08:11:45 <redacted> cadvisor[124466]: F1123 08:11:45.689216 124466 cadvisor.go:137] Failed to create a Container Manager: mountpoint for cpu not found

[dashpole]

I don't believe it works with unified cgroup.

[bobrik]

Looks like there's some progress: #2309.

[AkihiroSuda]

setMemoryStats() still needs to be updated to support v2.

cadvisor/container/libcontainer/handler.go

Lines 706 to 740 in d16bbe8

	func setMemoryStats(s *cgroups.Stats, ret *info.ContainerStats) {
	ret.Memory.Usage = s.MemoryStats.Usage.Usage
	ret.Memory.MaxUsage = s.MemoryStats.Usage.MaxUsage
	ret.Memory.Failcnt = s.MemoryStats.Usage.Failcnt
	if s.MemoryStats.UseHierarchy {
	ret.Memory.Cache = s.MemoryStats.Stats["total_cache"]
	ret.Memory.RSS = s.MemoryStats.Stats["total_rss"]
	ret.Memory.Swap = s.MemoryStats.Stats["total_swap"]
	ret.Memory.MappedFile = s.MemoryStats.Stats["total_mapped_file"]
	} else {
	ret.Memory.Cache = s.MemoryStats.Stats["cache"]
	ret.Memory.RSS = s.MemoryStats.Stats["rss"]
	ret.Memory.Swap = s.MemoryStats.Stats["swap"]
	ret.Memory.MappedFile = s.MemoryStats.Stats["mapped_file"]
	}
	if v, ok := s.MemoryStats.Stats["pgfault"]; ok {
	ret.Memory.ContainerData.Pgfault = v
	ret.Memory.HierarchicalData.Pgfault = v
	}
	if v, ok := s.MemoryStats.Stats["pgmajfault"]; ok {
	ret.Memory.ContainerData.Pgmajfault = v
	ret.Memory.HierarchicalData.Pgmajfault = v
	}
	workingSet := ret.Memory.Usage
	if v, ok := s.MemoryStats.Stats["total_inactive_file"]; ok {
	if workingSet < v {
	workingSet = 0
	} else {
	workingSet -= v
	}
	}
	ret.Memory.WorkingSet = workingSet
	}

cc @giuseppe

[giuseppe]

I've opened another PR to take care of some other issues on cgroup v2: #2469

@AkihiroSuda are there new stats for cgroup v2 that weren't present when I've first added it?

[AkihiroSuda]

The stats file is just incompatible across v1 and v2. e.g. "total_inactive_file" doesn't exist on v2.

[giuseppe]

The stats file is just incompatible across v1 and v2. e.g. "total_inactive_file" doesn't exist on v2.

but on cgroup v2 the total_inactive_file will not be found in the s.MemoryStats.Stats map and it will be ignored.

I think leaving them not set is fine. Do you think we need a different handling for the stats that don't exist on cgroup v2?

[AkihiroSuda]

On v2 we should subtract inactive_file (rather than total_inactive_file) from workingset

cadvisor/container/libcontainer/handler.go

Lines 732 to 738 in d16bbe8

	if v, ok := s.MemoryStats.Stats["total_inactive_file"]; ok {
	if workingSet < v {
	workingSet = 0
	} else {
	workingSet -= v
	}
	}

[giuseppe]

On v2 we should subtract inactive_file (rather than total_inactive_file) from workingset

thanks, does 3074667 look good to you?

[AkihiroSuda]

thanks, looks good

[eero-t]

Are there still remaining issues or can this be closed?

[yriveiro]

I'm still having "mountpoint for cpu not found" error using version 0.47.2 on Ubuntu 23.04

[iwankgb]

I can't replicate your problem, @yriveiro:

iwan@ubuntu:~$ cat /etc/debian_version 
bookworm/sid
iwan@ubuntu:~$ uname -a
Linux ubuntu 6.2.0-27-generic #28-Ubuntu SMP PREEMPT_DYNAMIC Wed Jul 12 22:39:51 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
iwan@ubuntu:~$ sudo docker run -ti -v /:/rootfs:ro -v /sys:/sys:ro -v /var/run:/var/run:rw -p28080:8080 gcr.io/cadvisor/cadvisor:v0.47.2
W0820 16:01:14.827527       1 machine_libipmctl.go:64] There are no NVM devices!
W0820 16:01:14.833439       1 manager.go:286] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory

[iwankgb]

Verified on Ubuntu 23.04 running with a little help from Qemu. No error is reported and statistics are displayed as expected.

[yriveiro]

Hi @iwankgb you are running the command with docker, I don't have docker, I'm using rootless nerdctl, basically you ran:

sudo docker run -ti -v /:/rootfs:ro -v /sys:/sys:ro -v /var/run:/var/run:rw -p28080:8080 gcr.io/cadvisor/cadvisor:v0.47.2

In my case would be

nerdctl run -ti -v /:/rootfs:ro -v /sys:/sys:ro -v /var/run:/var/run:rw -p28080:8080 gcr.io/cadvisor/cadvisor:v0.47.2