contextual logging

[pohly]

Enhancement Description

• One-line enhancement description (can be used as a release note): Contextual logging enables the caller of a function to control all aspects of logging (output formatting, verbosity, additional values and names).

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3077-contextual-logging

• Discussion Link: WG Structured Logging

• Primary contact (assignee): @pohly

• Responsible SIGs: SIG Instrumentation, WG Structured Logging

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.24
  • Beta release target (x.y): 1.28
  • Stable release target (x.y): 1.30

• Alpha

  • KEP (k/enhancements) update PR(s):
    • KEP-3077: contextual logging #3078
    • KEP 3077: contextual logging: simplify scope, avoid transition #3222
  • Code (k/k) update PR(s):
    • contextual logging klog#296
    • logcheck: contextual logging + enhanced checks klog#297
    • logcheck update and golangci-lint integration kubernetes#108159
    • klog v2.60.1 kubernetes#108725
    • contextual logging kubernetes#108995
  • Docs (k/website) update PR(s):
    • features: add ContextualLogging website#32452
    • logging: include information about contextual logging community#6560

• Beta

  • KEP (k/enhancements) update PR(s):
    • KEP-3077: contextual logging: promotion to beta in 1.30 #4219
    • KEP-3077: contextual logging: sync with KEP template #4519
  • Code (k/k) update PR(s):
    • optional, can be added later: slog support kubernetes#120696
    • feature promotion: promote contextual logging to beta, enabled by default kubernetes#122589
  • Docs (k/website) update(s): contextual logging: beta in Kubernetes 1.30 website#45288

• [] Beta (1.35)

  • KEP (k/enhancements) update PR(s): KEP-1602, KEP-3077: structured, contextual logging in 1.35 #5621
  • Code (k/k) update PR(s)]:
    • query (not necessarily complete, some PRs didn't get labeled)
    • some recent examples of recent PRs: Migrate device manager to contextual logging kubernetes#134293, Kubelet: Migrate util/ and logs/ to contextual logging kubernetes#133217, Replace HandleCrash and HandleError calls to use context-aware alternatives kubernetes#134450, Make legacytokentracking controller context aware kubernetes#134259, apimachinery: Use informer.RunWithContext in various components kubernetes#134798, kube-aggregator: Refactor OpenAPI controllers to be context-aware kubernetes#134791, refactor(apiextensions-apiserver): Make finalizingController controller context-aware kubernetes#134451, dra: Use context-aware client-go informer functions kubernetes#134797, pkg/controller: Use informer.RunWithContext in tests kubernetes#134795,
  • Docs (k/website) update(s): none

---

Current configuration

https://github.com/kubernetes/kubernetes/blob/master/hack/logcheck.conf

Status

The following table counts log calls that need to be converted. The numbers for contextual logging include those for structured logging.

At this point, controllers could get converted to contextual logging or one of the components that was already converted to structured logging. If you want to pick one, ping @pohly on the #wg-structured-logging Slack channel. See structured and contextual logging migration instructions for guidance.

• PRs ready for approval
• PRs which need to be reviewed

Besides migrating log calls, we also might have to migrate from APIs which don't support contextual logging to APIs which do:

• k8s.io/client-go/tools/[events|record]: add WithContext
  • cronjob controller: reconsider log verbosity kubernetes#115574
• HandlePanic/HandleError -> HandlePanicWithContext, HandleErrorWithContext

From 2022-10-27 ~= Kubernetes 1.26

The focus was on converting kube-controller-manager. Of 1944 unstructured and/or non-contexual logging calls in pkg/controller and cmd/kube-controller-manager, 82% were converted to structured, contextual logging in Kubernetes 1.27.

Component	Non-Structured Logging	Non-Contextual Logging	Owner
pkg/controller/bootstrap	15	28	@mengjiao-liu, kubernetes/kubernetes#113464
pkg/controller/certificates	22	31	@mengjiao-liu, kubernetes/kubernetes#113994
pkg/controller/clusterroleaggregation	2	2	@mengjiao-liu, kubernetes/kubernetes#113910
pkg/controller/cronjob	1	44	@mengjiao-liu, kubernetes/kubernetes#113428
pkg/controller/daemon	45	85	@249043822, kubernetes/kubernetes#113622
pkg/controller/deployment	23	79	@249043822, kubernetes/kubernetes#113525
pkg/controller/disruption	29	56	@Namanl2001, kubernetes/kubernetes#116021
pkg/controller/endpoint	12	24	lunhuijie (Slack)
pkg/controller/endpointslice	22	36	@Namanl2001, kubernetes/kubernetes#115295
pkg/controller/endpointslicemirroring	18	28	@Namanl2001, kubernetes/kubernetes#114982
pkg/controller/garbagecollector	55	105	@ncdc, kubernetes/kubernetes#113471
pkg/controller/job	12	36	was: @sanwishe, kubernetes/kubernetes#113576, now: @mengjiao-liu
pkg/controller/namespace	30	55	@yangjunmyfm192085, kubernetes/kubernetes#113443
pkg/controller/nodeipam	135	210	@yangjunmyfm192085, kubernetes/kubernetes#112670
pkg/controller/nodelifecycle	60	106	@yangjunmyfm192085, kubernetes/kubernetes#112670
pkg/controller/podautoscaler	9	13	@freddie400, kubernetes/kubernetes#114687
pkg/controller/podgc	10	24	@pravarag, kubernetes/kubernetes#114689
pkg/controller/replicaset	20	49	@Namanl2001, kubernetes/kubernetes#114871
pkg/controller/resourcequota	24	37	@ncdc, kubernetes/kubernetes#113315
pkg/controller/serviceaccount	22	31	@Namanl2001, kubernetes/kubernetes#114918
pkg/controller/statefulset	19	59	@249043822, kubernetes/kubernetes#113840
pkg/controller/storageversiongc	4	6	@songxiao-wang87, kubernetes/kubernetes#113986
pkg/controller/testutil	9	9	@Octopusjust, kubernetes/kubernetes#114061
pkg/controller/ttl	4	8	wxs (Slack) = @songxiao-wang87, kubernetes/kubernetes#113916
pkg/controller/ttlafterfinished	9	15	@obaranov1, kubernetes/kubernetes#115332
pkg/controller/util	0	19	@fatsheep9146, kubernetes/kubernetes#115049
pkg/controller/volume	351	673	@yangjunmyfm192085, kubernetes/kubernetes#113584
pkg/kubelet	1	1805	@fmuyassarov
pkg/scheduler	0	348	@knelasevero, kubernetes/kubernetes#111155
staging/src/k8s.io/apiextensions-apiserver	57	81
staging/src/k8s.io/apimachinery	73	114	@yanjing1104
staging/src/k8s.io/apiserver	262	543
staging/src/k8s.io/client-go	161	267
staging/src/k8s.io/cloud-provider	108	146
staging/src/k8s.io/cluster-bootstrap	2	4
staging/src/k8s.io/code-generator	108	168
staging/src/k8s.io/component-base	32	63
staging/src/k8s.io/component-helpers	7	8
staging/src/k8s.io/controller-manager	10	10
staging/src/k8s.io/csi-translation-lib	3	4
staging/src/k8s.io/kube-aggregator	52	76
staging/src/k8s.io/kube-controller-manager	0	0
staging/src/k8s.io/kubectl	89	147	@yanjing1104
staging/src/k8s.io/legacy-cloud-providers	1445	2238
staging/src/k8s.io/mount-utils	54	92
staging/src/k8s.io/pod-security-admission	1	34	@Namanl2001, kubernetes/kubernetes#114471
staging/src/k8s.io/sample-controller	16	22	@pchan, kubernetes/kubernetes#113879

From 2023-03-17 = Kubernetes v1.27.0-beta.0

All of kube-controller-manager got converted.

Tables created with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && \
echo "------ | ------- | ------" && \
for i in $(find pkg/controller/* pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet  -maxdepth 0 -type d | sort); do \
     echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; \
done

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	7	8	on hold
cmd/kubelet	0	47	@fmuyassarov
cmd/kube-proxy	0	46	on hold
pkg/controller/certificates	22	31	@mengjiao-liu, kubernetes/kubernetes#113994
pkg/controller/deployment	2	5	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/disruption	29	54	@obaranov1, kubernetes/kubernetes#116021, @mengjiao-liu, kubernetes/kubernetes#119147
pkg/controller/endpoint	12	24	@my-git9, kubernetes/kubernetes#116755
pkg/controller/endpointslice	20	35	@Namanl2001, kubernetes/kubernetes#115295
pkg/controller/endpointslicemirroring	18	28	@Namanl2001, kubernetes/kubernetes#114982
pkg/controller/garbagecollector	3	3	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/job	12	35	@sanwishe, kubernetes/kubernetes#113576 (needs new owner?)
pkg/controller/nodeipam	8	13	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/podgc	10	24	@pravarag, kubernetes/kubernetes#114689, @pohly, kubernetes/kubernetes#119250
pkg/controller/replicaset	9	18	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/statefulset	3	5	@kerthcet, kubernetes/kubernetes#118071
pkg/controller/testutil	9	9	@Octopusjust, kubernetes/kubernetes#114061
pkg/controller/util	0	4	@fatsheep9146, kubernetes/kubernetes#116930
pkg/controller/volume	5	20	@fatsheep9146, kubernetes/kubernetes#116930
pkg/kubelet	2	1923	@fmuyassarov, kubernetes/kubernetes#114352
pkg/scheduler	2	349	@mengjiao-liu, kubernetes/kubernetes#91633

From 2023-09-18 =~ Kubernetes v1.28

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	6	7	on hold
cmd/kubelet	0	52	@fmuyassarov (?), kubernetes/kubernetes#114352
cmd/kube-proxy	0	41	on hold
pkg/kubelet	2	1942	@fmuyassarov (?)
pkg/scheduler	1	137	@mengjiao-liu, https://github.com/kubernetes/kubernetes/pulls/mengjiao-liu
staging/src/k8s.io/apiserver	?	?	@tallclair, kubernetes/kubernetes#114198
staging/src/k8s.io/client-go/discovery	11	21	on hold
staging/src/k8s.io/client-go/examples	14	14	on hold
staging/src/k8s.io/client-go/metadata	2	4	on hold
staging/src/k8s.io/client-go/plugin	5	8	on hold
staging/src/k8s.io/client-go/rest	16	37	on hold
staging/src/k8s.io/client-go/restmapper	3	6	on hold
staging/src/k8s.io/client-go/tools	104	171	@pohly, kubernetes/kubernetes#120729
staging/src/k8s.io/client-go/transport	17	31	on hold
staging/src/k8s.io/client-go/util	12	19	on hold

Table created manually and with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && \
echo "------ | ------- |  ------- | ------" && \
for i in $(find pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet staging/src/k8s.io/client-go/* -maxdepth 0 -type d | sort); do \
     echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; \
done

From 2023-11-20 =~ Kubernetes v1.29

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	6	7	@tallclair
cmd/kubelet	0	52	@fmuyassarov (?), kubernetes/kubernetes#114352
pkg/kubelet	2	1983	@fmuyassarov
cmd/kube-proxy	0	42	@ fatsheep9146, kubernetes/kubernetes#122197
pkg/proxy	0	360	@ fatsheep9146, see above
staging/src/k8s.io/apiserver	285	655	@tallclair, kubernetes/kubernetes#114198
staging/src/k8s.io/client-go/discovery	11	21
staging/src/k8s.io/client-go/examples	14	14
staging/src/k8s.io/client-go/metadata	2	4
staging/src/k8s.io/client-go/plugin	5	8
staging/src/k8s.io/client-go/rest	16	37
staging/src/k8s.io/client-go/restmapper	3	6
staging/src/k8s.io/client-go/tools	83	143	@pohly
staging/src/k8s.io/client-go/transport	17	31
staging/src/k8s.io/client-go/util	12	19

Table created with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && echo "------ | ------- |  ------- | ------" && for i in $(find pkg/scheduler pkg/kubelet pkg/apis pkg/api cmd/kube-* cmd/kubelet staging/src/k8s.io/client-go/* staging/src/k8s.io/apiserver -maxdepth 0 -type d | sort); do      echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; done

From 2025-08-26 = Kubernetes v1.34.0-rc.2

Done (summary, see logcheck.conf for details):

• kube-scheduler
• kube-controller-manager

Component	Non-Structured Logging	Non-Contextual Logging	Owner
cmd/kube-apiserver	2	3
pkg/auth	1	1
pkg/certauthorization	2	4
pkg/controller	0	4
pkg/controlplane	83	112
pkg/credentialprovider	27	51
pkg/kubeapiserver	3	17
pkg/kubectl	1	2
pkg/kubelet	1	1328
pkg/kubemark	7	7
pkg/probe	7	24
pkg/proxy	2	138
pkg/registry	58	113
pkg/routes	2	2
pkg/scheduler	2	2
pkg/serviceaccount	30	52
pkg/util	12	43
pkg/volume	551	897
pkg/windows	1	1
staging/src/k8s.io/apimachinery/pkg	74	119
staging/src/k8s.io/apiserver	322	739	@tallclair (?), kubernetes/kubernetes#114198
staging/src/k8s.io/client-go	107	190	@pohly, kubernetes/kubernetes#129125 (an integration branch of different PRs)

Table created with:

go install sigs.k8s.io/logtools/logcheck@latest

echo "Component | Non-Structured Logging | Non-Contextual Logging | Owner " && echo "------ | ------- |  ------- | ------" && for i in $(find cmd/kube-apiserver cmd/kubelet cmd/kubelet pkg/* staging/src/k8s.io/client-go staging/src/k8s.io/apimachinery/* staging/src/k8s.io/apiserver -maxdepth 0 -type d | sort); do      echo "$i | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false 2>&1 ./... | wc -l ) | $(cd $i; ${GOPATH}/bin/logcheck -check-structured -check-deprecations=false -check-contextual ./... 2>&1 | wc -l ) |" | grep -v '| 0 | 0 |'; done

Note that the table ignores logcheck suppressions (= calls where it was decided to not convert them).

[pohly]

/sig instrumentation
/wg structured-logging

[hosseinsalahi]

Hello @pohly

v1.24 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00pm PT on Thursday Feb 3rd, 2022. This enhancement is targeting alpha for v1.24,

Here’s where this enhancement currently stands:

• Updated KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for this release
• KEP has a test plan section filled out.
• KEP has up to date graduation criteria.
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is marked as tracked. Please keep the issue description and the targeted stage up-to-date for release v1.24.
Thanks!

[pohly]

@encodeflush : the KEP PR was merged, all criteria for alpha in 1.24 should be met now.

[chrisnegus]

Hi @pohly 👋 1.24 Docs shadow here.

This enhancement is marked as 'Needs Docs' for the 1.24 release.

Please follow the steps detailed in the documentation to open a PR against the dev-1.24 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thu March 31, 11:59 PM PDT.

Also, if needed take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thanks!