Creating a CRD with broken converter webhook prevents GC controller from initialization

[jprzychodzen]

What happened:

Creating a CRD with broken converter webhook prevents GC controller from initialization, which breaks on informer sync. Additionally, this issue is not visible until gc controller restarts - dynamically added crd resources with non-working converter webhook do not break running GC.

What you expected to happen:

GC controller should initialize with available informers. CRDs with broken converter webhook should not prevent GC controller from working on other resources.

How to reproduce it (as minimally and precisely as possible):

1. Create a cluster
2. Create a CRD (1_crd.yaml)
3. Create a CR (2_crd.yaml)
4. Change CRD to add another version and webhook converter (3_crd.yaml)
5. Restart kube-controll-manager
6. Add deployment
7. Delete deployment
8. Check that pod from deployment is not GC'ed

gc-bug.zip

[neolit123]

/sig api-machinery

[neolit123]

please follow the issue template correctly:
https://github.com/kubernetes/kubernetes/blob/master/.github/ISSUE_TEMPLATE/bug-report.md

including k8s version is important.

[jprzychodzen]

Sure, it happens on current K8s master branch - exact commit is b0abe89ae259d5e891887414cb0e5f81c969c697

• Kubernetes version (use kubectl version):

  kubectl version
  Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", 
  GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.0-alpha.0.30+b0abe89ae259d5-dirty", GitCommit:"b0abe89ae259d5e891887414cb0e5f81c969c697", GitTreeState:"dirty", 
  BuildDate:"2021-04-13T16:11:56Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
  

• Cloud provider or hardware configuration:
  K8s cluster running on GCE started with kubetest. 10 nodes with --gcp-node-size=n1-standard-1 and with preset-e2e-scalability-common env variables
• OS (e.g: cat /etc/os-release):

  cat /etc/os-release
  NAME="Container-Optimized OS"
  ID=cos
  PRETTY_NAME="Container-Optimized OS from Google"
  HOME_URL="https://cloud.google.com/container-optimized-os/docs"
  BUG_REPORT_URL="https://cloud.google.com/container-optimized-os/docs/resources/support-policy#contact_us"
  GOOGLE_METRICS_PRODUCT_ID=26
  GOOGLE_CRASH_ID=Lakitu
  KERNEL_COMMIT_ID=9ca830b4d7ae9ff76f64f4f9f78a0a0b88dfcda4
  VERSION=85
  VERSION_ID=85
  BUILD_ID=13310.1041.9
  

• Kernel (e.g. uname -a):

  Linux e2e-test-jprzychodzen-master 5.4.49+ #1 SMP Wed Sep 23 19:45:38 PDT 2020 x86_64 Intel(R) Xeon(R) CPU @ 2.00GHz GenuineIntel GNU/Linux
  

• Install tools:
  kubetest

[fedebongio]

/assign @yliaog
/cc @caesarxuchao @leilajal
/triage accepted

[yliaog]

i think this is the same issue as reported in #90597

[jprzychodzen]

It might share root cause - informer sync in case of GC should non-block on CRDs (and possibly on other resources?)

I guess we would need some metrics about unsynced informers to handle this properly.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale