branchprotector continuously failing

[BenTheElder]

It's timing out after 5h20m: https://prow.k8s.io/?job=ci-test-infra-branchprotector

This is causing us to not sync branch protection rules, e.g. related to #33857 / #33880

We have a big problem with this tooling not scaling, it uses a ton of API quota and needs to run continuously and scales with the number of repos/branches which is only growing.

/area prow
/sig testing
/priority important-soon

[BenTheElder]

filed #33901 as a stopgap. it's not great.

[BenTheElder]

Aborted the current run and started a new re-run with the latest config and the 12h timeout, but we know it will be at least 5+ hours before we can tell if that worked as a stopgap ... https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-test-infra-branchprotector/1864084620887199744

[BenTheElder]

That run failed at 11h35m57s.

The logs are enourmous and full of 404 errors:

protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.2 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.4 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.5 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-7.0 from protected=false: get current branch protection: getting branch protection 404: Not Found, update jobselector from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.1 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.3 from protected=true: get current branch protection: getting branch protection 404: Not Found], update kubectl-validate: [update alexzielenski-patch-1 from protected=true: get current branch protection: getting branch protection 404: Not Found, update alexzielenski-patch-2 from protected=true: get current branch protection: getting branch protection 404: Not Found, update apelisse-exit-code-suggestion from protected=true: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update kube-scheduler-wasm-extension: [update golang-sdk from protected=false: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.1 from protected=true: get current branch protection: getting branch protection 404: Not Found], update sigs-github-actions: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update knftables: update master from protected=true: get current branch protection: getting branch protection 404: Not Found, update noderesourcetopology-api: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update cloud-pv-admission-labeler: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update cluster-inventory-api: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update obscli: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update node-ipam-controller: [update readme from protected=true: get current branch protection: getting branch protection 404: Not Found, update update-codegen/fix from protected=true: get current branch protection: getting branch protection 404: Not Found, update init from protected=false: get current branch protection: getting branch protection 404: Not Found, update e2e_aojea from protected=true: get current branch protection: getting branch protection 404: Not Found, update integration-tests from protected=true: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update lws: [update release-0.2.0 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.4.0 from protected=false: get current branch protection: getting branch protection 404: Not Found, update release-0.4.1 from protected=false: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.1 from protected=true: get current branch protection: getting branch protection 404: Not Found, update release-0.4.2 from protected=false: get current branch protection: getting branch protection 404: Not Found, update release-0.3.0 from protected=true: get current branch protection: getting branch protection 404: Not Found, update rupliu/migrate from protected=true: get current branch protection: getting branch protection 404: Not Found, update rupliu/refactor from protected=true: get current branch protection: getting branch protection 404: Not Found], update testgrid: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update referencegrant-poc: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update karpenter-provider-cluster-api: [update multiplex-client from protected=false: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update kube-network-policies: [update aojea-patch-3 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-4 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-6 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-7 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-1 from protected=true: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-2 from protected=true: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-5 from protected=false: get current branch protection: getting branch protection 404: Not Found, update aojea-patch-8 from protected=false: get current branch protection: getting branch protection 404: Not Found], update secrets-store-sync-controller: [update gh-pages from protected=false: get policy: kubernetes-sigs/secrets-store-sync-controller=gh-pages defines a policy, which requires protect: true, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update wg-device-management: update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update etcd-manager: [update fix-release-process-docs from protected=false: get current branch protection: getting branch protection 404: Not Found, update main from protected=true: get current branch protection: getting branch protection 404: Not Found], update cve-feed-osv: [update main from protected=true: get current branch protection: getting branch protection 404: Not Found, update vulns-CVE-2020-10749.json from protected=true: get current branch protection: getting branch protection 404: Not Found, update vulns-CVE-2024-5321.json from protected=true: get current branch protection: getting branch protection 404: Not Found, update vulns-CVE-2020-8553.json from protected=false: get current branch protection: getting branch protection 404: Not Found, update vulns-CVE-2024-7646.json from protected=false: get current branch protection: getting branch protection 404: Not Found], update wg-serving: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update multi-network: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update multi-network-api: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update llm-instance-gateway: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update gwctl: [update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update release-0.1 from protected=false: get current branch protection: getting branch protection 404: Not Found], update cni-dra-driver: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update container-object-storage-interface: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update ingate: update main from protected=false: get current branch protection: getting branch protection 404: Not Found, update kjob: update main from protected=false: get current branch protection: getting branch protection 404: Not

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-test-infra-branchprotector/1864084620887199744

[BenTheElder]

I think it at least barely completed running without being killed by the timeout, but the error output is massive and it feels like it's not running correctly.

NOTE: this has actually been ongoing for months AFAICT, there have been scattered threads in slack. It's just gotten enough attention now to make sure we have a tracking issue >.<

[BenTheElder]

some discussion but nothing conclusive in https://kubernetes.slack.com/archives/CDECRSC5U/p1733337720772809

[pohly]

This causes the "Waiting for status to be reported" for the removed pull-kubernetes-verify-lint because it is the job of the branch protector to tell GitHub which statuses it should wait for - or in this case, not wait for anymore.

The removal of that job has not been mirrored to GitHub yet, despite that completed run.

[pacoxu]

A quick proposal can be dividing this CI to several CIs group by sig/project/org. Is this doable?