Skip to content

🐛 fix(addons): avoid infinite reconciliation loop #5458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 21, 2025
Merged

🐛 fix(addons): avoid infinite reconciliation loop #5458

merged 1 commit into from
May 21, 2025

Conversation

@mloiseleur
Copy link
Contributor

@mloiseleur mloiseleur commented Apr 7, 2025 •
edited
Loading

What type of PR is this?

/kind regression

What this PR does / why we need it:

With latest v2.7.2 and this PR, there is an infinite reconciliation loop for eks addons.

We have seen two problems:

  1. It tries to compare resolveConflicts despite the fact that it's not part of describe-addon output
  2. It fails to compare emptyString ("") (desired when not set) with nil pointer (describe-addon output) on optional Configuration field

Special notes for your reviewer:

Checklist:

  • squashed commits
  • includes documentation
  • includes emoji in title
  • adds unit tests
  • adds or updates e2e tests

Release note:

Fix: Ignore conflict resolution and correctly compare configuration for AddOn reconciliation

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/regression Categorizes issue or PR as related to a regression from a prior release. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority labels Apr 7, 2025
@k8s-ci-robot k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Apr 7, 2025
@mloiseleur
Copy link
Contributor Author

mloiseleur commented Apr 7, 2025 •
edited
Loading

I have confirmed this PR fixes the issue on a live cluster.

cc @damdo @richardcase @gutmensch @faiq

I did not test it, but this issue should also affect v2.8.2 release.
I can rebase this PR on master branch if needed.

@mloiseleur mloiseleur force-pushed the fix/eks-addon-reconciliation branch from 8d226cb to 72b584d Compare April 7, 2025 10:30
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 7, 2025
@mloiseleur mloiseleur force-pushed the fix/eks-addon-reconciliation branch 2 times, most recently from 4591066 to 393b45c Compare April 7, 2025 11:54
@faiq
Copy link
Contributor

faiq commented Apr 7, 2025

/test ?

@k8s-ci-robot
Copy link
Contributor

@faiq: The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-aws-build

/test pull-cluster-api-provider-aws-build-docker

/test pull-cluster-api-provider-aws-build-docker-release-2-7

/test pull-cluster-api-provider-aws-build-release-2-7

/test pull-cluster-api-provider-aws-test-release-2-7

/test pull-cluster-api-provider-aws-verify-release-2-7

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-aws-apidiff-release-2-7

/test pull-cluster-api-provider-aws-e2e-blocking-release-2-7

/test pull-cluster-api-provider-aws-e2e-conformance-release-2-7

/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts-release-2-7

/test pull-cluster-api-provider-aws-e2e-eks-gc-release-2-7

/test pull-cluster-api-provider-aws-e2e-eks-release-2-7

/test pull-cluster-api-provider-aws-e2e-eks-testing-release-2-7

/test pull-cluster-api-provider-aws-e2e-release-2-7

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-aws-apidiff-release-2-7

pull-cluster-api-provider-aws-build

pull-cluster-api-provider-aws-build-docker

pull-cluster-api-provider-aws-build-docker-release-2-7

pull-cluster-api-provider-aws-build-release-2-7

pull-cluster-api-provider-aws-test-release-2-7

pull-cluster-api-provider-aws-verify-release-2-7

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@faiq
Copy link
Contributor

faiq commented Apr 7, 2025

/test pull-cluster-api-provider-aws-e2e-eks-release-2-7

@faiq
Copy link
Contributor

faiq commented Apr 7, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Apr 7, 2025
@richardcase
Copy link
Member

I did not test it, but this issue should also affect v2.8.2 release.
I can rebase this PR on master branch if needed.

Thanks for this @mloiseleur :bow.

Would you be able to rebase on `main? We can then cherrypick the change to release-2.8 and release2-7.

@mloiseleur mloiseleur force-pushed the fix/eks-addon-reconciliation branch from 393b45c to aa414e6 Compare April 13, 2025 15:28
@k8s-ci-robot k8s-ci-robot added do-not-merge/contains-merge-commits needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Apr 13, 2025
@mloiseleur mloiseleur changed the base branch from release-2.7 to main April 13, 2025 15:29
@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 13, 2025
@mloiseleur
Copy link
Contributor Author

/retest

@mloiseleur
Copy link
Contributor Author

@richardcase It's rebased. I changed the PR toward main branch.

@mloiseleur
Copy link
Contributor Author

@faiq @richardcase Friendly reminder. This PR is ready for merge & cherry-picking on last release.

damdo
damdo reviewed May 13, 2025
View reviewed changes
Copy link
Member

@damdo damdo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

/assign @nrb

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 13, 2025
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 18b599c802dc74e39fb26a395f177691bfe6e374

@damdo
Copy link
Member

damdo commented May 21, 2025

/assign @nrb @richardcase @Ankitasw @dlipovetsky @AndiDog

for approval

@nrb
Copy link
Contributor

nrb commented May 21, 2025

/approve
/cherry-pick release-2.7

@k8s-infra-cherrypick-robot
Copy link

@nrb: once the present PR merges, I will cherry-pick it on top of release-2.7 in a new PR and assign it to you.

In response to this:

/approve
/cherry-pick release-2.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nrb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 21, 2025
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented May 21, 2025 •
edited
Loading

@mloiseleur: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cluster-api-provider-aws-verify-release-2-7 aa414e6 link true /test pull-cluster-api-provider-aws-verify-release-2-7
pull-cluster-api-provider-aws-build-release-2-7 aa414e6 link true /test pull-cluster-api-provider-aws-build-release-2-7
pull-cluster-api-provider-aws-build-docker-release-2-7 aa414e6 link true /test pull-cluster-api-provider-aws-build-docker-release-2-7
pull-cluster-api-provider-aws-apidiff-release-2-7 aa414e6 link false /test pull-cluster-api-provider-aws-apidiff-release-2-7
pull-cluster-api-provider-aws-test-release-2-7 aa414e6 link true /test pull-cluster-api-provider-aws-test-release-2-7

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@nrb
Copy link
Contributor

nrb commented May 21, 2025

/test pull-cluster-api-provider-aws-test

@nrb
Copy link
Contributor

nrb commented May 21, 2025

/cherry-pick release-2.8

@k8s-infra-cherrypick-robot
Copy link

@nrb: once the present PR merges, I will cherry-pick it on top of release-2.8 in a new PR and assign it to you.

In response to this:

/cherry-pick release-2.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot merged commit b925771 into kubernetes-sigs:main May 21, 2025
16 of 17 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v2.8 milestone May 21, 2025
@k8s-infra-cherrypick-robot
Copy link

@nrb: new pull request created: #5500

In response to this:

/approve
/cherry-pick release-2.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-infra-cherrypick-robot k8s-infra-cherrypick-robot mentioned this pull request May 21, 2025
Merged
@k8s-infra-cherrypick-robot
Copy link

@nrb: new pull request created: #5501

In response to this:

/cherry-pick release-2.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-infra-cherrypick-robot k8s-infra-cherrypick-robot mentioned this pull request May 21, 2025
Merged
@mloiseleur mloiseleur deleted the fix/eks-addon-reconciliation branch May 28, 2025 21:34
@MinhNguyen-at MinhNguyen-at mentioned this pull request Jun 13, 2025
Open
yongs2 pushed a commit to yongs2/cluster-api-provider-aws that referenced this pull request Jul 21, 2025
@k8s-ci-robot @yongs2
Merge pull request kubernetes-sigs#5458 from mloiseleur/fix/eks-addon…
fd33875 
…-reconciliation

🐛 fix(addons): avoid infinite reconciliation loop
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@damdo damdo damdo left review comments

@dlipovetsky dlipovetsky Awaiting requested review from dlipovetsky

@luthermonson luthermonson Awaiting requested review from luthermonson

Assignees

@richardcase richardcase

@nrb nrb

@AndiDog AndiDog

@dlipovetsky dlipovetsky

@damdo damdo

@Ankitasw Ankitasw

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/regression Categorizes issue or PR as related to a regression from a prior release. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Milestone

v2.8

Development

Successfully merging this pull request may close these issues.

10 participants

@mloiseleur @faiq @k8s-ci-robot @richardcase @damdo @nrb @k8s-infra-cherrypick-robot @AndiDog @dlipovetsky @Ankitasw