Error when using multiple cidrBlocks

[marvinWolff]

What steps did you take and what happened:
I tried to migrate a cluster to clusterclass.
The cluster has multiple cidr entries specified:

  clusterNetwork:
    pods:
      cidrBlocks:
      # The customer has several vpn-networks which should be excluded from pod-cidr-range, to ensure access from pods to these networks
      - "192.168.0.0/20"
      - "192.168.64.0/18"
      - "192.168.128.0/18"
      - "192.168.192.0/19"
      - "192.168.224.0/20"
    serviceDomain: cluster.local

When applied, we get the following error:

E1006 13:36:33.544202 1 controller.go:326] "Reconciler error" err="error computing the desired state of the Cluster topology: failed to apply patches: failed to generate patch request: failed to calculate global variables: pods: too many CIDRs specified" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" cluster="xxx" namespace="xxx" name="xxx" reconcileID=xxxx

What did you expect to happen:

That the cluster is created with the specified cidrs.

Anything else you would like to add:
I found the check that causes the error in here: https://github.com/kubernetes-sigs/cluster-api/blob/v1.2.0/api/v1beta1/cluster_types.go

The number of entries is limited to 2.

Environment:

• Cluster-api version: 1.2.0
• minikube/kind version: -
• Kubernetes version: (use kubectl version): v1.23.9
• OS (e.g. from /etc/os-release): ubuntu 20.04

/kind bug
/area label topology
/area label control-plane
/area label operator
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[k8s-ci-robot]

@marvinWolff: The label(s) area/label, area/label, area/label cannot be applied, because the repository doesn't have them.

In response to this:

What steps did you take and what happened:
I tried to migrate a cluster to clusterclass.
The cluster has multiple cidr entries specified:

clusterNetwork:
   pods:
     cidrBlocks:
     # The customer has several vpn-networks which should be excluded from pod-cidr-range, to ensure access from pods to these networks
     - "192.168.0.0/20"
     - "192.168.64.0/18"
     - "192.168.128.0/18"
     - "192.168.192.0/19"
     - "192.168.224.0/20"
   serviceDomain: cluster.local

When applied, we get the following error:

E1006 13:36:33.544202 1 controller.go:326] "Reconciler error" err="error computing the desired state of the Cluster topology: failed to apply patches: failed to generate patch request: failed to calculate global variables: pods: too many CIDRs specified" controller="topology/cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" cluster="xxx" namespace="xxx" name="xxx" reconcileID=xxxx

What did you expect to happen:

That the cluster is created with the specified cidrs.

Anything else you would like to add:
I found the check that causes the error in here: https://github.com/kubernetes-sigs/cluster-api/blob/v1.2.0/api/v1beta1/cluster_types.go

The number of entries is limited to 2.

Environment:

• Cluster-api version: 1.2.0
• minikube/kind version: -
• Kubernetes version: (use kubectl version): v1.23.9
• OS (e.g. from /etc/os-release): ubuntu 20.04

/kind bug
/area label topology
/area label control-plane
/area label operator
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@marvinWolff: This issue is currently awaiting triage.

If CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[killianmuldoon]

From my reading of the code we are currently validating in the Cluster Topology controller if a Cluster has more than two cidr ranges defined. This validation isn't being done in the webhook, however, leading to a case like this, where creation works, but reconcile does not.

Previously the GetIPFamily wasn't called in the Cluster controller - it was introduced when the cidrs were added to variables.

cluster-api/api/v1beta1/cluster_types.go

Line 362 in aebeed8

func (c *Cluster) GetIPFamily() (ClusterIPFamily, error) {

Is there any good reason to limit the number of cidrs that can be defined to two? If not the limitation could be removed from the function. If necessary (e.g. if the limitation is correct in other usages) a new function could be created for the topology controller.

[sbueringer]

No idea, but maybe the idea was to allow either IPv4 only, IPv6 only or one CIDR for IPv4 one for IPv6 (just guessing)

What is Kubernetes supporting?

[killianmuldoon]

The change was introduced here: 54c0d30 as you said with the dual-stack update.

Kubernetes seems to support two in the flags to its control plane components: https://kubernetes.io/docs/concepts/services-networking/dual-stack/#configure-ipv4-ipv6-dual-stack

From that it seems that we shouldn't support more than 2 CIDR ranges (and should possibly even validate to ensure they're not both in the same IPv). This is could be a disruptive change though, given we've not been validating this to date.

[killianmuldoon]

@marvinWolff could the use case you're describing be done through an IPAM policy on the nodes (in whatever IPAM CNI you're using) instead of trying to limit the CIDRs at the control plane level?

[killianmuldoon]

When I try to set the pod cidrs as above (and disable the check) this fails to bootstrap with an error from CAPD complaining about a failure to run the kubeadm command:
failed to exec DockerMachine bootstrap: failed to run cloud config

podSubnet: Invalid value: "192.168.0.0/20,192.168.64.0/18,192.168.128.0/18,192.168.192.0/19,192.168.224.0/20": expected one (IPv4 or IPv6) CIDR or two CIDRs from each family for dual-stack networking

@marvinWolff what infrastructure and bootstrap providers are you doing this with? With the current cluster can you see how the multiple CIDR ranges are propogated to the Kubernetes components?

[sbueringer]

From that it seems that we shouldn't support more than 2 CIDR ranges (and should possibly even validate to ensure they're not both in the same IPv). This is could be a disruptive change though, given we've not been validating this to date.

I think ideally we align our validation to what Kubernetes supports. It shouldn't be disruptive to block cases that are not supported by Kubernetes (I assume the api-server/controller/.. don't even come up with the wrong CIDRs (as in your example))

[fabriziopandini]

it seems that also kubeadm is rejecting more than 2 CIDR

https://github.com/kubernetes/kubernetes/blob/93d1c43b491a39740297cf1e76782b99d8739865/cmd/kubeadm/app/apis/kubeadm/validation/validation.go#L391-L393