add upgrade downgrade version skep

Author: aojea

keps/sig-network/1880-services-ip-ranges/README.md

@@ -282,7 +282,7 @@ type IPRangeStatus struct {
 
 ##### Controller
 
-The IPRange object will be reconciled by a controller running in the kube-controller-manager that will be update the Status accordenly.
+The IPRange object will be reconciled by a controller running in the kube-controller-manager that will be update the Status accordingly.
 The controller with perform the following operations:
 - cross validation, per example, avoiding overlapping IP Ranges
 - resizing subnets
@@ -453,18 +453,19 @@ We expect no non-infra related flakes in the last month as a GA graduation crite
 
 - Feature implemented behind a feature flag
 - Initial unit, integration and e2e tests completed and enabled
+- Only basic functionality implemented
 
 #### Beta
 
-- Gather feedback from developers and surveys
-- Complete features for IPRange resizing and changing primary IP family
+- API stability, no disruptive changes on types and behaviors
+- Gather feedback from developers and users
+- Complete all the advanced features: resizing, changing primary IP family, ...
 - Additional tests are in Testgrid and linked in KEP
 - Allowing time for feedback
 
 #### GA
 
-- 3 examples of real-world usage
-- 3 installs
+- 2 examples of real-world usage
 - More rigorous forms of testing—e.g., downgrade tests and scalability tests
 - Allowing time for feedback
 
@@ -479,34 +480,38 @@ in back-to-back releases.
 
 #### Deprecation
 
-### Upgrade / Downgrade Strategy
+### Upgrade / Downgrade / Version Skew Strategy
 
-<!--
-If applicable, how will the component be upgraded and downgraded? Make sure
-this is in the test plan.
+Currently, the Service CIDRs are configured independently in each kube-apiserver using flags.
+During the bootstrap process, the apiserver uses the first IP of each range to create the special "kubernetes.default" service.
+It also starts a reconcile loop, that synchronize the state of the bitmap with the assigned IPs to the Services.
 
-Consider the following in developing an upgrade/downgrade strategy for this
-enhancement:
-- What changes (in invocations, configurations, API use, etc.) is an existing
-  cluster required to make on upgrade, in order to maintain previous behavior?
-- What changes (in invocations, configurations, API use, etc.) is an existing
-  cluster required to make on upgrade, in order to make use of the enhancement?
--->
+It is a known limitation that each kube-apiserver can boot with different ranges and create conflicts.
 
-### Version Skew Strategy
+In order to be completely backwards compatible, the bootstrap process will remain the same, the difference is
+that instead of creating a bitmap based on the flags, it will create a well-known IPRange object with name "default"
+and configured with the CIDRs passed as flags.
 
-<!--
-If applicable, how will the component handle version skew with other
-components? What are the guarantees? Make sure this is in the test plan.
+```
+<<[UNRESOLVED conflict ]>>
+Currently, there is no conflict resolution for incompatible configurations on kube-apiservers, should we
+maintain current behavior and let the apiservers "keep fighting" to set their own parameters, or just
+fail to start?
+<<[/UNRESOLVED]>>
+```
 
-Consider the following in developing a version skew strategy for this
-enhancement:
-- Does this enhancement involve coordinating behavior in the control plane and
-  in the kubelet? How does an n-2 kubelet without this feature available behave
-  when this feature is used?
-- Will any other components on the node change? For example, changes to CSI,
-  CRI or CNI may require updating that component before the kubelet.
--->
+The source of truth are the IPs assigned to the Services, both the old and new methods have reconcile loops that
+rebuild the state of the allocators based on the assigned IPs to the Services, this allows to support
+upgrades and skewed clusters.
+
+
+Since the new allocation model will remove some of the limitations of the current model, skewed versions and downgrades
+can only work if the configurations are fully compatible, per example, current CIDRs are limited to a /112 max for IPv6,
+if an user configures a /64 to their IPv6 subnets in the new model, and IPs are assigned out of the first /112 block,
+the old allocator based in bitmap will not be able to use those IPs creating an inconsistency in the cluster.
+
+It is recommended that those Services are recreated to get IP addresses inside the configured ranges, for consistency,
+but there should not be any functional problem in the cluster.
 
 ## Production Readiness Review Questionnaire