AWS AutoDiscovery ASG Limits

[JoelSpeed]

I'm running Kubernetes V1.6.4, CA V0.6.0 with autodiscovery of ASGs on AWS. If I change the maximum or minimum number of nodes in the AWS console, these changes are not reflected in the AutoScaler's config.

I chose to use the autodiscovery as I thought it would enable me to keep a single source of truth for ASG limits.

If I decrease the maximum number of nodes in AWS, CA can try to scale up but will throw an error when aws declines the request.

If I increase the maximum number of nodes in AWS, because the cluster has run out of resource, CA doesn't notice this and won't scale up even though there is capacity to

Given that AWS is constantly being polled for updates to the ASG, shouldn't changes in max/min number of nodes be updated?

I'm not sure whether this behaviour has been implemented and isn't working or whether this would be classed as a feature request but it definitely seems like the intuitive behaviour of the autodiscovery

[MaciekPytel]

cc: @mumoshu

[mumoshu]

Hi @JoelSpeed, thanks for the report 👍

I chose to use the autodiscovery as I thought it would enable me to keep a single source of truth for ASG limits.

Yes, this is the intended use-case of the auto-discovery feature.

Given that AWS is constantly being polled for updates to the ASG, shouldn't changes in max/min number of nodes be updated?

Yes, changes in an ASG's min/max size should be read from and synced to CA when the auto-discovery feature is configured correctly.

---

Would you mind helping me to clarify several things came up in my mind?

• How did you provision your k8s cluster?
• How may k8s clusters in your AWS account?
• Could you share us the YAML definition of the CA deployment?
• What tags are assigned to your ASG?
• Could you share us the result of kubectl logs <your CA pod name> so that we can see if there're error(s)

[JoelSpeed]

Hi @mumoshu :)

Thanks for your reply, to answer your questions:

How did you provision your k8s cluster?

Custom EC2 instances (CoreOS + cloud-config) deployed via terraform

How may k8s clusters in your AWS account?

3 presently, but there could be more

Could you share us the YAML definition of the CA deployment?

--- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: cluster-autoscaler namespace: kube-system labels: app: cluster-autoscaler spec: replicas: 2 selector: matchLabels: app: cluster-autoscaler template: metadata: labels: app: cluster-autoscaler annotations: prometheus.io/scrape: "true" prometheus.io/scheme: http spec: containers: - image: gcr.io/google_containers/cluster-autoscaler:v0.6.0 name: cluster-autoscaler resources: limits: cpu: 20m memory: 100Mi requests: cpu: 10m memory: 50Mi command: - ./cluster-autoscaler - --v=4 - --stderrthreshold=info - --cloud-provider=aws - --skip-nodes-with-local-storage=true - --balance-similar-node-groups=true - --expander=least-waste - --node-group-auto-discovery=asg:tag=cluster-autoscaler/enabled,cluster/deneb env: - name: AWS_REGION value: us-east-1 ports: - name: http containerPort: 8085 volumeMounts: - name: ssl-certs mountPath: /etc/ssl/certs/ca-certificates.crt readOnly: true imagePullPolicy: "Always" volumes: - name: ssl-certs hostPath: path: "/etc/ssl/certs/ca-certificates.crt"

What tags are assigned to your ASG?

Could you share us the result of kubectl logs so that we can see if there're error(s)

Not sure how much you wanted so tailed the last 10mb of it
ca-log
ca-log.txt

Hope that helps to clarify everything

The autoscaler itself is working and is scaling up and down, obeying limits, autodiscovering all of our ASGs that are tagged appropriately, just not detecting changes to ASG limits

[JoelSpeed]

I've had another look through the codebase and found

autoscaler/cluster-autoscaler/core/polling_autoscaler.go

Lines 91 to 92 in 44e0897

	// We should not consider it changed when:
	// * min/max/target/current size of node group(s) changed

I think this is wrong, I think the ASG should be considered changed when the min/max values change.

It seems the only way to update the values is by recreating the autoscaler so should some logic be added here to compare the min/max values?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

/remove-lifecycle stale

This is an actual bug in the code that needs a fix.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[StevenACoffman]

/remove-lifecycle rotten

[StevenACoffman]

This appears related to #647