Skip to content

configurable utilityImage + clusterDomain and increase default db timeouts to avoid replicaset timeout #356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 22, 2023
Merged

configurable utilityImage + clusterDomain and increase default db timeouts to avoid replicaset timeout #356

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1332ae8
work with docker proxy
guzzijones Jan 31, 2023
5791b33
utilityImage
guzzijones Feb 1, 2023
df15944
clusterDomain values entry
guzzijones Feb 1, 2023
79711af
remove quote around default value
guzzijones Feb 1, 2023
d968971
remove registry variable from _helpers.tpl
guzzijones Feb 6, 2023
7ec5fb0
add in comments for utilityImage and clusterDomain value parameters
guzzijones Feb 6, 2023
8e8c04b
Merge branch 'master' into alternate_registry
cognifloyd Feb 10, 2023
3f0601a
remove hardcoded settings for retry
guzzijones Feb 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,18 @@ app.kubernetes.io/name: {{ $name }}
app.kubernetes.io/instance: {{ $root.Release.Name }}
{{- end -}}

{{/*
Generate Docker utility image line
*/}}
{{- define "stackstorm-ha.utilityImage" -}}
{{- if .Values.image.utilityImage -}}
{{ .Values.image.utilityImage }}
{{- else -}}
docker.io/library/busybox:1.28
{{- end -}}
{{- end -}}


{{/*
Generate Docker image repository: Public Docker Hub 'stackstorm' for FOSS version
*/}}
Expand Down Expand Up @@ -91,7 +103,7 @@ Generate comma-separated list of nodes for MongoDB-HA connection string, based o
{{- range $index0 := until $replicas -}}
{{- $index1 := $index0 | add1 -}}
{{- if eq $architecture "replicaset" }}
{{- $mongo_fullname }}-{{ $index0 }}.{{ $mongo_fullname }}-headless{{ if ne $index1 $replicas }},{{ end }}
{{- $mongo_fullname }}-{{ $index0 }}.{{ $mongo_fullname }}-headless.{{ $.Release.Namespace }}.svc.{{ $.Values.clusterDomain }}{{ if ne $index1 $replicas }},{{ end }}
{{- else }}
{{- $mongo_fullname }}-{{ $index0 }}.{{ $mongo_fullname }}{{ if ne $index1 $replicas }},{{ end }}
{{- end -}}
Expand All @@ -110,9 +122,9 @@ Generate list of nodes for Redis with Sentinel connection string, based on numbe
{{- $sentinel_port := (index .Values "redis" "sentinel" "port") }}
{{- range $index0 := until $replicas -}}
{{- if eq $index0 0 -}}
{{ $.Release.Name }}-redis-node-{{ $index0 }}.{{ $.Release.Name }}-redis-headless:{{ $sentinel_port }}?sentinel={{ $master_name }}
{{ $.Release.Name }}-redis-node-{{ $index0 }}.{{ $.Release.Name }}-redis-headless.{{ $.Release.Namespace }}.svc.{{ $.Values.clusterDomain }}:{{ $sentinel_port }}?sentinel={{ $master_name }}
{{- else -}}
&sentinel_fallback={{ $.Release.Name }}-redis-node-{{ $index0 }}.{{ $.Release.Name }}-redis-headless:{{ $sentinel_port }}
&sentinel_fallback={{ $.Release.Name }}-redis-node-{{ $index0 }}.{{ $.Release.Name }}-redis-headless.{{ $.Release.Namespace }}.svc.{{ $.Values.clusterDomain }}:{{ $sentinel_port }}
{{- end -}}
{{- end -}}
{{- end -}}
Expand Down Expand Up @@ -165,7 +177,7 @@ Reduce duplication of the st2.*.conf volume details
{{- if index .Values "mongodb" "enabled" }}
{{- $mongodb_port := (int (index .Values "mongodb" "service" "port")) }}
- name: wait-for-db
image: busybox:1.28
image: {{ template "stackstorm-ha.utilityImage" . }}
command:
- 'sh'
- '-c'
Expand All @@ -185,7 +197,7 @@ Reduce duplication of the st2.*.conf volume details
{{- if index .Values "rabbitmq" "enabled" }}
{{- $rabbitmq_port := (int (index .Values "rabbitmq" "service" "port")) }}
- name: wait-for-queue
image: busybox:1.28
image: {{ template "stackstorm-ha.utilityImage" . }}
command:
- 'sh'
- '-c'
Expand Down
2 changes: 1 addition & 1 deletion templates/deployments.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1138,7 +1138,7 @@ spec:
initContainers:
{{- if $some_sensors_per_pod }}
- name: {{ $name }}-init-config
image: busybox:1.28
image: '{{ template "stackstorm-ha.utilityImage" $ }}'
volumeMounts:
- name: st2-sensor-config-vol
mountPath: /tmp/st2
Expand Down
2 changes: 1 addition & 1 deletion templates/jobs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ spec:
initContainers:
{{- include "stackstorm-ha.init-containers-wait-for-db" . | nindent 6 }}
- name: wait-for-api
image: busybox:1.28
image: {{ template "stackstorm-ha.utilityImage" . }}
{{- with .Values.securityContext }}
securityContext: {{- toYaml . | nindent 10 }}
{{- end }}
Expand Down
24 changes: 23 additions & 1 deletion values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@
## Docker image settings, applied to all StackStorm pods
##
image:
# utility image to use as sidecar for running startup scripts
# allows you to override the registry namespace as well if you are using a private registry
# ie
# utilityImage: "myprivateregsistry.local/ext.docker.io/library/busybox:1.28"
utilityImage: ""
# Image pull policy
pullPolicy: IfNotPresent
# st2 image repository. Set this to override the default ("stackstorm").
Expand All @@ -20,7 +25,13 @@ image:
# See: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
#pullSecret: "your-pull-secret"


##
## local cluster domain suffix to enable fqdn lookups for redis, mongo
## kube-dns defaults to cluster.local and allows override with --domain
## core-dns has no default but allows config in k8 configmap
## https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configuration-of-stub-domain-and-upstream-nameserver-using-coredns
##
clusterDomain: cluster.local
Copy link
Member

@cognifloyd cognifloyd Feb 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs some kind of doc string.

The clusterDomain is defined here: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration
And the DNS server has to be configured with it as well:
https://stackoverflow.com/a/52941407/1134951

Looks like kube-dns is the thing that defaults to cluster.local (overridable with the --domain param): https://github.com/kubernetes/dns/blob/master/cmd/kube-dns/app/options/options.go#L59

I use coredns where there is not a default domain, and I did not use the cluster.local domain shown in the docs: https://coredns.io/plugins/kubernetes/

How is your cluster's dns configured?

Copy link
Contributor Author

@guzzijones guzzijones Feb 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I use coredns.
My configmap:

.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Copy link
Contributor Author

@guzzijones guzzijones Feb 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my /etc/resolv.conf on the st2-api pod

search default.svc.cluster.local svc.cluster.local cluster.local [redacted]
nameserver 10.100.0.10 
nameserver [redacted]
options ndots:5

Copy link
Contributor Author

@guzzijones guzzijones Feb 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on the node my kubelet-config.json does have a line for:

{
...
    clusterDomain": "cluster.local",
...
}

Copy link
Contributor Author

@guzzijones guzzijones Feb 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a comment here as well.


##
## Service Account
Expand Down Expand Up @@ -70,6 +81,17 @@ st2:
config: |
[api]
allow_origin = '*'
# fixes no replicaset found bug;
[database]
# Connection retry backoff max (seconds).
connection_retry_backoff_max_s = 10
# Backoff multiplier (seconds).
connection_retry_backoff_mul = 1
# Connection retry total time (minutes).
connection_retry_max_delay_m = 3
# Connection and server selection timeout (in ms).
connection_timeout = 12000


#Override Definitions can be added here.
#https://docs.stackstorm.com/latest/packs.html#overriding-pack-defaults
Expand Down