configurable utilityImage + clusterDomain and increase default db timeouts to avoid replicaset timeout

[guzzijones]

allow alternate proxy registry to be used with helm chart.
fix replica set timeout error
fix dns resolution errors for mongo and redis

[CLAassistant]

All committers have signed the CLA.

[guzzijones]

Not sure why the key is missing for the checks

[cognifloyd]

This would break my setup, as, in my values:

• I include my private image registry in the name of all the images.
• I include dnsConfig.searches with the cluster namespace so that searches for the short name always resolve.

[cognifloyd]

This needs some kind of doc string.

The clusterDomain is defined here: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration
And the DNS server has to be configured with it as well:
https://stackoverflow.com/a/52941407/1134951

Looks like kube-dns is the thing that defaults to cluster.local (overridable with the --domain param): https://github.com/kubernetes/dns/blob/master/cmd/kube-dns/app/options/options.go#L59

I use coredns where there is not a default domain, and I did not use the cluster.local domain shown in the docs: https://coredns.io/plugins/kubernetes/

How is your cluster's dns configured?

[guzzijones]

I use coredns.
My configmap:

.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

[guzzijones]

my /etc/resolv.conf on the st2-api pod

search default.svc.cluster.local svc.cluster.local cluster.local [redacted]
nameserver 10.100.0.10 
nameserver [redacted]
options ndots:5                                         

[guzzijones]

on the node my kubelet-config.json does have a line for:

{
...
    clusterDomain": "cluster.local",
...
}

[guzzijones]

I added a comment here as well.

[arm4b]

Added a comment for the DB settings

[arm4b]

LGTM

[arm4b]

👍 LGTM. With 3s being default in st2, bumping it to 5s for K8s sounds like an improvement for distributed environment.