st2api pod cannot connect to mongo service

[guzzijones]

Initially we were recieving the dreaded no replica sets found. I increased all the timeouts and now we are getting the following:

vagrant@devbox:~/stackstorm-k8s$ kubectl logs stackstorm-ha-st2api-5c84c8b77c-c22v5
2023-01-31 20:52:17,451 INFO [-] Using Python: 3.8.10 (/opt/stackstorm/st2/bin/python)
2023-01-31 20:52:17,452 INFO [-] Using fs encoding: utf-8, default encoding: utf-8, locale: en_US.UTF-8, LANG env variable: en_US.UTF-8, PYTHONIOENCODING env variable: notset
2023-01-31 20:52:17,452 INFO [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2023-01-31 20:52:17,452 INFO [-] Using logging config: /etc/st2/logging.api.gunicorn.conf
2023-01-31 20:52:17,453 INFO [-] Using coordination driver: redis
2023-01-31 20:52:17,453 INFO [-] Using metrics driver: noop
2023-01-31 20:52:17,469 INFO [-] Connecting to database "st2" @ "stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless:27017,stackstorm-ha-mongodb-1.stackstorm-ha-mongodb-headless:27017,stackstorm-ha-mongodb-2.stackstorm-ha-mongodb-headless:27017 (replica set)" as user "st2-admin".
2023-01-31 20:52:29,675 ERROR [-] Failed to connect to database "st2" @ "stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless:27017,stackstorm-ha-mongodb-1.stackstorm-ha-mongodb-headless:27017,stackstorm-ha-mongodb-2.stackstorm-ha-mongodb-headless:27017 (replica set)" as user "st2-admin": stackstorm-ha-mongodb-2.stackstorm-ha-mongodb-headless:27017: [Errno -2] Name or service not known,stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless:27017: [Errno -2] Name or service not known, Timeout: 12.0s, Topology Description: <TopologyDescription id: 63d97f812d1fa52f9c889a95, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless:27017: [Errno -2] Name or service not known')>, <ServerDescription ('stackstorm-ha-mongodb-1.stackstorm-ha-mongodb-headless', 27017) server_type: Unknown, rtt: None>, <ServerDescription ('stackstorm-ha-mongodb-2.stackstorm-ha-mongodb-headless', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('stackstorm-ha-mongodb-2.stackstorm-ha-mongodb-headless:27017: [Errno -2] Name or service not known')>]>
2023-01-31 20:52:29,675 ERROR [-] (PID=1) ST2 API quit due to exception.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/cmd/api.py", line 98, in main
    _setup()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2api/cmd/api.py", line 60, in _setup
    common_setup(
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/service_setup.py", line 252, in setup
    db_setup()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/database_setup.py", line 55, in db_setup
    connection = db_init.db_setup_with_retry(**db_cfg)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/persistence/db_init.py", line 79, in db_setup_with_retry
    return db_func_with_retry(
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/persistence/db_init.py", line 58, in db_func_with_retry
    return retrying_obj.call(db_func, *args, **kwargs)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/six.py", line 696, in reraise
    raise value
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/models/db/__init__.py", line 243, in db_setup
    connection = _db_connect(
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/models/db/__init__.py", line 217, in _db_connect
    raise e
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/models/db/__init__.py", line 208, in _db_connect
    connection.admin.command("ping")
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/database.py", line 736, in command
    with self.__client._socket_for_reads(
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1325, in _socket_for_reads
    server = self._select_server(read_preference, session)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1278, in _select_server
    server = topology.select_server(server_selector)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/topology.py", line 241, in select_server
    return random.choice(self.select_servers(selector,
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/topology.py", line 199, in select_servers
    server_descriptions = self._select_servers_loop(
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/pymongo/topology.py", line 215, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: stackstorm-ha-mongodb-2.stackstorm-ha-mongodb-headless:27017: [Errno -2] Name or service not known,stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless:27017: [Errno -2] Name or service not known, Timeout: 12.0s, Topology Description: <TopologyDescription id: 63d97f812d1fa52f9c889a95, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless:27017: [Errno -2] Name or service not known')>, <ServerDescription ('stackstorm-ha-mongodb-1.stackstorm-ha-mongodb-headless', 27017) server_type: Unknown, rtt: None>, <ServerDescription ('stackstorm-ha-mongodb-2.stackstorm-ha-mongodb-headless', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('stackstorm-ha-mongodb-2.stackstorm-ha-mongodb-headless:27017: [Errno -2] Name or service not known')>]>

[guzzijones]

the services:

vagrant@devbox:~/stackstorm-k8s$ kubectl get services
NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
kubernetes                        ClusterIP   10.100.0.1       <none>        443/TCP                                 6d1h
stackstorm-ha-mongodb-headless    ClusterIP   None             <none>        27017/TCP                               19m
stackstorm-ha-rabbitmq            ClusterIP   10.100.185.74    <none>        5672/TCP,4369/TCP,25672/TCP,15672/TCP   19m
stackstorm-ha-rabbitmq-headless   ClusterIP   None             <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   19m
stackstorm-ha-redis               ClusterIP   10.100.92.70     <none>        6379/TCP,26379/TCP                      19m
stackstorm-ha-redis-headless      ClusterIP   None             <none>        6379/TCP,26379/TCP                      19m
stackstorm-ha-st2api              ClusterIP   10.100.203.223   <none>        9101/TCP                                19m
stackstorm-ha-st2auth             ClusterIP   10.100.248.116   <none>        9100/TCP                                19m
stackstorm-ha-st2stream           ClusterIP   10.100.32.227    <none>        9102/TCP                                19m
stackstorm-ha-st2web              NodePort    10.100.41.173    <none>        80:32634/TCP                            19m

[guzzijones]

nslookup can resolve stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless but dig cannot

^Croot@dnsutils:/# dig stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless

; <<>> DiG 9.9.5-9+deb8u19-Debian <<>> stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 4590
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless.        IN A

;; Query time: 4007 msec
;; SERVER: 10.100.0.10#53(10.100.0.10)
;; WHEN: Tue Jan 31 21:32:59 UTC 2023
;; MSG SIZE  rcvd: 83

root@dnsutils:/# dig stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless.default.svc.cluster.local

; <<>> DiG 9.9.5-9+deb8u19-Debian <<>> stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57485
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless.default.svc.cluster.local. IN A

;; ANSWER SECTION:
stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless.default.svc.cluster.local. 5 IN A 128.126.31.30

;; Query time: 0 msec
;; SERVER: 10.100.0.10#53(10.100.0.10)
;; WHEN: Tue Jan 31 21:33:07 UTC 2023
;; MSG SIZE  rcvd: 205

root@dnsutils:/# nslookup stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless
Server:         10.100.0.10
Address:        10.100.0.10#53

Name:   stackstorm-ha-mongodb-0.stackstorm-ha-mongodb-headless.default.svc.cluster.local
Address: 128.126.31.30

[guzzijones]

This change fixes my issue.
how do i dynamically set this in the helm chart template _helper.yaml

vagrant@devbox:~/stackstorm-k8s$ git diff
diff --git a/templates/_helpers.tpl b/templates/_helpers.tpl
index d04e5c0..bff586f 100644
--- a/templates/_helpers.tpl
+++ b/templates/_helpers.tpl
@@ -103,7 +103,7 @@ Generate comma-separated list of nodes for MongoDB-HA connection string, based o
 {{- range $index0 := until $replicas -}}
   {{- $index1 := $index0 | add1 -}}
   {{- if eq $architecture "replicaset" }}
-    {{- $mongo_fullname }}-{{ $index0 }}.{{ $mongo_fullname }}-headless{{ if ne $index1 $replicas }},{{ end }}
+    {{- $mongo_fullname }}-{{ $index0 }}.{{ $mongo_fullname }}-headless.default.svc.cluster.local{{ if ne $index1 $replicas }},{{ end }}
   {{- else }}
     {{- $mongo_fullname }}-{{ $index0 }}.{{ $mongo_fullname }}{{ if ne $index1 $replicas }},{{ end }}
   {{- end -}}

[arms11]

questions:

1. Do you have more than one replicas for MongoDB?
2. are you suggesting explicit reference to .default.svc.cluster.local solves the issue? If yes, I thought it was a default behavior. just curious what is the output when you describe your mogodb headless service.
   command = kubectl describe service st2ha-mongodb-headless

[guzzijones]

1. yes we have 3 instances of mongodb running

vagrant@devbox:~/stackstorm-ha-lm$ ~/kubectl describe service stackstorm-ha-mongodb-headless
Name:              stackstorm-ha-mongodb-headless
Namespace:         default
Labels:            app.kubernetes.io/component=mongodb
                   app.kubernetes.io/instance=stackstorm-ha
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=mongodb
                   helm.sh/chart=mongodb-10.0.1
Annotations:       meta.helm.sh/release-name: stackstorm-ha
                   meta.helm.sh/release-namespace: default
Selector:          app.kubernetes.io/component=mongodb,app.kubernetes.io/instance=stackstorm-ha,app.kubernetes.io/name=mongodb
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              mongodb  27017/TCP
TargetPort:        mongodb/TCP
Endpoints:         xxxxxxxxxx:27017,xxxxxxxxx:27017,xxxxxxxxxx:27017
Session Affinity:  None
Events:            <none>

[cognifloyd]

Try setting dnsConfig.searches to add your cluster dns namespace. Hopefully that will resolve the lookup issues.

[chain312]

Try setting dnsConfig.searches to add your cluster dns namespace. Hopefully that will resolve the lookup issues.

I added the mongo connection timeout to the container and did curl stackstorm-mongodb-1.stackstorm-mongodb-headless:27017 works

But the in-code connection failed

Using the command parsing contact also

dig - t A stackstorm mongo - 0. Stackstorm - directing - headless. Default. SVC. Cluster. The local. @ 10.43.0.10 + short

Do you have any other methods?@cognifloyd @guzzijones

[guzzijones]

pull this branch for your chart dependency. There is a yaml key for clusterDomain fix