Skip to content

Conversation

@arms11
Copy link
Contributor

@arms11 arms11 commented Feb 11, 2021

Closes #11

Added initContainers for waiting on DB and MQ dependencies to be formed and connected before proceeding to initialize dependent containers of ST2 services.

@pull-request-size pull-request-size bot added the size/L PR that changes 100-499 lines. Requires some effort to review. label Feb 11, 2021
@arms11
Copy link
Contributor Author

arms11 commented Feb 11, 2021

Attempt to resolve #11 . Please review.

@arm4b
Copy link
Member

arm4b commented Feb 11, 2021

Nice!
First side effect from the change is that st2 Pods are reporting Init state until DB/MQ connections are established. I think that's helpful from the user's perspective and signalizes that st2 Pods are waiting for something:

NAME                                            READY   STATUS             RESTARTS   AGE
pod/test-job-st2-apikey-load-bpvcp              0/1     Init:1/3           0          6m46s
pod/test-mongodb-0                              1/1     Running            0          6m46s
pod/test-mongodb-1                              0/1     CrashLoopBackOff   2          21s
pod/test-rabbitmq-0                             0/1     CrashLoopBackOff   0          6m46s
pod/test-redis-node-0                           2/2     Running            0          6m46s
pod/test-redis-node-1                           2/2     Running            0          6m20s
pod/test-redis-node-2                           2/2     Running            2          6m9s
pod/test-st2actionrunner-7f68887485-dfwd5       0/1     Init:1/2           0          6m45s
pod/test-st2actionrunner-7f68887485-g8zqm       0/1     Init:1/2           0          6m45s
pod/test-st2actionrunner-7f68887485-kf7jl       0/1     Init:1/2           0          6m46s
pod/test-st2actionrunner-7f68887485-ml5vv       0/1     Init:1/2           0          6m45s
pod/test-st2actionrunner-7f68887485-qb5kq       0/1     Init:1/2           0          6m45s
pod/test-st2api-6cc98fb84f-fmxtm                0/1     Init:1/2           0          6m46s
pod/test-st2api-6cc98fb84f-rf9dv                0/1     Init:1/2           0          6m46s
pod/test-st2auth-694df655c5-4rbk2               0/1     Init:1/3           0          6m46s
pod/test-st2auth-694df655c5-ggx7p               0/1     Init:1/3           0          6m45s
pod/test-st2client-7b689b8cb9-zg9td             0/1     Init:1/3           0          6m45s
pod/test-st2garbagecollector-67bbb56d54-xsdkp   0/1     Init:1/2           0          6m46s
pod/test-st2notifier-5d94775646-96vgp           0/1     Init:1/2           0          6m46s
pod/test-st2notifier-5d94775646-wwbv2           0/1     Init:1/2           0          6m46s
pod/test-st2rulesengine-7d986597c9-gz44n        0/1     Init:1/2           0          6m46s
pod/test-st2rulesengine-7d986597c9-zrg5k        0/1     Init:1/2           0          6m46s
pod/test-st2scheduler-856ccc64bb-grkxt          0/1     Init:1/2           0          6m46s
pod/test-st2scheduler-856ccc64bb-rxmrg          0/1     Init:1/2           0          6m46s
pod/test-st2sensorcontainer-59cc5cff7-tzm8b     0/1     Init:1/2           0          6m46s
pod/test-st2stream-6464974955-76vm2             0/1     Init:1/2           0          6m46s
pod/test-st2stream-6464974955-jlqcq             0/1     Init:1/2           0          6m46s
pod/test-st2timersengine-5c96d79cbd-xn545       0/1     Init:1/2           0          6m46s
pod/test-st2web-5d5d948d5-5qc5f                 1/1     Running            0          6m46s
pod/test-st2web-5d5d948d5-wkfgz                 1/1     Running            0          6m46s
pod/test-st2workflowengine-66555df5b4-kmt44     0/1     Init:1/2           0          6m46s
pod/test-st2workflowengine-66555df5b4-shdx6     0/1     Init:1/2           0          6m46s

👍

Any downsides or edge cases anyone can think of?

@arm4b arm4b added the enhancement New feature or request label Feb 11, 2021
Copy link
Member

@arm4b arm4b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, thanks for the contribution 👍

Left first comments on top of my head that need addressing.

We'll definitely need a Changelog for this enhancement.
I also think we may omit wait-for scenario for st2client Pod so it can start without Init.

@arms11
Copy link
Contributor Author

arms11 commented Feb 12, 2021

Thanks @armab for the feedback. I will make the requested changes.

@pull-request-size pull-request-size bot added size/M PR that changes 30-99 lines. Good size to review. and removed size/L PR that changes 100-499 lines. Requires some effort to review. labels Feb 12, 2021
@arms11 arms11 requested a review from arm4b February 12, 2021 23:50
{{- end -}}

{{- define "init-containers-wait-for-db" -}}
{{- if index .Values "mongodb" "enabled" }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just thought about that edge case. Perfect to have it already codified here 👍

# Changelog

## In Development
* Implemented initContainers to wait for DB/MQ to be available for St2 Pods (#178)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI I moved this changelog under the In Development section.

Copy link
Member

@arm4b arm4b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this stability improvement! 👍

@arm4b arm4b merged commit 808b83d into StackStorm:master Feb 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size/M PR that changes 30-99 lines. Good size to review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pod scheduling priority: Helm release could be stuck in a restart loop due Mongo/Rabbit clusters glued with a "Pending" state

2 participants