Skip to content

Commit 09e6360

Browse files
author
George Harley
committed
Add initial version of Vault integration example
* setup.sh installs Vault to current cluster and configures it for the work carried out in the test.sh script * test.sh goes through a series of steps to demonstrate how the operator can use Vault for accessing admin credentials for a RabbitMQ cluster * README.md describes the basic flow of the example as well as stating that the RabbitMQ Cluster operator and Messaging Topology operators are pre-requisites for this example
1 parent 2b8ee9f commit 09e6360

File tree

10 files changed

+413
-11
lines changed

10 files changed

+413
-11
lines changed

‎docs/examples/vault-support/README.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# HashiCorp Vault Support Example
2+
3+
The RabbitMQ Cluster Operator supports storing RabbitMQ default user (admin)
4+
credentials and RabbitMQ server certificates in
5+
[HashiCorp Vault](https://www.vaultproject.io/) and in K8 Secrets. And
6+
accordingly, the RabbitMQ Messaging Topology operator can operate on clusters
7+
which are vault-enabled or k8s-secrets enabled.
8+
9+
As explained in [this KubeCon talk](https://youtu.be/w0k7MI6sCJg?t=177) there
10+
are four different approaches in Kubernetes to consume external secrets:
11+
12+
1. Direct API
13+
2. Controller to mirrors secrets in K8s
14+
3. Sidecar + MutatingWebhookConfiguration
15+
4. Secrets Store CSI Driver
16+
17+
The RabbitMQ Cluster Operator integrates with Vault using the third approach
18+
(Sidecar + MutatingWebhookConfiguration) whereas the RabbitMQ Messaging
19+
Topology Operator uses the first approach (Direct API).
20+
21+
## Vault-related configuration required
22+
23+
The Vault server must have the version 2 key value secret engine and the
24+
[Vault Kubernetes auth method](https://www.vaultproject.io/docs/auth/kubernetes)
25+
enabled.
26+
27+
```bash
28+
$ vault secrets enable -path=secret kv-v2
29+
```
30+
31+
```bash
32+
$ vault auth enable kubernetes
33+
```
34+
35+
In order for the RabbitMQ Messaging Topology operator to authenticate with
36+
a Vault server and access RabbitMQ cluster default user credentials it is
37+
necessary for the operator container to have one or more environment variables
38+
set.
39+
40+
- `VAULT_ADDR` should be set to the URL of the Vault server API
41+
- `OPERATOR_VAULT_ROLE` should be set to the name of the Vault role that is used when accessing credentials. This defaults to "messaging-topology-operator"
42+
- `OPERATOR_VAULT_NAMESPACE` may be set to the Vault namespace to use when the Messaging Topology operator is authenticating. If not set then the default Vault namespace is assumed
43+
- `OPERATOR_VAULT_AUTH_PATH` may be set to the auth path that the operator ought to use when authenticating to Vault. If not set then it is assumed that the “auth/kubernetes” path should be used
44+
45+
In this example the Vault configuration is carried out automatically using
46+
the [setup.sh](./setup.sh) script.
47+
48+
## Usage
49+
50+
This example requires:
51+
1. RabbitMQ Cluster operator is installed
52+
2. RabbitMQ Messaging Topology operator is installed
53+
54+
Run the [setup.sh](./setup.sh) script to install Vault to the current
55+
Kubernetes cluster. This will also create the necessary credentials,
56+
policy, and configuration in preparation for running the example.
57+
58+
Next, run the [test.sh](./test.sh) script. This automates the execution of the
59+
following steps:
60+
1. Update the existing Messaging Topology operator deployment to set the `VAULT_ADDR` and `OPERATOR_VAULT_ROLE` environment variables. This triggers a redeployment.
61+
2. Create two RabbitMQ clusters: `cluster-vault-a` which uses Vault to store its admin credentials and `cluster-b` which uses the Kubernetes secret alternative.
62+
3. Creates a queue in each of the RabbitMQ clusters. This demonstrates that the Messaging Topology operator is able to use both access methods to interact with the clusters.
63+
4. Deletes both queues from the RabbitMQ clusters.
64+
5. Updates the Messaging Topology operator configuration to set an incorrect value of `OPERATOR_VAULT_ROLE`. Doing this intentionally breaks the operator authentication with Vault. This triggers a redeployment.
65+
6. Repeats the creation of a queue in each of the RabbitMQ clusters. This will succeed for the `cluster-b` but not for `cluster-vault-a` which will no longer be able to connect to Vault.
66+
7. Updates the Messaging Topology operator configuration a further time to restore the correct Vault configuration. This triggers a redeployment.
67+
8. Deletes both queues.
68+
9. Deletes both RabbitMQ clusters.
69+
10. Removes the `VAULT_ADDR` and `OPERATOR_VAULT_ROLE` environment variables from the Messaging Topology operator.
70+
11. Removes Vault.
71+

‎docs/examples/vault-support/rabbitmq-clusters.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
apiVersion: rabbitmq.com/v1beta1
2+
kind: RabbitmqCluster
3+
metadata:
4+
name: cluster-vault-a
5+
namespace: default
6+
annotations:
7+
rabbitmq.com/topology-allowed-namespaces: "*"
8+
spec:
9+
image: rabbitmq:3.9.7-management
10+
replicas: 1
11+
service:
12+
type: NodePort
13+
rabbitmq:
14+
additionalConfig: |
15+
loopback_users = none
16+
secretBackend:
17+
vault:
18+
role: messaging-topology-operator
19+
defaultUserPath: secret/data/rabbitmq/cluster-vault-a/creds
20+
21+
---
22+
23+
apiVersion: rabbitmq.com/v1beta1
24+
kind: RabbitmqCluster
25+
metadata:
26+
name: cluster-b
27+
namespace: default
28+
annotations:
29+
rabbitmq.com/topology-allowed-namespaces: "*"
30+
spec:
31+
image: rabbitmq:3.9.7-management
32+
replicas: 1
33+
service:
34+
type: NodePort
35+
rabbitmq:
36+
additionalConfig: |
37+
loopback_users = none
38+
39+

‎docs/examples/vault-support/rabbitmq-queue-a.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: rabbitmq.com/v1beta1
2+
kind: Queue
3+
metadata:
4+
name: queue-a
5+
spec:
6+
name: queue-a # name of the queue
7+
type: quorum # without providing a queue type, rabbitmq creates a classic queue
8+
autoDelete: false
9+
durable: true # seting 'durable' to false means this queue won't survive a server restart
10+
rabbitmqClusterReference:
11+
name: cluster-vault-a # rabbitmqCluster must exist in the same namespace as this resource
12+

‎docs/examples/vault-support/rabbitmq-queue-b.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: rabbitmq.com/v1beta1
2+
kind: Queue
3+
metadata:
4+
name: queue-b
5+
spec:
6+
name: queue-b # name of the queue
7+
type: quorum # without providing a queue type, rabbitmq creates a classic queue
8+
autoDelete: false
9+
durable: true # seting 'durable' to false means this queue won't survive a server restart
10+
rabbitmqClusterReference:
11+
name: cluster-b # rabbitmqCluster must exist in the same namespace as this resource
12+

‎docs/examples/vault-support/rabbitmq-queue-c.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: rabbitmq.com/v1beta1
2+
kind: Queue
3+
metadata:
4+
name: queue-c
5+
spec:
6+
name: queue-c # name of the queue
7+
type: quorum # without providing a queue type, rabbitmq creates a classic queue
8+
autoDelete: false
9+
durable: true # seting 'durable' to false means this queue won't survive a server restart
10+
rabbitmqClusterReference:
11+
name: cluster-vault-a # rabbitmqCluster must exist in the same namespace as this resource
12+

‎docs/examples/vault-support/rabbitmq-queue-d.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: rabbitmq.com/v1beta1
2+
kind: Queue
3+
metadata:
4+
name: queue-d
5+
spec:
6+
name: queue-d # name of the queue
7+
type: quorum # without providing a queue type, rabbitmq creates a classic queue
8+
autoDelete: false
9+
durable: true # seting 'durable' to false means this queue won't survive a server restart
10+
rabbitmqClusterReference:
11+
name: cluster-b # rabbitmqCluster must exist in the same namespace as this resource
12+

‎docs/examples/vault-support/setup.sh‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
#!/bin/bash
2+
3+
set -euo pipefail
4+
5+
RABBITMQ_NAMESPACE=${RABBITMQ_NAMESPACE:-'default'}
6+
7+
vault_exec () {
8+
kubectl exec vault-0 -c vault -- /bin/sh -c "$*"
9+
}
10+
11+
echo "Installing Vault server and Vault agent injector..."
12+
helm repo add hashicorp https://helm.releases.hashicorp.com
13+
helm repo update
14+
# For OpenShift deployments, also set the following:
15+
# --set "global.openshift=true"
16+
helm install vault hashicorp/vault \
17+
--set='server.dev.enabled=true' \
18+
--set='server.logLevel=debug' \
19+
--set='injector.logLevel=debug' \
20+
--wait
21+
sleep 5
22+
kubectl wait --for=condition=Ready pod/vault-0
23+
24+
echo "Configuring K8s authentication..."
25+
# Required so that Vault init container and sidecar of RabbitmqCluster can authenticate with Vault.
26+
vault_exec "vault auth enable kubernetes"
27+
28+
# In Kubernetes 1.21+ clusters, issuer may need to be configured as described in https://www.vaultproject.io/docs/auth/kubernetes#discovering-the-service-account-issuer
29+
# Otherwise, vault-agent-init container will output "error authenticating".
30+
issuer=$(kubectl get --raw=http://127.0.0.1:8001/.well-known/openid-configuration | jq -r .issuer)
31+
vault_exec "vault write auth/kubernetes/config issuer=\"$issuer\" token_reviewer_jwt=\"\$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" kubernetes_host=https://\${KUBERNETES_PORT_443_TCP_ADDR}:443 kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
32+
#vault_exec "vault write auth/kubernetes/config token_reviewer_jwt=\"\$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" kubernetes_host=https://\${KUBERNETES_PORT_443_TCP_ADDR}:443 kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
33+
34+
# Each RabbitMQ cluster may have its own secret path.
35+
echo "Creating credentials for rabbitmq default user in cluster-vault-a..."
36+
vault_exec "vault kv put secret/rabbitmq/cluster-vault-a/creds username='rabbitmq' password='pwda'"
37+
#echo "Creating credentials for rabbitmq default user in cluster-vault-b..."
38+
#vault_exec "vault kv put secret/rabbitmq/cluster-vault-b/creds username='rabbitmq' password='pwdb'"
39+
40+
# Create a policy that allows reading of the default user credentials for the RabbitMQ cluster cluster-vault-a
41+
# The path must be referenced from the RabbitmqCluster cluster-vault-a CRD spec.secretBackend.vault.defaultUserPath
42+
echo "Creating Vault policy named cluster-vault-a-policy for reading of cluster-vault-a credentials..."
43+
vault_exec "vault policy write cluster-vault-a-policy - <<EOF
44+
path \"secret/data/rabbitmq/cluster-vault-a/creds\" {
45+
capabilities = [\"read\"]
46+
}
47+
EOF
48+
"
49+
50+
# Create a separate policy that allows reading of the default user credentials for the RabbitMQ cluster cluster-vault-b
51+
# The path must be referenced from the RabbitmqCluster cluster-vault-b CRD spec.secretBackend.vault.defaultUserPath
52+
#echo "Creating Vault policy named cluster-vault-b-policy for reading of cluster-vault-b credentials..."
53+
#vault_exec "vault policy write cluster-vault-b-policy - <<EOF
54+
#path \"secret/data/rabbitmq/cluster-vault-b/creds\" {
55+
# capabilities = [\"read\"]
56+
#}
57+
#EOF
58+
#"
59+
60+
# Define a Vault role that needs to be referenced from the RabbitmqCluster cluster-vault-a and cluster-vault-b CRD spec.secretBackend.vault.role
61+
# bound_service_account_names values follow the pattern "<RabbitmqCluster name>-server”.
62+
# bound_service_account_namespaces values must include rabbitmq-system (where the messaging topology operator is deployed) and the namespaces where the RabbitmqClusters are deployed.
63+
#vault_exec "vault write auth/kubernetes/role/messaging-topology-operator bound_service_account_names=cluster-vault-a-server,cluster-vault-b-server,vault-tls-server,messaging-topology-operator bound_service_account_namespaces=$RABBITMQ_NAMESPACE,rabbitmq-system policies=cluster-vault-a-policy,cluster-vault-b-policy ttl=24h"
64+
vault_exec "vault write auth/kubernetes/role/messaging-topology-operator bound_service_account_names=cluster-vault-a-server,vault-tls-server,messaging-topology-operator bound_service_account_namespaces=$RABBITMQ_NAMESPACE,rabbitmq-system policies=cluster-vault-a-policy ttl=24h"
65+

‎docs/examples/vault-support/test.sh‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
#!/bin/bash
2+
3+
set -euo pipefail
4+
5+
# Update the Messaging Topology operator deployment, providing it with the Vault related environment variables
6+
echo "Updating Messaging Topology operator to work with Vault..."
7+
kubectl set env deployment/messaging-topology-operator -n rabbitmq-system --containers='manager' OPERATOR_VAULT_ROLE=messaging-topology-operator VAULT_ADDR=http://vault.default.svc.cluster.local:8200
8+
kubectl wait --for=condition=Available deployment/messaging-topology-operator -n rabbitmq-system
9+
sleep 10
10+
11+
echo
12+
echo "Creating RabbitMQ clusters..."
13+
kubectl apply -f rabbitmq-clusters.yaml
14+
sleep 10
15+
kubectl wait --for=condition=Ready pod/cluster-b-server-0
16+
kubectl wait --for=condition=Ready pod/cluster-vault-a-server-0
17+
18+
echo
19+
echo "Messaging Topology operator attempting to create a queue in each of the RabbitMQ clusters..."
20+
kubectl apply -f rabbitmq-queue-a.yaml
21+
kubectl apply -f rabbitmq-queue-b.yaml
22+
kubectl wait --for=condition=Ready queue.rabbitmq.com/queue-a
23+
kubectl wait --for=condition=Ready queue.rabbitmq.com/queue-b
24+
25+
# Verify that the expected queues exist (one in each cluster)
26+
echo
27+
echo "Listing all queues in RabbitMQ cluster 'cluster-vault-a'..."
28+
kubectl exec cluster-vault-a-server-0 -c rabbitmq -- rabbitmqadmin list queues
29+
echo
30+
echo "Listing all queues in RabbitMQ cluster 'cluster-b'..."
31+
kubectl exec cluster-b-server-0 -c rabbitmq -- rabbitmqadmin list queues
32+
33+
echo
34+
echo "Deleting queues from clusters..."
35+
kubectl delete -f rabbitmq-queue-a.yaml
36+
kubectl delete -f rabbitmq-queue-b.yaml
37+
sleep 10
38+
39+
# Update the Messaging Topology operator deployment, providing it with an intentionally incorrect Vault role identifier
40+
echo
41+
echo "Updating Messaging Topology operator to intentionally break authentication with Vault..."
42+
kubectl set env deployment/messaging-topology-operator -n rabbitmq-system --containers='manager' OPERATOR_VAULT_ROLE=no-such-role
43+
kubectl wait --for=condition=Available deployment/messaging-topology-operator -n rabbitmq-system
44+
sleep 30
45+
46+
# Attempt to create queues in each cluster (creation in the cluster using Vault should fail)
47+
echo
48+
echo "Messaging Topology operator attempting to create a queue in each of the RabbitMQ clusters..."
49+
kubectl apply -f rabbitmq-queue-c.yaml
50+
kubectl apply -f rabbitmq-queue-d.yaml
51+
sleep 10
52+
kubectl wait --for=condition=Ready queue.rabbitmq.com/queue-d
53+
54+
# Verify that only the expected queue in the cluster not using Vault for admin credentials exists
55+
echo
56+
echo "Listing all queues in RabbitMQ cluster 'cluster-vault-a'..."
57+
kubectl exec cluster-vault-a-server-0 -c rabbitmq -- rabbitmqadmin list queues
58+
echo
59+
echo "Listing all queues in RabbitMQ cluster 'cluster-b'..."
60+
kubectl exec cluster-b-server-0 -c rabbitmq -- rabbitmqadmin list queues
61+
62+
# Update the Messaging Topology operator deployment to restore correct Vault role identifier before continuing
63+
echo
64+
echo "Updating Messaging Topology operator to repair Vault authentication configuration..."
65+
kubectl set env deployment/messaging-topology-operator -n rabbitmq-system --containers='manager' OPERATOR_VAULT_ROLE=messaging-topology-operator
66+
kubectl wait --for=condition=Available deployment/messaging-topology-operator -n rabbitmq-system
67+
sleep 30
68+
69+
echo
70+
echo "Deleting queues from clusters..."
71+
kubectl delete -f rabbitmq-queue-c.yaml
72+
kubectl delete -f rabbitmq-queue-d.yaml
73+
sleep 10
74+
75+
echo
76+
echo "Deleting RabbitMQ clusters..."
77+
kubectl delete -f rabbitmq-clusters.yaml
78+
sleep 10
79+
kubectl wait --for=delete pod/cluster-vault-a-server-0
80+
kubectl wait --for=delete pod/cluster-b-server-0
81+
82+
echo
83+
echo "Removing Vault env vars from Messaging Topology operator deployment..."
84+
kubectl set env deployment/messaging-topology-operator -n rabbitmq-system --containers='manager' OPERATOR_VAULT_ROLE- VAULT_ADDR-
85+
kubectl wait --for=condition=Available deployment/messaging-topology-operator -n rabbitmq-system
86+
sleep 20
87+
88+
echo
89+
echo "Deleting Vault..."
90+
helm uninstall vault
91+
sleep 10
92+
kubectl wait --for=delete pod/vault-0

‎go.mod‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,6 @@ require (
4646
github.com/bgentry/speakeasy v0.1.0 // indirect
4747
github.com/cenkalti/backoff/v3 v3.0.0 // indirect
4848
github.com/census-instrumentation/opencensus-proto v0.3.0 // indirect
49-
github.com/cespare/xxhash v1.1.0 // indirect
5049
github.com/cespare/xxhash/v2 v2.1.1 // indirect
5150
github.com/cncf/udpa/go v0.0.0-20210322005330-6414d713912e // indirect
5251
github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158 // indirect
@@ -179,7 +178,7 @@ require (
179178
go.uber.org/atomic v1.9.0 // indirect
180179
go.uber.org/multierr v1.7.0 // indirect
181180
go.uber.org/zap v1.19.0 // indirect
182-
golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf // indirect
181+
golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97 // indirect
183182
golang.org/x/mod v0.4.2 // indirect
184183
golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d // indirect
185184
golang.org/x/oauth2 v0.0.0-20210805134026-6f1e6394065a // indirect
@@ -192,7 +191,7 @@ require (
192191
gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect
193192
google.golang.org/appengine v1.6.7 // indirect
194193
google.golang.org/genproto v0.0.0-20210821163610-241b8fcbd6c8 // indirect
195-
google.golang.org/grpc v1.40.0 // indirect
194+
google.golang.org/grpc v1.41.0 // indirect
196195
google.golang.org/protobuf v1.27.1 // indirect
197196
gopkg.in/cheggaaa/pb.v1 v1.0.28 // indirect
198197
gopkg.in/inf.v0 v0.9.1 // indirect

0 commit comments

Comments
 (0)