Topology Operator does not respect value of `management.path_prefix ` configuration

[matbarker]

Describe the bug

Declaring a user with the topology operator fails to declare with the error "Error 404 (Object Not Found): Not Found"

To Reproduce

apiVersion: v1
kind: Secret
metadata:
  name: user-secret
  namespace: xxx #same namespace as the deployed RabbitmqCluster
type: Opaque
stringData:
  username: test
  password: test
---
apiVersion: rabbitmq.com/v1beta1
kind: User
metadata:
  name: tutorial-user
  namespace: xxx #same namespace as the deployed RabbitmqCluster
spec:
  importCredentialsSecret:
    name: user-secret
  rabbitmqClusterReference:
    name: rabbitmq # name of the RabbitmqCluster

kubectl apply -f .\user-repro.yaml

secret/user-secret created
user.rabbitmq.com/tutorial-user created

kubectl describe user.rabbitmq.com tutorial-user -n xxx

Name:         tutorial-user
Namespace:    xxx
Labels:       <none>
Annotations:  <none>
API Version:  rabbitmq.com/v1beta1
Kind:         User
Metadata:
  Creation Timestamp:  2022-01-24T01:10:48Z
  Finalizers:
    deletion.finalizers.users.rabbitmq.com
  Generation:  2
  Managed Fields:
    API Version:  rabbitmq.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:importCredentialsSecret:
          .:
          f:name:
        f:rabbitmqClusterReference:
          .:
          f:name:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-01-24T01:10:48Z
    API Version:  rabbitmq.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"deletion.finalizers.users.rabbitmq.com":
      f:spec:
        f:rabbitmqClusterReference:
          f:namespace:
      f:status:
        .:
        f:conditions:
        f:credentials:
          .:
          f:name:
    Manager:         manager
    Operation:       Update
    Time:            2022-01-24T01:10:48Z
  Resource Version:  91619254
  Self Link:         /apis/rabbitmq.com/v1beta1/namespaces/xxx/users/tutorial-user
  UID:               c90214f0-dd7d-46f4-acae-ce6746d63c88
Spec:
  Import Credentials Secret:
    Name:  user-secret
  Rabbitmq Cluster Reference:
    Name:       rabbitmq
    Namespace:
Status:
  Conditions:
    Last Transition Time:  2022-01-24T01:10:48Z
    Message:               Error 404 (Object Not Found): Not Found
    Reason:                FailedCreateOrUpdate
    Status:                False
    Type:                  Ready
  Credentials:
    Name:  tutorial-user-user-credentials
Events:
  Type     Reason                  Age                 From             Message
  ----     ------                  ----                ----             -------
  Normal   SuccessfulStatusUpdate  92s                 user-controller  Successfully updated user status
  Warning  FailedDeclare           10s (x15 over 92s)  user-controller  failed to declare user

kubectl logs pod/messaging-topology-operator-f9c69d45b-xf62c -n rabbitmq-system

{"level":"info","ts":1642985902.2593372,"logger":"controller-runtime.manager.controller.queue","msg":"Start reconciling","reconciler group":"rabbitmq.com","reconciler kind":"Queue","name":"tutorial","namespace":"xxx","spec":"{\"name\":\"tutorial\",\"vhost\":\"/\",\"rabbitmqClusterReference\":{\"name\":\"rabbitmq\",\"namespace\":\"\"}}"}
{"level":"error","ts":1642985902.2618473,"logger":"controller-runtime.manager.controller.queue","msg":"failed to declare queue","reconciler group":"rabbitmq.com","reconciler kind":"Queue","name":"tutorial","namespace":"xxx","queue":"tutorial","error":"Error 404 (Object Not Found): Not Found","stacktrace":"github.com/rabbitmq/messaging-topology-operator/controllers.(*QueueReconciler).Reconcile\n\t/workspace/controllers/queue_controller.go:113\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:214"}
{"level":"error","ts":1642985902.2669504,"logger":"controller-runtime.manager.controller.queue","msg":"Reconciler error","reconciler group":"rabbitmq.com","reconciler kind":"Queue","name":"tutorial","namespace":"xxx","error":"Error 404 (Object Not Found): Not Found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:214"}

kubectl get all -n xxx
NAME                    READY   STATUS    RESTARTS   AGE
pod/rabbitmq-server-0   1/1     Running   0          5m25s

NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                        AGE
service/rabbitmq         ClusterIP   10.107.239.163   <none>        5672/TCP,15672/TCP,15692/TCP   5m25s
service/rabbitmq-nodes   ClusterIP   None             <none>        4369/TCP,25672/TCP             5m25s

NAME                               READY   AGE
statefulset.apps/rabbitmq-server   1/1     5m25s

NAME                                    ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/rabbitmq   True               True               5m25s

NAME                              AGE
user.rabbitmq.com/tutorial-user   4m11s

Expected behavior

Expect a user to be declared and be able to use the username/password to connect to the cluster

Version and environment information

• Messaging Topology Operator: [e.g. 0.1.0 or commit if building from source]

    Image:         rabbitmqoperator/cluster-operator:1.11.0
    Image ID:      docker-pullable://rabbitmqoperator/cluster-operator@sha256:2292f5ebac8982c2b9d9d8b75f15a4eee6774807ae625d23d1a8f91d703f9032

• RabbitMQ: [e.g. 3.8.0]
  Image: rabbitmq:3.8.21-management

• RabbitMQ Cluster Operator: [e.g. 1.1.0 or commit if building from source]
  Image: rabbitmqoperator/cluster-operator:1.11.0
  Image ID: docker-pullable://rabbitmqoperator/cluster-operator@sha256:2292f5ebac8982c2b9d9d8b75f15a4eee6774807ae625d23d1a8f91d703f9032

• Kubernetes: [e.g. 1.18.0]
  v1.22.5 on docker-desktop running on Windows 10

Maintainer note: edited title as per this comment

[coro]

Hi @matbarker, thanks for the Issue submission. Since importCredentialsSecret is only applied at creation time, I wonder if there's some form of timing window here. Do you see the same behaviour if you create the secret, then create the user, rather than both at the same time?

It would also be interesting to see if there's any logs about the user controller; it looks like the logs attached are for the queue controller.

Thanks!

[matbarker]

Looks like the same result:

try2usersecret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: my-second-rabbit-user-secret
stringData:
  username: try2user
  password: j3jf3jf3j3j

try2user.yaml:

apiVersion: rabbitmq.com/v1beta1
kind: User
metadata:
  name: my-second-rabbit-user
spec:
  tags:
  - policymaker
  - monitoring # other available tags are 'management' and 'administrator'
  rabbitmqClusterReference:
    name: rabbitmq
  importCredentialsSecret:
    name: my-second-rabbit-user-secret # name of the secret

kubectl apply -f .\try2usersecret.yaml -n xxx
secret/my-second-rabbit-user-secret created
kubectl apply -f .\try2user.yaml -n xxx
user.rabbitmq.com/my-second-rabbit-user created

kubectl describe user.rabbitmq.com my-second-rabbit-user -n xxx
Name:         my-second-rabbit-user
Namespace:    xxx
Labels:       <none>
Annotations:  <none>
API Version:  rabbitmq.com/v1beta1
Kind:         User
Metadata:
  Creation Timestamp:  2022-02-01T00:31:09Z
  Finalizers:
    deletion.finalizers.users.rabbitmq.com
  Generation:  2
  Managed Fields:
    API Version:  rabbitmq.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:importCredentialsSecret:
          .:
          f:name:
        f:rabbitmqClusterReference:
          .:
          f:name:
        f:tags:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-02-01T00:31:09Z
    API Version:  rabbitmq.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"deletion.finalizers.users.rabbitmq.com":
      f:spec:
        f:rabbitmqClusterReference:
          f:namespace:
      f:status:
        .:
        f:conditions:
        f:credentials:
          .:
          f:name:
    Manager:         manager
    Operation:       Update
    Time:            2022-02-01T00:31:09Z
  Resource Version:  94539149
  Self Link:         /apis/rabbitmq.com/v1beta1/namespaces/xxx/users/my-second-rabbit-user
  UID:               dcf17ce6-9608-4b09-bd1b-4d4de391455e
Spec:
  Import Credentials Secret:
    Name:  my-second-rabbit-user-secret
  Rabbitmq Cluster Reference:
    Name:       rabbitmq
    Namespace:
  Tags:
    policymaker
    monitoring
Status:
  Conditions:
    Last Transition Time:  2022-02-01T00:31:09Z
    Message:               Error 404 (Object Not Found): Not Found
    Reason:                FailedCreateOrUpdate
    Status:                False
    Type:                  Ready
  Credentials:
    Name:  my-second-rabbit-user-user-credentials
Events:
  Type     Reason                  Age                 From             Message
  ----     ------                  ----                ----             -------
  Normal   SuccessfulStatusUpdate  22s                 user-controller  Successfully updated user status
  Warning  FailedDeclare           12s (x12 over 22s)  user-controller  failed to declare user

kubectl logs messaging-topology-operator-f9c69d45b-xf62c -n rabbitmq-system

{"level":"info","ts":1643675469.488246,"logger":"controller-runtime.manager.controller.user","msg":"Start reconciling","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","spec":"{\"tags\":[\"policymaker\",\"monitoring\"],\"rabbitmqClusterReference\":{\"name\":\"rabbitmq\",\"namespace\":\"\"},\"importCredentialsSecret\":{\"name\":\"my-second-rabbit-user-secret\"}}"}
{"level":"info","ts":1643675469.4882884,"logger":"controller-runtime.manager.controller.user","msg":"User does not yet have a Credentials Secret; generating","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","user":"my-second-rabbit-user"}
{"level":"info","ts":1643675469.4883528,"logger":"controller-runtime.manager.controller.user","msg":"generating/importing credentials for User my-second-rabbit-user: &v1beta1.User{TypeMeta:v1.TypeMeta{Kind:\"User\", APIVersion:\"rabbitmq.com/v1beta1\"}, ObjectMeta:v1.ObjectMeta{Name:\"my-second-rabbit-user\", GenerateName:\"\", Namespace:\"xxx\", SelfLink:\"/apis/rabbitmq.com/v1beta1/namespaces/xxx/users/my-second-rabbit-user\", UID:\"dcf17ce6-9608-4b09-bd1b-4d4de391455e\", ResourceVersion:\"94539143\", Generation:2, CreationTimestamp:time.Date(2022, time.February, 1, 0, 31, 9, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"rabbitmq.com/v1beta1\\\",\\\"kind\\\":\\\"User\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"name\\\":\\\"my-second-rabbit-user\\\",\\\"namespace\\\":\\\"xxx\\\"},\\\"spec\\\":{\\\"importCredentialsSecret\\\":{\\\"name\\\":\\\"my-second-rabbit-user-secret\\\"},\\\"rabbitmqClusterReference\\\":{\\\"name\\\":\\\"rabbitmq\\\"},\\\"tags\\\":[\\\"policymaker\\\",\\\"monitoring\\\"]}}\\n\"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string{\"deletion.finalizers.users.rabbitmq.com\"}, ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:\"kubectl-client-side-apply\", Operation:\"Update\", APIVersion:\"rabbitmq.com/v1beta1\", Time:time.Date(2022, time.February, 1, 0, 31, 9, 0, time.Local), FieldsType:\"FieldsV1\", FieldsV1:(*v1.FieldsV1)(0xc001e01b30)}, v1.ManagedFieldsEntry{Manager:\"manager\", Operation:\"Update\", APIVersion:\"rabbitmq.com/v1beta1\", Time:time.Date(2022, time.February, 1, 0, 31, 9, 0, time.Local), FieldsType:\"FieldsV1\", FieldsV1:(*v1.FieldsV1)(0xc001e01de8)}}}, Spec:v1beta1.UserSpec{Tags:[]v1beta1.UserTag{\"policymaker\", \"monitoring\"}, RabbitmqClusterReference:v1beta1.RabbitmqClusterReference{Name:\"rabbitmq\", Namespace:\"\"}, ImportCredentialsSecret:(*v1.LocalObjectReference)(0xc00139e9a0)}, Status:v1beta1.UserStatus{ObservedGeneration:0, Conditions:[]v1beta1.Condition(nil), Credentials:(*v1.LocalObjectReference)(nil)}}","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx"}
{"level":"info","ts":1643675469.4883945,"logger":"controller-runtime.manager.controller.user","msg":"An import secret was provided in the user spec","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","user":"my-second-rabbit-user","secretName":"my-second-rabbit-user-secret"}
{"level":"info","ts":1643675469.4884045,"logger":"controller-runtime.manager.controller.user","msg":"Importing user credentials from provided Secret","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","secretName":"my-second-rabbit-user-secret","secretNamespace":"xxx"}
{"level":"info","ts":1643675469.4884374,"logger":"controller-runtime.manager.controller.user","msg":"Retrieved credentials from Secret","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","secretName":"my-second-rabbit-user-secret","retrievedUsername":"try2user"}
{"level":"info","ts":1643675469.4884782,"logger":"controller-runtime.manager.controller.user","msg":"Credentials generated for User","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","user":"my-second-rabbit-user","generatedUsername":"try2user"}
{"level":"info","ts":1643675469.49071,"logger":"controller-runtime.manager.controller.user","msg":"Successfully declared credentials secret","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","secret":"my-second-rabbit-user-user-credentials","namespace":"xxx"}
{"level":"info","ts":1643675469.494901,"logger":"controller-runtime.manager.controller.user","msg":"Successfully updated secret status credentials","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","user":"my-second-rabbit-user","secretRef":"&LocalObjectReference{Name:my-second-rabbit-user-user-credentials,}"}
{"level":"info","ts":1643675469.494998,"logger":"controller-runtime.manager.controller.user","msg":"Retrieved credentials for user","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","user":"my-second-rabbit-user","credentials":"my-second-rabbit-user-user-credentials"}
{"level":"info","ts":1643675469.4950156,"logger":"controller-runtime.manager.controller.user","msg":"Generated user settings","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","user":"my-second-rabbit-user","settings":{"name":"try2user","tags":"policymaker,monitoring","password_hash":"Wkg3MLyaxAynL6hioy5BEz/XQzewY6I5yi0yo3UkdNO6SrD4Y7vER8qbEttGt1zIDhZYO3B4ZRvivacVKYLkmP2YkN8=","hashing_algorithm":"rabbit_password_hashing_sha512"}}
{"level":"error","ts":1643675469.4977117,"logger":"controller-runtime.manager.controller.user","msg":"failed to declare user","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","user":"my-second-rabbit-user","error":"Error 404 (Object Not Found): Not Found","stacktrace":"github.com/rabbitmq/messaging-topology-operator/controllers.(*UserReconciler).Reconcile\n\t/workspace/controllers/user_controller.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:214"}
{"level":"error","ts":1643675469.501709,"logger":"controller-runtime.manager.controller.user","msg":"Reconciler error","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","error":"Error 404 (Object Not Found): Not Found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:214"}
{"level":"info","ts":1643675469.5018651,"logger":"controller-runtime.manager.controller.user","msg":"Start reconciling","reconciler group":"rabbitmq.com","reconciler kind":"User","name":"my-second-rabbit-user","namespace":"xxx","spec":"{\"tags\":[\"policymaker\",\"monitoring\"],\"rabbitmqClusterReference\":{\"name\":\"rabbitmq\",\"namespace\":\"\"},\"importCredentialsSecret\":{\"name\":\"my-second-rabbit-user-secret\"}}"}

[coro]

I can't seem to reproduce with those manifests. Interestingly, it seems like the 404 we're seeing is directly from the RabbitMQ Management API, in response to the PUT that the operator does to create the user. A 404 in response to a PUT in this case is rather odd, and it looks like it's the same case for the queue reconciler from your other set of logs. That would lead me to believe that the path that's used to create the user is wrong in this case. Would you mind double checking:

• That you can access the management UI on the RabbitMQ Cluster (kubectl port-forward -n xxx service/rabbitmq 15672, then log in at localhost:15672 using the credentials in the secret rabbitmq-default-user)
• Is there any configuration on the RabbitmqCluster that would modify the management API, e.g. management.path_prefix? If you're not sure what config would affect this, it might be most helpful for us to see your full RabbitmqCluster manifest if you're comfortable sharing it.

[matbarker]

That you can access the management UI on the RabbitMQ Cluster

Yes, I can login using those credentials, however at http://localhost:15672/rmq-ui/

Is there any configuration on the RabbitmqCluster that would modify the management API

Yes:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq

spec:
  replicas: 1
  resources:
    requests:
      cpu: 0
      memory: 0
  rabbitmq:
    additionalPlugins:
    - rabbitmq_amqp1_0
    additionalConfig:
    |
      management.path_prefix="rmq-ui"

[coro]

Ah, I see. The operator assumes that the management API is served at NAME.NAMESPACE.svc:PORT, so when it attempts to access the API on your cluster, it receives a 404, since in your case the API is served at NAME.NAMESPACE.svc:PORT/rmq-ui.

What is your use case for having the path prefix, out of interest? You should find the topology operator will work if you remove that line from the RabbitmqCluster config, but I'd like to understand if there's a use case we're not currently supporting with that line.

I'll edit the title of this issue to represent this, if that's ok.

[matbarker]

The management UI is exposed via a reverse proxy with other websites on the same host and port, so the prefix is needed to route the traffic to the correct service.

[ChunyiLyu]

@matbarker In the latest topology operator (1.4.0), you can specify a connection secret with a custom url, username, and password for the operator to use. You can find more details here.

[coro]

Going to close this as I believe the latest topology operator has solved this issue. Please feel free to reopen and let us know if the linked solution does not work for you.

[blackjid]

Couldn't the topology operator look into the cluster reference resource to see if there is a path_prefix config. And us it to define where to connect??

Maybe the Cluster resource could have a specific key in the spec to add a management path prefix, so it would be easier for the topology operator to find the information.