Be able to use existing RabbitMQ clusters that are not created/managed by https://github.com/rabbitmq/cluster-operator

[vaikas]

Is your feature request related to a problem? Please describe.
Some users might want to decouple the creation of a rabbitmqcluster from using messaging-topology operator to manage resources on it. Think of it as "Bring my own RabbitMQ". Currently I do not see a way to do this, since all the existing resources require a reference to RabbitMQCluster, for example here:
https://github.com/rabbitmq/messaging-topology-operator/blob/main/api/v1beta1/binding_types.go#L43

And seems like we don't really need the rabbitmqcluster reference except to pull in the connection information so that we can create the client connection, so wondering if we could investigate providing a mechanism to be able to specify the connection information (say in a secret with predefined fields for example) so that we could support the use case for bringing your own existing RabbitMQ broker and not require that it be created via the RabbitMQCluster operator.
https://github.com/rabbitmq/messaging-topology-operator/blob/main/internal/rabbitmq_client_factory.go#L76

Describe the solution you'd like
Perhaps instead of requiring that the credentials for the connection come from RabbitMQCluster reference:
https://github.com/rabbitmq/messaging-topology-operator/blob/main/api/v1beta1/binding_types.go#L43

We could add another way to specify required connection information to be able to connect to existing RabbitMQ clusters?

Describe alternatives you've considered
I don't currently see a way to support this use case because you must use a RabbitMQClusterReference.

Additional context
Add any other context or screenshots about the feature request here.

[ChunyiLyu]

@vaikas Like you said, topology operator needs the right credentials to be able to connect to a RabbitMQ cluster, and adopting the operator to take in a connection string is easy to achieve. However, there are requirements for user permissions and the RabbitMA management plugin to be enabled for that RabbitMQ clusters. Supporting arbitrary Rabbit cluster is more complicated and involved than coming up with a standard way to consume connection information.

The team has developed topology operator with the assumption that people use it only with the cluster operator. In the future, we might consider merging the two operators into one RabbitMQ operator (we were discussing that last week). I don't think this request fits the direction that we are planning, but I will direct the feedback to the team.

[ChunyiLyu]

This is a valid ask to support rabbitmqsources. The use case for rabbitmqsources is to be able to bring in messages from an existing RabbitMQ cluster ANYWHERE. Constraining the RabbitMQ to be cluster operator managed does not make sense. Plus, this should provide better usability for anyone who would like to use topology operator to manage their RabbitMQ cluster as well.

[Zerpet]

I agree there's value in supporting rabbitmqsources, this change will also decouple this operator from the Cluster Operator. This change sounds like it is going to be breaking, by either removing the rabbitmq cluster reference, or by making it optional. I think it'll be valuable to write a design document/KEP for this change if we are going down that route.

[ChunyiLyu]

Topology operator has been released and available for over half a year. We know that it has a group of users and customers. So far, we haven't received any angry complaint from them about our requirement on cluster operator managed RabbitMQ. This tells me that we should preserve the ability to be able to refer to a cluster operator RabbitMQ via a reference (providing name and namespace), so users don't need to worry about getting user credentials and URI themselves, and we should not introduce any breaking behavior by adding this feature.

This feature will be an additive feature that provide an alternative method for connecting to a RabbitMQ. IMO, the ask/requirement that I see for the api change is straightforward: 1) make providing name &/ namespace to a RabbitmqCluster optional 2) add fields to current spec to take in RabbitMQ connection information (username, password, uri) 3) add validation to webhook that people can only provide a name reference to RabbitMQ or connection information but not both.

I plan to present what I did in terms of API in coming team standup to gather some feedbacks on naming and the design. Personally I don't think there is need to write a full-on design document for it because user facing API for this feature and the technical mechanics are rather straightforward in my opinion. Happy to discuss more in our standup either tomorrow or Monday as a team.

[ChunyiLyu]

We had a follow up team discussion about supporting the feature and decided to:

1. preserve the current workflow so users can specify rabbitmqClusterReference.name and .namespace as they use to
2. Add a connectionSecret field to RabbitmqClusterReference as an alternative way to provide reference to users
3. This will make RabbitmqClusterReference.name an optional field and it's a change of current behavior. We decided not to bump the API group version because it's an additive change that does not alter the data structure and therefore there is no need for a conversion webhook