get_all_records_where for message is not currently used

[gthvn1]

👋,

When looking the API for message class it shows that the string expression used as a a parameter for filtering get_all_records_where is not used. And indeed when looking the code the string is ignored and a function that always returns true is used instead.

It looks like for example for VDIs it is implemented and as I don't see the implementation in xapi_vdi.ml I suppose that it is autogenerated? Because for VDIs you can create query like:

    let query = Printf.sprintf "(field \"location\"=\"%s\") and (field \"SR\"=\"%s\")" remote_vdi (Ref.string_of dest_sr_ref) in
    let vdis = XenAPI.VDI.get_all_records_where remote.rpc remote.session query in

Are there any reasons why for the "message" class we don't use the filter? would it be easy to implement the same filtering as for VDIs? maybe there is something already in progress or some other plans ?

Thanks for your insights!

[contificate]

It would likely require some special casing, because messages are not stored in the database but, rather, on disk.

[psafont]

Are there any reasons why for the "message" class we don't use the filter?

It needs to be written from scratch because they are not stored in the database, but as files in a directory.

[gthvn1]

So we could reuse the same "query language" as the one used for DB (I think about db_filter_parse.mly) but instead of querying the database we will "apply" the query to a file.
And I guess that it doesn't make a lot of sense to start with something really simple like grepping a string. Because the filter function is passed until reaching get_real_inner (in xapi_message.ml) so maybe just check that message contains the string by passing a simple function that does that. Something like:
let vm_started_messages = XenAPI.message.get_all_records_where remote.rpc remote.session "VM_STARTED" in could be interesting maybe. And it looks like we just need to check if the message contains "VM_STARTED"?

Edit: Of course it is limited and also return message if body has VM_STARTED so if a VM is called like that we will have the message. But maybe it is better than ignoring the filter ?

[psafont]

usually the way it works is using field equality, there's a whole language for this:

  let master_condition = Eq (Field "host", Literal (Ref.string_of master)) in
  let sr_condition = Eq (Field "SR", Literal (Ref.string_of self)) in
  let plugged_condition =
    Eq (Field "currently_attached", Literal (string_of_bool attached))
  in
  let all = List.fold_left (fun acc p -> And (acc, p)) True in
  let master_pbds =
    Db.PBD.get_refs_where ~__context
      ~expr:(all [master_condition; sr_condition; plugged_condition])

You can see these atoms in ocaml/database/db_filter_types.ml

[gthvn1]

Yes absolutely I saw the grammar (that's why I mention the file db_filter_parse.mly previously). But it looks not so easy to implement all the stuff for message. That's why I was thinking that initially we could perhaps do something very simple that would filter on a string like "VM_STARTED". But I admit that if we had to do something, probably the complete solution is interesting. I don't really realize how difficult it would be.

[contificate]

but instead of querying the database we will "apply" the query to a file.

You wouldn't need to apply it to the file directly. There are functions in Xapi_message for loading the messages from disk, which get marshalled as API.message_t. Then, you provide a marshaller that maps _val (e.g. Field of string | ...) to the field contents, as a string (akin to what the database stores - again, there are functions for this somewhere). So, really, it's just a small adapter you'd need to provide.

---

I don't care for the filtering functionality in Xapi. It's rather problematic that it only really works for non-structural types and doesn't provide a type-safe interface - e.g. logically identical _ sets (encoded as S-exprs) wouldn't be equal at the encoding level (if the order of the listing differs). I don't believe there's any internal sorting to normalise the contents being compared (which would mitigate this).

In future, it may be worth generating combinators (in Db_actions) for allowing developers to build up type-safe queries internally.

[gthvn1]

So IIUC I think that at this point we already have the message from disk as an API.message_t. So we basically need to replace the filter that currently is just fun _ -> true by something that interpret the query using what we have for db filter and match it with message.body.

[contificate]

So we basically need to replace the filter that currently is just fun _ -> true by something that interpret the query using what we have for db filter and match it with message.body.

I would attempt to do it at the level of the evaluator.

See here:

let eval_expr (lookup_val : _val -> string) =
  let compare f _a _b = f (lookup_val _a) (lookup_val _b) in
  let rec f = function
  (* ... *)

So, you would provide such a lookup_val which would map Field f : _val to the (marshalled) contents of the corresponding field in API.message_t. I forget which functions you would need to call to do this conveniently (see Db_actions).