[Security Solution][Alerts] Saved query rule type UX improvements

[marshallmain]

Acceptance Criteria

• User should be able to tell when creating/editing/viewing a rule if it is dynamically loading the saved query on rule execution or the saved query is copied on rule creation
• User should be able to edit a rule and change between a static query and a dynamically loaded saved query

Related Issues

• [Security App][Detections] Unable to make changes to older rules that were saved_query type #127429
• [Security Solution] Saved Query Rule: rule-modified queries are shown in the UI, but do not affect rule execution #136905
• [Security Solution][Detections] Unexpected behaviors of saved queries in detection rules #76592
• [Security Solution] Id is displayed instead of Saved Query name under Rule details page #136178

Proposed Implementation

In the Alerts Area sync we discussed the possibility of adding a checkbox in the rule creation/editing flow to the effect of "Check this box to dynamically load the saved query on each rule execution". In the backend, checking this box is equivalent to the saved_query rule type, while leaving it unchecked is equivalent to the query rule type. A user can still load a saved query for a "query" rule type, but it will only be loaded by the frontend and the query itself will be copied from the saved query into the rule.

In order to allow users to check/uncheck this box when editing a rule, we also need to re-combine the query and saved query rule types at the alerting framework level. This means migrating the existing saved_query rules to be query rules instead in the framework and updating the siem.queryRule schema to allow both query and saved_query rules. The query and saved_query rule types both use the same executor function internally, so we should not need many changes in the executors to re-combine them.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[marshallmain]

Slight change in direction re: migrating siem.savedQueryRule rules to siem.queryRule: since this migration would require disabling rules and user action to re-enable the rules after upgrading, we want to avoid that approach. Instead, we would like to make both the siem.queryRule and siem.savedQueryRule executor functions support both the query and saved_query rule types. The executor functions are already close to identical, so combining them will hopefully not be too difficult.

The result with this approach is that no migration will be needed, the siem.queryRule and siem.savedQueryRule rule types at the framework level become identical except for the alertTypeId (and support both query and saved_query as the type param in the rule), and users can freely edit rules to switch between query and saved_query - however once created, at the framework level a rule will always remain either siem.queryRule or siem.savedQueryRule. The framework level difference is not exposed to users through either the Detection Rules API or UI.

[vitaliidm]

@marshallmain
A few observations while working on it

1. It's possible to save rule with type query with saved_id property and rule with type saved_query with query , filters property. Would it make sense do not store saved_id if rule has query type, and do not store query, filters if rule has saved_query type. Are there any concerns with this approach? Of course, we are not going to change schema: as it could be a breaking change, but only save these properties depending on a rule type. I think, it would make cleaner saved rules, as it would strip them from unnecessary properties, thus removing confusion which field is used in which circumstances (saved_id and query)

2. I can see, other rule types are saved with saved_id field as well, but I don't see any use of it anywhere in executors. Is there any reasons of adding saved_id to other rule types?

[marshallmain]

Would it make sense do not store saved_id if rule has query type, and do not store query, filters if rule has saved_query type

I think that would be good. Eventually we may want to look at making that breaking change as well and removing the unused fields from the schema entirely, but it's not an immediate concern.

I can see, other rule types are saved with saved_id field as well, but I don't see any use of it anywhere in executors. Is there any reasons of adding saved_id to other rule types?

I don't think there's any reason as far as how the rules actually execute - the saved_id should only be used for saved_query rules. It's included in the schema though because prior to the conversion to separate schemas for each rule type, the schema allowed saved_id and all the other rule type specific fields to be set for any rule type, since all type specific fields were just optional in the combined schema. I don't remember why saved_id specifically was added to each schema - maybe it had the highest chance of actually being set on other rule types, and we wanted to ensure that rules that set saved_id would still validate?

In any case, it's basically a holdover from the old combined schema that wasn't strict enough about preventing irrelevant fields from being set, so we continue to allow it until we have time to justify and make that breaking change.

[vitaliidm]

@yiyangliu9286 , @jethr0null

Currently, on rule details page, when rule has saved query, details section look like this

Query and filters, which displayed there are the ones saved within a rule, and not used during the rule execution

With changes, proposed in this ticket, these values will be fetched from saved query. And always will be up to date.
Would it make sense to change its their titles to reflect those change:

Or, should we stick to existing UI, and just show fields as Custom query and Filters?

[vitaliidm]

Per discussion at Advanced Correlation Working Group, labels for saved query will be changed as displayed on this screenshot