-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Closed
Labels
Description
Original comment by @droberts195:
If we're going to introduce completely new job types in the future, we need to change the way unknown job/datafeed cluster state is validated.
While trying to add categorizer jobs, which are quite similar to anomaly_detector jobs, I ran into the following problem:
- Logically, a
categorizerjob should have nodetectors - But the
AnalysisConfigclass requiresdetectors - There are two possible solutions that seem reasonable at first glance:
- Have
categorizerjobs have acategorization_configinstead ofanalysis_config - Change
analysis_configso thatdetectorsis not required if thejob_typeiscategorizer
- Have
- Unfortunately neither of these works:
- Old nodes will ignore
categorization_configwhen parsing metadata, but then error becauseJobrequires ananalysis_config - Old nodes will not tolerate an
analysis_configwith no detectors
- Old nodes will ignore
- This results in the messy solution that
categorizerjobs will have to have ananalysis_configthat includes unnecessary fields - new nodes will ignore these fields and mask them when printing the config in REST responses, but old nodes will show the unnecessary bits
I think the only long term solution that allows the necessary degree of extensibility is to hold Jobs as arbitrary Map<String, Object> or BytesReference when parsing from cluster state, and only interpret what's in the Map or BytesReference if the job_type is understood. This is pretty much how index settings work.