KV Ingest Processor splitting on whitespace in message

[Evesy]

I'm trying to parse some logs in the logfmt format using the KV processor, example log line below:

time="2018-07-04T09:36:25Z" level=info msg="Schedule is not due, skipping" logSource="pkg/controller/schedule_controller.go:325" nextRunTime="2018-07-05 01:00:00 +0000 UTC" schedule=daily

The processor being used is:

"kv": {
  "field": "message",
  "field_split": " ",
  "value_split": "="
}

Due to the whitespaces in the msg key the logs are being incorrectly split midway through the message, resulting in the msg field being "Schedule

There's a similar issue open for the Logstash equivalent plugin here: logstash-plugins/logstash-filter-kv#9

It's my understanding that quoted values as above should be treated and parsed as a single value, and the quotes should then be stripped from the resulting field value.

If this isn't the case it would be good to expose these options, as it makes the kv processor a lot less versatile without.

Cheers,
Mike

[dliappis]

Reading the docs about the kv processor I think what's happening here is the expected behavior as:

• field_split: Regex pattern to use for splitting key-value pairs
• value_split: Regex pattern to use for splitting the key from the value within a key-value pair

The KV processor maybe too simple for what you need to achieve, given that some of your values are enclosed in double quotes, if they contain spaces and some not.

If the order of your fields doesn't change, perhaps the grok processor could be of better use here, as for example you can use the quoted string pattern to match values enclosed in double quotes.

[elasticmachine]

Pinging @elastic/es-core-infra

[Evesy]

Thanks for the response @dliappis

Unfortunately, the fields aren't consistent across all applications, they're just all using the logfmt format. It would be great if the kv filter was to be extended a bit further to match similar capabilities as the Logstash one.

[original-brownbear]

@jakelandis I'll handle this unless you already started somehow ?:)

[jakelandis]

@original-brownbear - all yours