Unexpected end-of-input in field name

[kenyhs]

When I configure Palo Alto to send netflow packets, this error occurs:

2017-08-18T18:20:18.938-03:00 ERROR [NetFlowV9TemplateCache] Couldn't load template cache from disk com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input in field name at [Source: /tmp/netflow-templates.json; line: 1, column: 16001] at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:483) ~[graylog.jar:?] at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:2020) ~[graylog.jar:?] at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName(UTF8StreamJsonParser.java:1921) ~[graylog.jar:?] at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1705) ~[graylog.jar:?] at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:776) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:144) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBind(MapDeserializer.java:458) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:365) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3814) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2775) ~[graylog.jar:?] at org.graylog.plugins.netflow.v9.NetFlowV9TemplateCache.loadCache(NetFlowV9TemplateCache.java:71) [graylog-plugin-netflow-2.3.0-rc.4.jar:?] at org.graylog.plugins.netflow.v9.NetFlowV9TemplateCache.<init>(NetFlowV9TemplateCache.java:62) [graylog-plugin-netflow-2.3.0-rc.4.jar:?] at org.graylog.plugins.netflow.v9.NetFlowV9TemplateCache.<init>(NetFlowV9TemplateCache.java:48) [graylog-plugin-netflow-2.3.0-rc.4.jar:?] at org.graylog.plugins.netflow.codecs.NetFlowCodec.<init>(NetFlowCodec.java:85) [graylog-plugin-netflow-2.3.0-rc.4.jar:?] at org.graylog.plugins.netflow.codecs.NetFlowCodec$$FastClassByGuice$$5770acd4.newInstance(<generated>) [graylog-plugin-netflow-2.3.0-rc.4.jar:?] at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) [graylog.jar:?] at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) [graylog.jar:?] at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) [graylog.jar:?] at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) [graylog.jar:?] at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [graylog.jar:?] at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [graylog.jar:?] at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [graylog.jar:?] at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:776) [graylog.jar:?] at com.sun.proxy.$Proxy118.create(Unknown Source) [?:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:132) [graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:87) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

Below is the template received

netflow-templates.txt

[kroepke]

Thanks for the bug report @kenyhs
Unfortunately the error message and the template file you linked to don't match up (the linked file is shorter than the error message implies).

The next version of the plugin will no longer use the disk-based cache and I assume this error will be solved, too.
I'll reference this bug report when merging and releasing the new version, please watch for the notification.

[michelealbrigo]

Hi. I experienced issues in a similar context: Graylog 2.3.0, plugin version 2.3.0-rc4, Netflow v9 and Palo Alto Firewalls.
After a short (volume-dependent, I think) time something crashes and the whole netlog server stops processing messages, even the ordinary syslog ones, leading to buffer and journal exhaustion if left unattended. Turning off the whole netflow part (stopping input, stopping stream and stopping the firewalls from sending it) and restarting the graylog-server resumes processing and deals with everything in queue. The same server works fine without netflow (2-3k messages/second, 0% on buffers, no indexing errors, low journal utilization, etc., everything oversized for the load).

I am attaching a snippet of the errors I found in my logs, they seem all to be of these two types.
The first one I get after starting the server is the "ERROR [NetFlowCodec] Error parsing NetFlow packet" one.

graylog-netflow.txt

Hope this can be useful to solve the issue. Let me know if you need any further data.

[kenyhs]

I experienced same issue of @michelealbrigo. After a short time Graylog stops processing every messages.