Log 'RecordTooLargeException' errors as warn #179
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jsvd
reviewed
Jun 4, 2018
lib/logstash/outputs/kafka.rb
Outdated
| futures.each_with_index do |future, i| | ||
| begin | ||
| result = future.get() | ||
| rescue org.apache.kafka.common.errors.RecordTooLargeException => x |
Member
There was a problem hiding this comment.
@robbavey there's another PR that catches this exception during producer.send
https://github.com/logstash-plugins/logstash-output-kafka/pull/191/files#diff-c6a637e53249d2c5a0ee842b0bc19e82R265 instead of this PR's future.get
I believe this one makes sense as the other operation will only return the future so no exception should be raised, right?
Member
Author
There was a problem hiding this comment.
Yes, this PR ups the visibility of an error that was already being logged at debug in the following rescue => e block. I'm kind of tempted to change this PR to just increase the log level of the blanket rescue to increase the visibility of any error caught here, and reduce if the noise levels get too high.
Member
There was a problem hiding this comment.
Agreed, we want to know whenever an error happens, otherwise logstash will either be silently retrying things or dropping data (Both terrible).
Log errors encountered when sending messages to Kafka at warn, rather than debug, to improve the visibility of errors such as 'RecordTooLargeException.' Fixes logstash-plugins#177
robbavey
added a commit
to robbavey/logstash-output-kafka
that referenced
this pull request
Feb 11, 2019
Log errors encountered when sending messages to Kafka at warn, rather than debug, to improve the visibility of errors such as 'RecordTooLargeException.' Fixes logstash-plugins#177 Fixes logstash-plugins#179
robbavey
added a commit
to robbavey/logstash-output-kafka
that referenced
this pull request
Feb 11, 2019
Fixed incorrect millisecond to second conversion for retry_backoff_ms logstash-plugins#216 Fixed unnecessary sleep after exhausted retries logstash-plugins#166 Changed Kafka send errors to log as warn logstash-plugins#179
robbavey
added a commit
to robbavey/logstash-output-kafka
that referenced
this pull request
Feb 11, 2019
Fixed incorrect millisecond to second conversion for retry_backoff_ms logstash-plugins#216 Fixed unnecessary sleep after exhausted retries logstash-plugins#166 Changed Kafka send errors to log as warn logstash-plugins#179
robbavey
added a commit
to robbavey/logstash-output-kafka
that referenced
this pull request
Feb 11, 2019
Fixed incorrect millisecond to second conversion for retry_backoff_ms logstash-plugins#216 Fixed unnecessary sleep after exhausted retries logstash-plugins#166 Changed Kafka send errors to log as warn logstash-plugins#179
robbavey
added a commit
to robbavey/logstash-output-kafka
that referenced
this pull request
Feb 11, 2019
Fixed unnecessary sleep after exhausted retries logstash-plugins#166 Changed Kafka send errors to log as warn logstash-plugins#179
robbavey
added a commit
to robbavey/logstash-output-kafka
that referenced
this pull request
Feb 11, 2019
Fixed unnecessary sleep after exhausted retries logstash-plugins#166 Changed Kafka send errors to log as warn logstash-plugins#179
robbavey
added a commit
to robbavey/logstash-output-kafka
that referenced
this pull request
Feb 11, 2019
Fixed unnecessary sleep after exhausted retries logstash-plugins#166 Changed Kafka send errors to log as warn logstash-plugins#179
robbavey
added a commit
to robbavey/logstash-output-kafka
that referenced
this pull request
Feb 11, 2019
Fixed unnecessary sleep after exhausted retries logstash-plugins#166 Changed Kafka send errors to log as warn logstash-plugins#179
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a common error, which cannot be fixed without changing Logstash setting
and should not be hidden as a debug log entry.
Fixes #177