Stop supporting older versions of Kafka - plus improve user and maintainer experience

[jordansissel]

Supporting multiple versions of Kafka is too much of a burden, so let us just focus on one and be specific about two things:

1. Explicitly state the version range of Kafka the output supports.

Maintaining multiple major branches of this plugin has demonstrably been bad for users and bad for us as maintainers. It's confusing to users, and it's frustrating for maintainers when we have to invest time in figuring out what can and cannot be backported.

Proposal:

1. Simplify user experience: Remove the compatibility matrix and replace it with a version range.
2. Simplify maintenance: Only maintain one branch of Kafka plugins.

2. Grace period after Kafka upstream releases

Problem: Literally day-1 of every Kafka release we have 2-3 users pinging us "When will Kafka X.Y.Z be supported?" and frankly this is setting everyone up for a constant stream of irritation and annoyance. It would be better for everyone if users did not feel the urgency in pinging us every time Kafka is released and also if we did not feel like we are constantly chasing a never-ending stream of releases.

Proposal:

1. Set user expectations by documenting a grace period that begins with each Kafka change, during which, users should not be pinging us about new version support.
2. For during this grace period, one of the maintainers will test the new release of Kafka and update the documentation if it is supported.

---

Implementation:

The new documentation will read this way:

This plugin uses Kafka Client 1.0.0. For broker compatibility, see the official https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Ma
trix[Kafka compatibility reference].

If you're using a plugin version that was released after {version}, see the https://www.elastic.co/guide/en/logstash/master/plugins-inputs-kafka.h
tml[latest plugin documentation] for updated information about Kafka compatibility. If you require features not yet available in this plugin (incl
uding client version upgrades), please file an issue with details about what you need..

(With correct kafka client version details)

This should solve the problems mentioned above.

[jordansissel]

/cc @andrewvc @acchen97

[jordansissel]

/cc @ppf2

[jordansissel]

I'm going to move forward with this and add a note in the README of this repo and somewhere appropriate in the docs for both kafka input/output plugins.

Rough specification:

• Grace period: In order to set user expectations and reduce energy spent in when-will-it-be-released communication loops, we will set a 6-week grace period after every Kafka minor release during which any inquiries about release scheduling will be answered with a link to this policy (which will be documented on the kafka plugin documentation)
• Drop Kafka 0.8 support.
• Support Kafka 0.10 and 0.11 with the Kafka 0.11 client. Any integration problems between Kafka .11 client and Kafka 0.10-0.11 will be directed to the Kafka project.
• Maintenance on Kafka 0.9-broker branch will terminate in 6 months from now (April 2018)

Impact:

• Only support the latest Kafka client library which transitively selects which Kafka broker versions are supported

---

Steps:

• Remove Kafka 0.8 broker from the matrix
• Document grace period in both Kafka plugins.
• Document Kafka 0.9 support EOL will be April 2018 (6 months from today).

Rationale on dropping support:

• Kafka 0.9 was released November 2015 (2 years ago). It's time to upgrade.

[jordansissel]

As for compatibility, this relies on the Kafka java client documentation accuracy, and Kafka says this:

Starting with version 0.10.2, Java clients (producer and consumer) have acquired the ability to communicate with older brokers. Version 0.11.0 clients can talk to version 0.10.0 or newer brokers

This means we can have kafka output use the latest (0.11.0.0) client and it'll successfully talk with 0.10.0.0 brokers. If it breaks, we can delegate to the Kafka project for debugging/troubleshooting.

[acchen97]

@jordansissel thanks for putting thought into this, I like the overall goals.

I'm good with just maintaining one branch going forward and defining compatibility with a client version range. We'll still need to support 0.9 for some time; I'm not too worried about 0.8 and we can figure out a deprecation path for it. For some context, HDP 2.4 shipped Mar 2016 and bundled Kafka 0.9, but HDP 2.5 and 2.6 (latest) both bundle 0.10.x, so 0.9 should hopefully be nearing the end of its cycle there as well. I suspect Hadoop vendors still support a majority of Kafka deployments out there today.

For handling upstream releases, are you proposing we move to 2 month time based releases for the Kafka input/output plugins? We could also instead have a 2 month grace period per upstream Kafka minor release where we wouldn't be committing to a specific release cadence, and if we don't have compatibility after 2 months they can still file issues/ERs for it.

[jordansissel]

@acchen97 good feedback! THank you :)

I've updated my rough proposal in the comment: #156 (comment)

Changes:

• Change "release cycle" to "grace period"
• Note that Kafka 0.9 is still supported for at least 6 months.
• Note that Kafka 0.8 is not supported

[acchen97]

@jordansissel I'm good with the changes, thanks.

Let's define the grace period as 6 weeks after each Kafka minor release. Also, it'd be a good idea to document this grace period both in the input/output READMEs and plugin documentation.

[jordansissel]

^^ minor point -- we only document the client version we use of kafka, not the broker. broker-client compatibility is up to Confluent.

This has historically meant that, based on Confluent's documentation on Kafka, client version X works with broker X+1. We don't need to answer questions anymore about broker compatibility, because Confluent answers this already.

[jordansissel]

The initial implementation of this doc change is now merged: logstash-plugins/logstash-input-kafka#237. Additional revision made here: logstash-plugins/logstash-input-kafka#244

Next, I need to update the docs for all supported versions of logstash on both input and output kafka plugins.

The update (for whatever versions) will be changed to be accurate with respect to the Kafka client version. Otherwise the content of the doc change will be the same.

Task table

Each entry shows the logstash version and the corresponding kafka plugin version. The docs for each of these Logstash minor branches (5.0, 5.1, etc) need to be updated.

Branch update tasks:

type	branch	documentation	released
input	master (8.x)	✔️	❌
input	7.x	pr	❌
input	6.x	pr	❌
input	5.x	pr	❌
input	4.x	pr	❌
output	master (7.x)	pr	❌
output	6.x	pr	❌
output	5.x	pr	❌
output	4.x	pr	❌

Logstash docs update tasks:

Logstash	Input	Output	Kafka Client	Status
2.4	4.x	4.x	0.9.0.1	docs pr
5.0.2	5.0.6	5.0.4	0.10.0.1	docs pr
5.1.2	5.1.0	5.1.1	0.10.0.1	docs pr
5.2.2	5.1.6	5.1.5	0.10.0.1	docs pr
5.3.3	5.1.7	5.1.6	0.10.0.1	docs pr
5.4.3	5.1.7	5.1.6	0.10.0.1	docs pr
5.5.3	5.1.10	5.1.9	0.10.0.1	docs pr
5.6.4	5.1.11	5.1.11	0.10.0.1	docs pr
6.0.0	8.0.2	7.0.4	0.11.0.0	docs pr
6.1.0	8.0.2	7.0.4	0.11.0.0	docs pr
master	master	master	1.0.0	❌

(*) The table above was built by hand looking at Logstash releases -> input/output plugin versions, then looking at rubygems.org for each of these plugins and noting the version of the jar requirement for the org.apache.kafka:kafka-clients dependency.

(Some of these may not require any changes if the version of the plugin is the same across Logstash releases)