plugin 5.1.0: Bootstrap broker [hostname] disconnected error with SSL.

[lin-zhao]

Getting this error after upgrading to 5.1.0 plugin. 5.0.4 plugin works well.

• Version:
  logstash: 5.0.1
  kafka plugin: 5.1.0
  kafka: kafka_2.11-0.10.0.1
• Operating System:
  CentOS 7
• Config File (if you have sensitive info, please remove it):
  logstash:

input { stdin {
  }
}
output { kafka {
  codec => json_lines
  topic_id => "lms.kafka.topic"
  bootstrap_servers => "10.10.2.77:9093"
  ssl => true
  ssl_truststore_location => "/truststore.jks"
  ssl_truststore_password => "password"
}}

kafka:

listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9093
advertised.listeners=PLAINTEXT://10.10.2.77:9092,SSL://10.10.2.77:9093
security.protocol=SSL
ssl.keystore.location=/opt/kafka/tls.jks
ssl.keystore.password=some_passwd
ssl.truststore.location=/opt/kafka/tls.jks
ssl.truststore.password=some_passwd

• Sample Data:
  echo "hello world" | bin/logstash -f logstash-file.yml
• Steps to Reproduce:
  Use above config, get repeated error:

[2016-11-22T23:04:35,017][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker 10.10.2.77:9093 disconnected
[2016-11-22T23:04:35,474][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker 10.10.2.77:9093 disconnected
[2016-11-22T23:04:35,951][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker 10.10.2.77:9093 disconnected
[2016-11-22T23:04:36,430][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker 10.10.2.77:9093 disconnected
[2016-11-22T23:04:36,934][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker 10.10.2.77:9093 disconnected

[lin-zhao]

More logstash log:

[2016-11-22T23:04:33,965][INFO ][org.apache.kafka.clients.producer.ProducerConfig] ProducerConfig values: 
	metric.reporters = []
	metadata.max.age.ms = 300000
	reconnect.backoff.ms = 10
	sasl.kerberos.ticket.renew.window.factor = 0.8
	bootstrap.servers = [10.10.2.77:9093]
	ssl.keystore.type = JKS
	sasl.mechanism = GSSAPI
	max.block.ms = 60000
	interceptor.classes = null
	ssl.truststore.password = [hidden]
	client.id = producer-1
	ssl.endpoint.identification.algorithm = null
	request.timeout.ms = 30000
	acks = 1
	receive.buffer.bytes = 32768
	ssl.truststore.type = JKS
	retries = 0
	ssl.truststore.location = /exassl/kafka-master1.jks
	ssl.keystore.password = null
	send.buffer.bytes = 131072
	compression.type = none
	metadata.fetch.timeout.ms = 60000
	retry.backoff.ms = 100
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	buffer.memory = 33554432
	timeout.ms = 30000
	key.serializer = class org.apache.kafka.common.serialization.StringSerializer
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	ssl.trustmanager.algorithm = PKIX
	block.on.buffer.full = false
	ssl.key.password = null
	sasl.kerberos.min.time.before.relogin = 60000
	connections.max.idle.ms = 540000
	max.in.flight.requests.per.connection = 5
	metrics.num.samples = 2
	ssl.protocol = TLS
	ssl.provider = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	batch.size = 16384
	ssl.keystore.location = null
	ssl.cipher.suites = null
	security.protocol = PLAINTEXT
	max.request.size = 1048576
	value.serializer = class org.apache.kafka.common.serialization.StringSerializer
	ssl.keymanager.algorithm = SunX509
	metrics.sample.window.ms = 30000
	partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
	linger.ms = 0

[2016-11-22T23:04:33,968][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 0.10.0.1
[2016-11-22T23:04:33,968][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : a7a17cdec9eaa6c5
[2016-11-22T23:04:34,039][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2016-11-22T23:04:34,051][INFO ][logstash.pipeline        ] Pipeline main started
[2016-11-22T23:04:34,143][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2016-11-22T23:04:35,017][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker 10.10.2.77:9093 disconnected
[2016-11-22T23:04:35,474][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker 10.10.2.77:9093 disconnected
[2016-11-22T23:04:35,951][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker 10.10.2.77:9093 disconnected
[2016-11-22T23:04:36,430][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker 10.10.2.77:9093 disconnected

[gquintana]

Sadly, the ssl => true doesn't set security.protocol=SSL anymore.
You should replace, this line by security_protocol=>"SSL" in output 5.1

[Nishanthk3]

@gquintana I have included this security.protocol=SSL in the broker config's and I still have this issue,
Can you help me figure this issue out.
Thank You.

[gquintana]

@Nishanthk3 Get the producer config from logs (as @lin-zhao did above) and check security.protocol value.

[Nishanthk3]

@gquintana I don't see the setting security.protocol at-all, even though I set that value in the broker configuration.

[callmkarthik]

Clarification: It is Producer Config and not broker Config