Drain does not check for Galera cluster health

[CAFxX]

The drain script currently does not check whether the cluster is healthy before stopping a node:

cf-mysql-release/jobs/mysql/templates/drain.sh

Lines 1 to 6 in f78967b

	#!/bin/bash -e
	/var/vcap/packages/mariadb/support-files/mysql.server stop --pid-file=/var/vcap/sys/run/mysql/mysql.pid > /dev/null
	return_code=$?
	echo 0
	exit ${return_code}

We think it would be safer to check if all cluster nodes are online and in sync before asking the local node to shutdown.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/157141182

The labels on this github issue will be updated when the story is started.

[ctaymor]

@CAFxX If the drain script can only succeed when the cluster is healthy, that would prevent anyone from bosh stopping an unhealthy cluster.

We can see how this change would be valuable in some cases, but we're concerned it would be harmful in others. We recommend using the proxy dashboard api to check if the cluster is healthy before doing a deploy.

[CAFxX]

@ctaymor isn't that why bosh deploy and bosh recreate both have a --skip-drain option? If we applied the logic that the operator had to check manually on a job-specific dashboard before running deploy wouldn't cf-deployment become unusable pretty quickly? Also the bosh docs seem to agree that's the whole point of drain scripts (emphasis mine):

This script allows the job to clean up and get into a state where it can be safely stopped.

I think the safer option is to check for obvious signs of the cluster being unhealthy and allow the operator to force the deployment if they think it's the correct thing to do in their case.

[ctaymor]

@CAFxX Yeah, that's a fair point about --skip-drain.

We'd be happy to accept a PR.

[CAFxX]

@ctaymor ok, just a question: is there currently any specific requirement requiring the node to be stopped during drain (instead of during monit stop)?

[ndhanushkodi]

Closing due to open PR to resolve: #209