RFE: Failover to another DC for service discovery list

[jasonmcintosh]

Looking at the ConsulServerList code, it wouldn't be super hard to enable a failover on a refresh of datacenter to fallback to a remote dc. Thinking something like this in
Add application property:
"spring.cloud.consul.discovery.fallbackDatacenters=us-west-1"

org.springframework.cloud.consul.discovery.ConsulServerList:

	private List<ConsulServer> getServers() {
		if (this.client == null) {
			return Collections.emptyList();
		}
		String tag = getTag(); // null is ok
		Response<List<HealthService>> response = this.client.getHealthServices(
				this.serviceId, tag, this.properties.isQueryPassing(),
				createQueryParamsForClientRequest(), this.properties.getAclToken());
		if (response.getValue() == null || response.getValue().isEmpty()) {
			return transformResponse(fallBackToOtherDCOrEmpty());
		}
		return transformResponse(response.getValue());
	}
       private List<HealthService> fallBackToOtherDCOrEmpty() {
             Response<List<HealthService>> response = this.client.getHealthServices(
				this.serviceId, tag, this.properties.isQueryPassing(),
				new QueryParams(this.properties.getFallbackDatacenters()[0], this.properties.getAclToken())
             return response.getValue() == null ? Collections.emtpyList() ? esponse.getValue();
       }

PURE thought process at this - but would enable you to configure a fallbackDatacenter property which would allow if a service had no health hosts, to automatically go to a fallback datacenter. I could see it being a recursive array through the available fallbacks. Another option would be to use prepared Queries in consul vs. trying to write it here, but would have to change how healthy services were discovered.

Thoughts? Other ways to implement this?

[jasonmcintosh]

Did some more digging. IF we follow Consul's implementation:
https://www.consul.io/docs/guides/geo-failover.html

What we need to do is use the query API. SO possible an alternative to calling client.getHealthServices would be instead:

@Configuration
@Component
class ConsulQueryDiscoveryClient extends ConsulDiscoveryClient {
	public ConsulQueryDiscoveryClient(ConsulClient client, ConsulDiscoveryProperties properties) {
		super(client, properties);
	}

	@Autowired
	ConsulClient client;

	@Override
	public List<ServiceInstance> getInstances(String serviceId) {
		List<ServiceInstance> servers = new ArrayList<ServiceInstance>();
		Response<QueryExecution> results = client.executePreparedQuery("", QueryParams.DEFAULT);
		List<QueryNode> nodes = results.getValue().getNodes();
		for (QueryNode queryNode : nodes) {
			Node node = queryNode.getNode();
			String[] address = node.getAddress().split(":");
			servers.add(new DefaultServiceInstance(serviceId, address[0], Integer.parseInt(address[1]), Boolean.parseBoolean(node.getMeta().get("secure"))));
		}
		return servers;
	}
}

Assuming you're using a query that only returns healthy servers, with fallback, etc. This SHOULD take care of it. Maybe trigger this behavior in case of a failure? Etc.?

[spencergibb]

PRs welcome.

[jasonmcintosh]

SO Worked around this somewhat with custom code for now... will work on this if I ever get time:

package mycode;

import java.util.ArrayList;
import java.util.List;

import org.springframework.cloud.consul.discovery.ConsulDiscoveryProperties;

import com.ecwid.consul.v1.ConsulClient;
import com.ecwid.consul.v1.QueryParams;
import com.ecwid.consul.v1.Response;
import com.ecwid.consul.v1.query.model.QueryExecution;
import com.ecwid.consul.v1.query.model.QueryNode;
import com.ecwid.consul.v1.query.model.QueryNode.Node;
import com.netflix.client.config.IClientConfig;
import com.netflix.loadbalancer.AbstractServerList;
import com.netflix.loadbalancer.Server;

public class ConsulPreparedQueryServerList extends AbstractServerList<Server> {

    private final ConsulClient client;
    private final ConsulDiscoveryProperties properties;

    private String serviceId;

    public ConsulPreparedQueryServerList(ConsulClient client, ConsulDiscoveryProperties properties) {
        this.client = client;
        this.properties = properties;
    }

    protected ConsulClient getClient() {
        return client;
    }

    protected ConsulDiscoveryProperties getProperties() {
        return properties;
    }

    protected String getServiceId() {
        return serviceId;
    }

    @Override
    public void initWithNiwsConfig(IClientConfig clientConfig) {
        this.serviceId = clientConfig.getClientName();
    }

    @Override
    public List<Server> getInitialListOfServers() {
        return getServers();
    }

    private List<Server> getServers() {
        System.err.println("Finding services for " + serviceId);
        List<Server> servers = new ArrayList<Server>();
        Response<QueryExecution> results = client.executePreparedQuery(serviceId, QueryParams.DEFAULT);
        List<QueryNode> nodes = results.getValue().getNodes();
        for (QueryNode queryNode : nodes) {
            Node node = queryNode.getNode();
            servers.add(new Server(node.getMeta().containsKey("secure") ? "https" : "http", node.getNode(), queryNode.getService().getPort()));
        }
        return servers;
    }


    @Override
    public List<Server> getUpdatedListOfServers() {
        return getServers();
    }
}

Enable ribbon to use this:

public class ConsulPreparedQueryRibbonClientConfig extends ConsulRibbonClientConfiguration {
    @Autowired
    private ConsulClient client;

    @Bean
    public ServerList<?> ribbonServerList(IClientConfig config, ConsulDiscoveryProperties properties) {
        ConsulPreparedQueryServerList serverList = new ConsulPreparedQueryServerList(client, properties);
        serverList.initWithNiwsConfig(config);
        return serverList;
    }
}
@RibbonClients(defaultConfiguration=ConsulPreparedQueryRibbonClientConfig.class)
@SpringBootApplication
@EnableCircuitBreaker
@EnableDiscoveryClient
public class Application ....

[spencergibb]

Closing as ribbon is in maintanence mode