LLClient: Support host selection #30523

nik9000 · 2018-05-10T21:27:28Z

Allows users of the Low Level REST client to specify which hosts a
request should be run on. They implement the NodeSelector interface
or reuse a built in selector like NOT_MASTER_ONLY to chose which nodes
are valid. Using it looks like:

Request request = new Request("POST", "/foo/_search");
RequestOptions options = request.getOptions().toBuilder();
options.setNodeSelector(NodeSelector.NOT_MASTER_ONLY);
request.setOptions(options);
...

This introduces a new Node object which contains a HttpHost and the
metadata about the host. At this point that metadata is just version
and roles but I plan to add node attributes in a followup. The
canonical way to get this metadata is to use the Sniffer to pull
the information from the Elasticsearch cluster.

I've marked this as "breaking-java" because it breaks custom
implementations of HostsSniffer by renaming the interface to
NodesSniffer and by changing it from returning a List<HttpHost> to a
List<Node>. It shouldn't break anyone else though.

Because we expect to find it useful, this also implements host_selector
support to do statements in the yaml tests. Using it looks a little
like:

---
"example test":
  - skip:
      features: host_selector
  - do:
      host_selector:
        version: " - 7.0.0" # same syntax as skip
      apiname:
        something: true

The do section parses the version string into a host selector that
uses the same version comparison logic as the skip section. When the
do section is executed it passed the off to the RestClient, using
the ElasticsearchHostsSniffer to sniff the required metadata.

The idea is to use this in mixed version tests to target a specific
version of Elasticsearch so we can be sure about the deprecation
logging though we don't currently have any examples that need it. We do,
however, have at least one open pull request that requires something
like this to properly test it.

Closes #21888

Allows users of the Low Level REST client to specify which hosts a request should be run on. They implement the `NodeSelector` interface or reuse a built in selector like `NOT_MASTER_ONLY` to chose which nodes are valid. Using it looks like: ``` Request request = new Request("POST", "/foo/_search"); request.setNodeSelector(NodeSelector.NOT_MASTER_ONLY); ... ``` This introduces a new `Node` object which contains a `HttpHost` and the metadata about the host. At this point that metadata is just `version` and `roles` but I plan to add node attributes in a followup. The canonical way to **get** this metadata is to use the `Sniffer` to pull the information from the Elasticsearch cluster. I've marked this as "breaking-java" because it breaks custom implementations of `HostsSniffer` by renaming the interface to `NodesSniffer` and by changing it from returning a `List<HttpHost>` to a `List<Node>`. It *shouldn't* break anyone else though. Because we expect to find it useful, this also implements `host_selector` support to `do` statements in the yaml tests. Using it looks a little like: ``` --- "example test": - skip: features: host_selector - do: host_selector: version: " - 7.0.0" # same syntax as skip apiname: something: true ``` The `do` section parses the `version` string into a host selector that uses the same version comparison logic as the `skip` section. When the `do` section is executed it passed the off to the `RestClient`, using the `ElasticsearchHostsSniffer` to sniff the required metadata. The idea is to use this in mixed version tests to target a specific version of Elasticsearch so we can be sure about the deprecation logging though we don't currently have any examples that need it. We do, however, have at least one open pull request that requires something like this to properly test it. Closes elastic#21888 (kind of, it isn't in the high level client, but we'll do that in a followup)

elasticmachine · 2018-05-10T21:27:30Z

Pinging @elastic/es-core-infra

nik9000 · 2018-05-10T21:28:35Z

I've not marked this as review yet because I've just ported #29211 without addressing the last round of comments, but the general flow is in. @javanna if you'd like to have a look, enjoy! Otherwise, I'll port your last round of comments from #29211 and then mark it as review.

nik9000 · 2018-05-11T14:12:26Z

client/rest/src/main/java/org/elasticsearch/client/NodeSelector.java

+     * first and then running the "left" selector on the results of the "right"
+     * selector.
+     */
+    class Compose implements NodeSelector {


Transferring a comment by @javanna from #29211:

I am not sure that we still need this given that users can provide metadata together with the corresponding host. Can we remove this?

Transferring my reply:
We still use this in the yaml testing framework to look up metadata.

Replying to my reply:
I'm going to move this to the yaml testing framework. I buy the argument that we may not need this ever and in that case I don't want to make it and commit to supporting it. If folks file an issue about it we can bring it back over.

nik9000 · 2018-05-11T14:15:09Z

client/rest/src/main/java/org/elasticsearch/client/Node.java

+     * if we don't know what roles the node has.
+     */
+    private final Roles roles;
+


I'd like to add node attributes to this in a followup. Folks can tag the elasticsearch node with the rack/row/availability zone that it is in and then use a NodeSelector to target nodes in the same rack/row/availability zone as the application server. It is a traditional elasticsearch feature and it is pretty sweet but I don't want to add yet more to this already very large PR.

nik9000 · 2018-05-11T17:03:14Z

OK @javanna! This is ready for you to have a look when you have a chance!

javanna

I did a first round and left a few comments, I will go deeper in the next round.

javanna · 2018-05-14T12:50:27Z

client/rest/src/main/java/org/elasticsearch/client/NodeSelector.java

+     * An implementation that sorts list consistently will consistently send
+     * requests to s single node, overloading it. So implementations that
+     * reorder the list should take the original order into account
+     * <strong>somehow</strong>.


food for thoughts: If this is dangerous, shall we not document it, or not allow it? Is it useful to reorder nodes?

You argued strongly a few months ago to allow it, mostly on the grounds that something like rack awareness features would want to sort them like <on_the_rack>, <off_the_rack>.

oops. that's what happens. I forget things. I wouldn't say I argued strongly about ordering necessarily, I just wanted the selector to get a list of all the node candidates rather than a single node. That way it can decide what to do and act as a preference, meaning if there's not nodes in the rack, it's ok to go to an alive node in another rack rather than trying to revive a node. That can be achieved without allowing the selector to mess with ordering?

What about something like this:

diff --git a/client/rest/src/main/java/org/elasticsearch/client/NodeSelector.java b/client/rest/src/main/java/org/elasticsearch/client/NodeSelector.java index e9fbc9b6999..273cecac7ef 100644 --- a/client/rest/src/main/java/org/elasticsearch/client/NodeSelector.java +++ b/client/rest/src/main/java/org/elasticsearch/client/NodeSelector.java @@ -54,15 +54,26 @@ public interface NodeSelector { * selector approves of, in the order that the selector would prefer * to use them. */ - List<Node> select(List<Node> nodes); + Selector prepare(List<Node> nodes); + + interface Selector { + boolean select(Node node); + } /** * Selector that matches any node. */ NodeSelector ANY = new NodeSelector() { + private final Selector SELECTOR = new Selector() { + @Override + public boolean select(Node node) { + return true; + } + }; + @Override - public List<Node> select(List<Node> nodes) { - return nodes; + public Selector select(List<Node> nodes) { + return SELECTOR; } @Override @@ -77,16 +88,19 @@ public interface NodeSelector { * role. It does not reorder the nodes sent to it. */ NodeSelector NOT_MASTER_ONLY = new NodeSelector() { - @Override - public List<Node> select(List<Node> nodes) { - List<Node> subset = new ArrayList<>(nodes.size()); - for (Node node : nodes) { - if (node.getRoles() == null) continue; - if (false == node.getRoles().isMasterEligible() || node.getRoles().isData()) { - subset.add(node); + private final Selector SELECTOR = new Selector() { + @Override + public boolean select(Node node) { + if (node.getRoles() == null) { + return false; } + return false == node.getRoles().isMasterEligible() || node.getRoles().isData(); } - return subset; + }; + + @Override + public Selector select(List<Node> nodes) { + return SELECTOR; } @Override diff --git a/client/rest/src/main/java/org/elasticsearch/client/RestClient.java b/client/rest/src/main/java/org/elasticsearch/client/RestClient.java index 1e6a477e215..d3c613c5535 100644 --- a/client/rest/src/main/java/org/elasticsearch/client/RestClient.java +++ b/client/rest/src/main/java/org/elasticsearch/client/RestClient.java @@ -648,9 +648,14 @@ public class RestClient implements Closeable { */ // TODO this is going to send more requests to nodes right *after* a node that the selector removes Collections.rotate(livingNodes, lastNodeIndex.getAndIncrement()); - List<Node> selectedLivingNodes = nodeSelector.select(livingNodes); - if (false == selectedLivingNodes.isEmpty()) { - return selectedLivingNodes; + Selector selector = nodeSelector.selector(livingNodes); + for (Iterator<Node> itr = livingNodes.iterator(); itr.hasNet();) { + if (false == selector.select(itr.next())) { + itr.remove(); + } + } + if (false == livingNodes.isEmpty()) { + return livingNodes; } } @@ -677,7 +682,12 @@ public class RestClient implements Closeable { for (DeadNodeAndRevival n : deadNodes) { selectedDeadNodes.add(n.node); } - selectedDeadNodes = nodeSelector.select(selectedDeadNodes); + Selector selector = nodeSelector.selector(selectedDeadNodes); + for (Iterator<Node> itr = selectedDeadNodes.iterator(); itr.hasNet();) { + if (false == selector.select(itr.next())) { + itr.remove(); + } + }

The NodeSelector can still look at all the nodes before it makes a decision but it can't reorder.

I do not love that two interfaces need to be implemented, especially the select method that returns a Selector feels a bit of a weird API to me. What do you think of accepting and returning a Collection or a Set instead?

I think a Set is pretty heavy. What if I send it an Iterable? They can iterate it once to figure out everything in the list and then they can iterate it again to remove things. And we already have an Iterable on the call site so I don't need to build any more objects.

javanna · 2018-05-14T12:50:45Z

client/rest/src/main/java/org/elasticsearch/client/NodeSelector.java

+     * Implementations <strong>may</strong> reorder the list but they should
+     * be careful in doing so as the original order is important (see above).
+     * An implementation that sorts list consistently will consistently send
+     * requests to s single node, overloading it. So implementations that


s/s single node/a single node

javanna · 2018-05-14T12:52:52Z

client/rest/src/main/java/org/elasticsearch/client/RestClient.java

     * Returns a new {@link RestClientBuilder} to help with {@link RestClient} creation.
     * Creates a new builder instance and sets the hosts that the client will send requests to.
+     * <p>
+     * Prefer this to {@link #builder(Node...)} if you have metadata up front about the nodes.


Prefer this to {@link #builder(HttpHost...)} ?

javanna · 2018-05-14T12:57:34Z

client/rest/src/main/java/org/elasticsearch/client/RestClient.java

+            if (node == null) {
+                throw new IllegalArgumentException("node cannot be null");
+            }
+            authCache.put(node.getHost(), new BasicScheme());


we were previously deduplicating hosts, isn't that needed anymore?

javanna · 2018-05-14T13:14:05Z

client/rest/src/test/java/org/elasticsearch/client/DeadHostStateTests.java

-                assertThat(deadHostState.shallBeRetried(), is(true));
-            }
-        }
-    }


this test should be restored

javanna · 2018-05-14T13:15:34Z

client/rest/src/main/java/org/elasticsearch/client/DeadHostState.java

-        };
-
-        long nanoTime();
-    }


I suppose these changes are gone due to a bad merge. I think this was a good improvement and I don't see a replacement for it and the corresponding tests in this PR.

Kind of. In my old PR DeadHostState didn't have a shallBeRetried because it looks at the getDeadUntilNanos directly so it can sort all the dead nodes. That was part of the thing that allowed us to get rid of the looping. I pulled on the string from there and got here. I agree about the missing test - I need to rebuild it.

would be nice to use the fact that DeadHostState is now Comparable and the shallBeRetried method was convenient too I think.

I still think the same here :)

I think I see what you are getting at. We can keep shallbeRetried if we keep the DeadHostState and sort on it. That'd be fine.

javanna · 2018-05-14T13:17:48Z

client/rest/src/main/java/org/elasticsearch/client/RestClient.java

+        if (false == deadNodes.isEmpty()) {
+            Collections.sort(deadNodes, new Comparator<DeadNodeAndRevival>() {
+                @Override
+                public int compare(DeadNodeAndRevival lhs, DeadNodeAndRevival rhs) {


DeadHostState is already Comparable, can we rely on that here too?

I've been wondering about combining the Node into DeadHostState which I think would make this simpler. I'm kind of weary to do it as part of this already large change though.

I wouldn't want to introduce more changes in this PR, but I would like to restore some of the previous stuff around comparable that was recently introduced.

javanna · 2018-05-14T13:18:05Z

client/rest/src/main/java/org/elasticsearch/client/RestClient.java

+                livingNodes.add(node);
+                continue;
+            }
+            deadNodes.add(new DeadNodeAndRevival(node, nanosUntilRevival));


maybe call the class DeadNodeState ?

I don't want to clash with DeadHostState. They sort of do different things too.

javanna · 2018-05-14T13:38:10Z

client/rest/src/main/java/org/elasticsearch/client/RestClient.java

+             * the selector is ok with any over the living nodes then use
+             * them for the request.
+             */
+            // TODO this is going to send more requests to nodes right *after* a node that the selector removes


Very good point! Can we have a test for this? And think about how to fix it next? :)

On one hand it is very problematic to have a different node selector per request (I am even wondering now why we are exposing that, I forgot :) ), as if the selector is not consistent, we can't be either when round-robining?

On the other hand, now we mess up even in case the node selector does the same thing over and over, which is bad.

Maybe we should simply take out the ordering aspect here then and perform the rotate after calling the node selector, and make it explicit that the node selector should only select and never rely on ordering? Maybe have a set instead that makes it even more explicit?

Another unrelated problem is that we have an AtomicInteger counter... that's going to overflow at some point. Not sure but maybe while we revise this rotate problem we also have a chance to fix the counter limitation.

I'd like to revisit this in a followup.

this one makes me very nervous. As we discussed yesterday, couldn't we just move the rotate after the select and be done with it? Or do you foresee more changes needed?

Yeah, I think we can indeed put the rotate after the select now that we've given up on the order meaning anything to folks that implement NodeSelector.

javanna · 2018-05-14T13:48:16Z

client/rest/src/main/java/org/elasticsearch/client/Node.java

+            && Objects.equals(boundHosts, other.boundHosts)
+            && Objects.equals(version, other.version)
+            && Objects.equals(name, other.name)
+            && Objects.equals(roles, other.roles);


this may sound controversial but in my mind the answer to the "when are two nodes equal?" question is "when they have the same host and port".

I think things that want to compare nodes based on the host and port should do lhs.getHost().equals(rhs.getHost()). I think most of the time they really do want to compare by host like you say, but I think we can be explicit those time. Mostly it just really confuses me when things ignore parts of themselves in equality checks....

shall we then drop these methods in this case, I think it confuses me to have equals working this way if most of the times we want to compare the hosts :)

sorry I just suggested that we re-implement this. Do what you prefer, I am fine with both at this point given that depending on the day I suggest to have equals or move it to tests :)

javanna · 2018-05-14T13:50:41Z

test/framework/src/main/java/org/elasticsearch/test/rest/yaml/ESClientYamlSuiteTestCase.java

        return true;
    }
+
+        /**


adjust the indentation?

javanna · 2018-05-14T13:54:14Z

test/framework/src/main/java/org/elasticsearch/test/rest/yaml/ESClientYamlSuiteTestCase.java

+     *     thread safe but it doesn't have to be for our tests.
+     * </ul>
+     */
+    private void sniffHostMetadata(RestClient client) throws IOException {


How about removing the need for this and for boundHosts (I think!) by enabling sniffing in our yaml tests? wouldn't that be a good way to test it too?

I think that'd be safe, but it feels like it is a larger change than this. I'll have to think about it some more.

larger change in terms of lines of code? Or it just does not belong here? My idea was to reduce the size of the PR to just enabling sniffing in the yaml test suite. But maybe I am not seeing where sniffing may cause problems.

nik9000 · 2018-06-01T14:50:27Z

I've merged master to this and I'm going to apply feedback now and try write some docs. At least that is my plan.

Stll a few comments left to get through.

javanna

this looks great thanks @nik9000 for being patient with me :) I left a few nits and I think the TODO in the docs may need to be addressed, LGTM besides that

javanna · 2018-06-11T09:18:25Z

client/rest/src/main/java/org/elasticsearch/client/RestClient.java

+            });
            if (false == selectedDeadNodes.isEmpty()) {
-                return singletonList(selectedDeadNodes.get(0));
+                return singletonList(Collections.min(selectedDeadNodes).node);


maybe silly question: how does min help compared to sorting? Doesn't min iterate over all the elements?

javanna · 2018-06-11T09:20:05Z

client/rest/src/main/java/org/elasticsearch/client/DeadHostState.java

+        if (timeSupplier != other.timeSupplier) {
+            throw new IllegalArgumentException("can't compare DeadHostStates with different clocks ["
+                    + timeSupplier + " != " + other.timeSupplier + "]");
+        }


javanna · 2018-06-11T09:24:58Z

client/rest/src/main/java/org/elasticsearch/client/DeadHostState.java

        long timeoutNanos = (long)Math.min(MIN_CONNECTION_TIMEOUT_NANOS * 2 * Math.pow(2, previousDeadHostState.failedAttempts * 0.5 - 1),
                MAX_CONNECTION_TIMEOUT_NANOS);
-        this.deadUntilNanos = timeSupplier.nanoTime() + timeoutNanos;
+        this.deadUntilNanos = previousDeadHostState.timeSupplier.nanoTime() + timeoutNanos;


javanna · 2018-06-11T09:25:59Z

client/rest/src/main/java/org/elasticsearch/client/NodeSelector.java

+    /**
+     * Selector that matches any node that has metadata and doesn't
+     * have the {@code master} role OR it has the data {@code data}
+     * role. It does not reorder the nodes sent to it.


the reordering bit can be removed given that it can't reorder anyways?

javanna · 2018-06-11T09:27:58Z

client/rest/src/main/java/org/elasticsearch/client/RestClient.java

+        Map<HttpHost, Node> nodesByHost = new LinkedHashMap<>();
+        for (Node node : nodes) {
+            Objects.requireNonNull(node, "node cannot be null");
+            // TODO should we throw an IAE if this happens?


can you clarify the comment so it is clear that it's about nodes with same host?

javanna · 2018-06-11T09:34:56Z

client/rest/src/main/java/org/elasticsearch/client/RestClient.java

+     * Contains a reference to a blacklisted node and the time until it is
+     * revived. We use this so we can do a single pass over the blacklist.
+     */
+    private static class DeadNodeAndDeadness implements Comparable<DeadNodeAndDeadness> {


nit: this is kind of a funny name to me, I would be ok with DeadNode, not a biggie though, it's a private class

Will rename.

javanna · 2018-06-11T09:36:56Z

client/rest/src/test/java/org/elasticsearch/client/RestClientMultipleHostsIntegTests.java

            httpServers[i] = httpServer;
            httpHosts[i] = new HttpHost(httpServer.getAddress().getHostString(), httpServer.getAddress().getPort());
        }
+                restClient = buildRestClient();


indentation looks off?

Weird. Not sure why I did that. Fixed.

javanna · 2018-06-11T09:37:36Z

client/rest/src/test/java/org/elasticsearch/client/documentation/RestClientDocumentation.java

            }
+            {
+                //tag::rest-client-node-selector
+                // TODO link me to docs


this is left to be done?

javanna · 2018-06-11T09:39:43Z

client/sniffer/src/test/java/org/elasticsearch/client/sniff/SnifferTests.java

+                assertEquals(expectedNode.getBoundHosts(), actualNode.getBoundHosts());
+                assertEquals(expectedNode.getName(), actualNode.getName());
+                assertEquals(expectedNode.getVersion(), actualNode.getVersion());
+                assertEquals(expectedNode.getRoles(), actualNode.getRoles());


if you prefer moving this back to Node I am fine with it, no biggie.

javanna · 2018-06-11T09:54:29Z

test/framework/src/main/java/org/elasticsearch/test/rest/yaml/ESClientYamlSuiteTestCase.java

+            ElasticsearchNodesSniffer.Scheme.valueOf(getProtocol().toUpperCase(Locale.ROOT));
+        ElasticsearchNodesSniffer sniffer = new ElasticsearchNodesSniffer(
+                adminClient(), ElasticsearchNodesSniffer.DEFAULT_SNIFF_REQUEST_TIMEOUT, scheme);
+        return sniffer.sniff();


I am quite surprised that all of the yaml tests are ok with this, given that we use potentially more nodes than the configured ones. Good though, and tests that need something different will have to configure a selector. I like it. Maybe this should be documented somewhere.

Hmmm. Let me find a spot.

This reverts commit d551dcb.

javanna · 2018-06-11T17:04:12Z

docs/java-rest/low-level/usage.asciidoc

+<3> Customize the response consumer.
+
+`addHeader` is for headers that are required for authorization or to work with
+a proxy that in front of Elasticsearch. There is no need to set the


some missing word here? that is?

javanna · 2018-06-11T17:04:43Z

docs/java-rest/low-level/usage.asciidoc

+`HttpEntity` attached to the request.
+
+You can set the `NodeSelector` which controls which nodes will receive
+requests. `NodeSelector.NOT_MASTER_ONLY` is a good choice.


this may need to become the default at some point ?

nik9000 · 2018-06-12T14:33:29Z

I'm working through issues backporting this with tribe.

nik9000 · 2018-06-15T17:24:57Z

Backport (finally) complete.

nik9000 added >enhancement v7.0.0 :Core/Java High Level REST Client v6.4.0 labels May 10, 2018

nik9000 requested a review from javanna May 10, 2018 21:27

Fix test

81dd654

nik9000 commented May 11, 2018

View reviewed changes

nik9000 added 2 commits May 11, 2018 12:19

Updates

73d710f

Move Compose

7eb276b

nik9000 added the review label May 11, 2018

javanna added the >breaking-java label May 14, 2018

javanna requested changes May 14, 2018

View reviewed changes

javanna reviewed May 14, 2018

View reviewed changes

Merge branch 'master' into pr/30523

9cda8c9

javanna added :Clients/Java Low Level REST Client Minimal dependencies Java Client for Elasticsearch and removed :Core/Java High Level REST Client labels May 29, 2018

nik9000 added 3 commits May 31, 2018 17:05

Merge branch 'master' into rest_node_selector

f05a7ca

Merge branch 'master' into rest_node_selector

d4c56a3

Fix javadoc

0e6c479

nik9000 removed the review label Jun 1, 2018

nik9000 added 3 commits June 1, 2018 14:42

Make NodeSelector work on mutable Iterable

e13171d

Some cleanup

7339b26

Stll a few comments left to get through.

Backwards

2ae7c27

nik9000 added 3 commits June 8, 2018 14:51

Merge branch 'master' into rest_node_selector

4652073

Cleanup

a6a5b46

Reuse method

7d6ee4f

javanna approved these changes Jun 11, 2018

View reviewed changes

nik9000 added 5 commits June 11, 2018 11:42

Docs!

25280ef

Fix import order

5eca290

Cleanups

0ab1b30

Revert "Drop equals from Node"

a9e682f

This reverts commit d551dcb.

Remove method we don't need any more

e29723a

javanna reviewed Jun 11, 2018

View reviewed changes

nik9000 added 2 commits June 11, 2018 14:28

Merge branch 'master' into rest_node_selector

07004c2

Words

f030c9b

nik9000 merged commit 0d9b788 into elastic:master Jun 11, 2018

nik9000 added the backport pending label Jun 12, 2018

nik9000 mentioned this pull request Jun 12, 2018

QA: Node name selector for yml tests #31276

Closed

nik9000 mentioned this pull request Jun 13, 2018

REST Client: NodeSelector for node attributes #31296

Merged

nik9000 removed the backport pending label Jun 15, 2018

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

LLClient: Support host selection #30523

LLClient: Support host selection #30523

Uh oh!

Conversation

nik9000 commented May 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented May 10, 2018

Uh oh!

nik9000 commented May 10, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik9000 commented May 11, 2018

Uh oh!

javanna left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javanna Jun 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nik9000 commented May 10, 2018 •

edited

Loading

javanna Jun 5, 2018 •

edited

Loading