Upgrade OkHttp to 3.11.0. #794

jkozlowski · 2018-08-30T14:52:18Z

Potentially supersedes #747.

Why?

The 2 releases since 3.9.0 fixed bugs in Http2 handling (especially wrt. flow control).

See:

Do not alter connection flow-control window on SETTINGS frame. square/okhttp#3920
Defer StreamResetException until response body buffer is fully read. square/okhttp#3960
Ensure discarded HTTP/2 data frames free flow-control space. square/okhttp#4033
Confirm concurrent HTTP/2 requests with empty flow-control window. square/okhttp#4127

These bugs have been hit by an internal product and are blocking feature work.

Remaining issues:

- New: Honor the Retry-After response header. HTTP 503 (Unavailable) responses are retried automatically if this header is present and its delay is 0 seconds. HTTP 408 (Client Timeout) responses are retried automatically if the header is absent or its delay is 0 seconds. @uschi2000
3.10.0 removed support for hostname verification via CN @cakofony
There is one test in the repo that checks handling of status code 100, but 100 is kinda special and magical, see linked resources. @jkozlowski
okhttp.Request#tag can now be associated with multiple values (keyed by Class instances). We reset the default tag in CurrentUrlInterceptor and RoundRobinInterceptor. @uschi2000 I wasn't sure what that is supposed to do so I preserved the behavior, but I'm not sure if we need to adjust it or not.

ferozco

Seems like a straight forward bump, I don't think any of the issues mentioned would affect us

jkozlowski · 2018-08-30T16:14:51Z

@ferozco what should I do about the OkHttpClientTest#handlesSuccessfulResponseCodesWithSuccessHandler?

iamdanfox · 2018-08-30T16:16:33Z

Note that on the last PR, there were a couple of questions:

from @uschi2000

What does this mean for us?

- New: Honor the Retry-After response header. HTTP 503 (Unavailable) responses are retried automatically if this header is present and its delay is 0 seconds. HTTP 408 (Client Timeout) responses are retried automatically if the header is absent or its delay is 0 seconds.

from @cakofony

Note that 3.10.0 removed support for hostname verification via CN, and only validates SAN values.
This is ideal and matches modern versions of chrome, though if we're worried about breaking things we can supply our own HostnameVerifier instance to okhttp.

ferozco · 2018-08-30T16:17:04Z

I think its fine to omit since it is 100s are such a special case

jkozlowski · 2018-08-30T16:24:04Z

@iamdanfox Yep, I copied those in this PR description, see above. Would like to get the opinions of those people, ideally.

@j-baker had an opinion on the SAN thing (shouldn't affect us), Retry-After I would need to check if we send those on QoS exceptions, but I don't believe so.

uschi2000 · 2018-08-30T16:33:57Z

okhttp-clients/src/main/java/com/palantir/remoting3/okhttp/CurrentUrlInterceptor.java

                // Request.this.tag field by default points to request itself if it was not set in RequestBuilder.
                // We don't want to reference old request in new one - that is why we need to reset tag to null.
-                .tag(originalRequest.tag().equals(originalRequest) ? null : originalRequest.tag())
+                .tag(originalRequest.equals(originalRequest.tag()) ? null : originalRequest.tag())


was this a bug? if so, can you make it a separate PR, please?

no, they broke

tags are now a map keyed by Class, and the old one is the Object case

https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/Request.java#L137

vs https://github.com/square/okhttp/blob/parent-3.9.0/okhttp/src/main/java/okhttp3/Request.java#L42

@jkozlowski this whole block should no longer be necessary, looking at the new code - don't need to unset the tag

but maybe we want it to continue working the way it does for people who force their okhttp version, and so this'd be fine

cool ignore me :)

Soo, here's my understanding: in 3.9.0 the Request.Builder#tag method had the following javadoc:

/** * Attaches {@code tag} to the request. It can be used later to cancel the request. If the tag * is unspecified or null, the request is canceled by using the request itself as the tag. */

The value, as @j-baker pointed out, is automatically set in Request(Builder) constructor to Request.this, and then propagated across to the next builder when the request is cloned (e.g. for retries). This means it builds a linked list of requests which means they all linger on in memory.

It would seem to me this was an omission and not intentional behavior: this is now gone in OkHttp 3.11 so I think we are good to remove this.

Going to resolve this point, unless people object.

uschi2000 · 2018-08-30T16:41:56Z

re 503: i'd prefer we continue using our explicit 503 handling and turn off the okhttp one. similar to the existing

        client.followRedirects(false);  // We implement our own redirect logic.

j-baker · 2018-08-30T16:47:51Z

you don't get to pick

j-baker · 2018-08-30T16:48:01Z

https://github.com/square/okhttp/pull/3753/files

j-baker · 2018-08-30T16:48:43Z

Unless we want to make all of our bodies Unrepeatable, which is its own can of worms

jkozlowski · 2018-08-30T18:35:46Z

Right, I'll clean up the test tomorrow, look into the 503 (although looks like we can't opt out of that behaviour).

Would be good to understand why we thought we needed to unset the tags and what's the behaviour we want.

jkozlowski · 2018-08-31T08:54:43Z

jaxrs-clients/versions.lock

                "com.palantir.remoting3:okhttp-clients"
            ]
        },
        "com.squareup.okhttp3:logging-interceptor": {


Let's discuss SAN vs CN verification here: the only thing that I can think would break were internal ETE tests, but we verified with a standard test certificate and they are already setup with SAN pointing to localhost and internal test domains.

As far as my understanding goes, we should already be generating correct certs in production, so I think we can close this issue?

jkozlowski · 2018-08-31T09:35:04Z

versions.props

 com.palantir.safe-logging:* = 1.3.0
 com.palantir.tritium:tritium-registry = 0.8.0
-com.squareup.okhttp3:* = 3.9.0
+com.squareup.okhttp3:* = 3.11.0


Now for Retry-After handling, let's discuss here. The behavior change is as follows:

408 (CLIENT_TIMEOUT) is now automatically retried if Retry-After header is not set or is set to 0, as per above and here's the code: (https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/internal/http/RetryAndFollowUpInterceptor.java#L367). I don't think we particularly care about this, but in practice it would mean one extra retry: it will give up if THAT extra retry fails.

503 are retried if Retry-After is set to 0, here's the code: https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/internal/http/RetryAndFollowUpInterceptor.java#L374. Again, I don't believe we ever set Retry-After in 503s and again it means one extra retry (it will bail if that retry fails). Someone could still do this by throwing javax.ws.rs.ServiceUnavailableException.

So, I think we should be good here too, unless anyone objects?.

jkozlowski · 2018-08-31T14:31:54Z

Can you guys review the reasoning and merge if we think this is good?

jkozlowski · 2018-09-03T07:33:56Z

@iamdanfox, @uschi2000, @ferozco ping, can we get a merge/no-merge?

* Expose HostMetricsRegistry record methods (#780) This is so that we can separately implement a HostMetricsSink for it (see #779) such that we can share host metrics when constructing both conjure and remoting3 clients * publish BOM for jaxrs-client (#789) * Excavator: Render CircleCI file using template specified in .circleci/template.sh (#791) * Upgrade OkHttp to 3.11.0. (#794) * AssertionErrors are converted into service exceptions with type internal (#727) * No more ThreadLocals, delegate everything to 'palantir/tracing-java' (#799) * Use BINTRAY_*_REMOTING envs (#802) The project's default bintray creds are currently set up to publish to `conjure-java-runtime`. Use these custom env vars to maintain ability to publish http-remoting. * Better behaviour in the presence of 429s (#786)

jkozlowski requested a review from a team as a code owner August 30, 2018 14:52

Upgrade OkHttp to 3.11.0.

30860e5

jkozlowski force-pushed the upgrade-okhttp branch from a379760 to 30860e5 Compare August 30, 2018 15:00

iamdanfox mentioned this pull request Aug 30, 2018

Upgrade okhttp to 3.10.0 #747

Closed

ferozco approved these changes Aug 30, 2018

View reviewed changes

uschi2000 reviewed Aug 30, 2018

View reviewed changes

jkozlowski added 2 commits August 31, 2018 09:18

Remove test for HTTP 100.

45619fb

Remove tag resetting.

abe4a6c

jkozlowski commented Aug 31, 2018

View reviewed changes

ferozco merged commit d1cab4a into palantir:develop Sep 3, 2018

jkozlowski deleted the upgrade-okhttp branch September 3, 2018 10:16

iamdanfox mentioned this pull request Oct 15, 2018

[fix] Preserve OkHostnameVerifier from okhttp 3.9 #843

Closed

Upgrade OkHttp to 3.11.0. #794

Upgrade OkHttp to 3.11.0. #794

Uh oh!

Conversation

jkozlowski commented Aug 30, 2018 • edited by dansanduleac Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why?

Remaining issues:

Uh oh!

ferozco left a comment • edited by j-baker Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkozlowski commented Aug 30, 2018

Uh oh!

iamdanfox commented Aug 30, 2018

Uh oh!

ferozco commented Aug 30, 2018

Uh oh!

jkozlowski commented Aug 30, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uschi2000 commented Aug 30, 2018

Uh oh!

j-baker commented Aug 30, 2018

Uh oh!

j-baker commented Aug 30, 2018

Uh oh!

j-baker commented Aug 30, 2018

Uh oh!

jkozlowski commented Aug 30, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkozlowski Aug 31, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkozlowski commented Aug 31, 2018

Uh oh!

jkozlowski commented Sep 3, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jkozlowski commented Aug 30, 2018 •

edited by dansanduleac

Loading

ferozco left a comment •

edited by j-baker

Loading

jkozlowski Aug 31, 2018 •

edited

Loading