gvfs-helper: auto-retry after network errors, resource throttling, split GET and POST semantics

[jeffhostetler]

I'm marking this WIP because I haven't done a cleanup round nor squashed things yet.
But I want to get the CI builds a chance to run tonight.

This series attempts to:
[x] auto-retry after network outages
[x] throttle back when requested (or demanded) by the server.

Questions:
[done] What is the right default for the network retry limit?
[done] Should the throttle back have a time limit? (It's one thing to wait 3 or 4 minutes between
packfiles because we hit it too hard, but another if it says we should wait an hour or two.)
[no] Should the network retry look at the amount of data received and try to resume it?
[no] Should the network retry split large packfile requests if we can tell the user's network is flakey?

Basic testing with 5 concurrent fetches shows that it is pretty easy to get throttled and
makes me wonder if we should even bother with multi-threading this. Perhaps we just
limit it to waiting for index-pack in another thread, but only plan to have 1 network thread.
Or maybe that is just when talking to the main server -- we might be able to multithread
when talking to the cache-server.

[wilbaker]

What is the right default for the network retry limit?

We might decide we no longer want this approach, but the approach that's been used by VFS4G is:

• By default 6 maximum retries for a total of 7 attempts: https://github.com/microsoft/VFSForGit/blob/a537d002be8d48675f48e3e97fe1c8c6359133a9/GVFS/GVFS.Common/RetryConfig.cs#L10

• The maximum number of retries can be overridden by a setting in git config:
  https://github.com/microsoft/VFSForGit/blob/a537d002be8d48675f48e3e97fe1c8c6359133a9/GVFS/GVFS.Common/RetryConfig.cs#L102

• A default timeout of 30 seconds (can also be overridden with the config):
  https://github.com/microsoft/VFSForGit/blob/a537d002be8d48675f48e3e97fe1c8c6359133a9/GVFS/GVFS.Common/RetryConfig.cs#L11

• There is no delay/backoff between the first and second attempt:
  https://github.com/microsoft/VFSForGit/blob/a537d002be8d48675f48e3e97fe1c8c6359133a9/GVFS/GVFS.Common/RetryWrapper.cs#L103

• There is a maximum backoff of 5 minutes:
  https://github.com/microsoft/VFSForGit/blob/a537d002be8d48675f48e3e97fe1c8c6359133a9/GVFS/GVFS.Common/RetryWrapper.cs#L12

• The backoff is exponential and randomly adjusted by 10% (the idea was that if there is a brief outage all clients won't retry at the exact same time):
  https://github.com/microsoft/VFSForGit/blob/785ccb4a67281d0eb10f19cb53e6431b62b88555/GVFS/GVFS.Common/RetryBackoff.cs#L34

[jeffhostetler]

@wilbaker Thanks for the pointers.

[derrickstolee]

/azp run Microsoft.git

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[derrickstolee]

@jeffhostetler after the 503 update, I was unable to repro any network failures in the C# functional tests. (I got a different flaky failure in the C# code, but that's different ;) )

I'm happy to re-test and approve after you do the cleanup to make this not WIP.

[jeffhostetler]

Thanks for the confirmation! Almost finished....

[jeffhostetler]

/azp run Microsoft.git

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[derrickstolee]

So, we are assuming that any queued objects will get a flush request eventually? That sounds reasonable.

[jeffhostetler]

Yeah, I've split the queued and immediate usage now. The dry-run/pre-scan loops already handle the queue and drain (for missing blobs usually). The main difference now is that any missing trees (or commits) during those loops will be immediately fetched in isolation, but the queue will remain.

[jeffhostetler]

@derrickstolee I think I'm done tinkering with this one. 2.20191023.1 has already been thru the functional tests and is good. Just did a final squash on my fixups and am running 2.20191023.2 thru the its paces.

[derrickstolee]

I'll rebase these commits onto tentative/features/sparse-checkout-2.24.0 for #214 after you merge.

[jeffhostetler]

Thanks for all your help!