In the load pipeline, separate the network request from the post-download CPU work, and throttle them differently

[nithinp7]

I am not sure how much of an impact it has, but our load thread pool is usually saturated by long network requests. Tiles that can be retrieved from the SQL cache however are extremely fast. This creates a conflict where cache retrievals are inevitably waiting on a load thread to finish resolving a network request. I think it ends up being bad scheduling to have very long-running tasks and very fast tasks in the same threads. If we can dedicate a thread pool to only process cache hits, cache loads may become even faster. The network load pool and cache load pool may need to be dynamically sized to not introduce any new bottlenecks.

[kring]

@nithinp7 load/worker threads shouldn't be blocking waiting for network requests. If you're seeing that, let me know, because it would be extremely concerning.

[kring]

Load threads aren't blocked, but there is that limit of 20 (by default) simultaneous loads, and there's no distinction there between loads from cache and loads from the network. So I guess we can have the situation you're referring to, where tiles that would be quick to load from cache have to wait their turn behind slower requests from the network.

I think the proper solution is to split the network load and the CPU work we do after we receive the response into separate tasks. Loading 20 tiles from the network simultaneously is fairly reasonable (on a decently fast connection, at least), but 20 worker threads doing CPU work at once is too much. In CesiumJS, for comparison, there is some browser dependent number of simultaneous network requests (probably 10+) but only a single thread doing CPU load work and only 5 tiles can be in the queue at once.