Batching semantics?

[srush]

I'm curious about how careful I need to be with batching. If I batch together 50 calls of which which there are 5 unique prefixes (a,b,c,d,e), will it know to group those together. Or should I be careful about making that 5 different batches?

[Ying1123]

This is a great question. The answer is you do not need to be careful about the order of sending your requests. You can send all the requests, and our scheduler will do some kind of clustering for you. The current default scheduler is optimized for throughput, which will prioritize the one with the longest shared prefix. So if you are sending in the order of (a, b, c, a, b, c, a, b, c), the real execution order will be (a, a, a, b, b, b, c, c, c) if there is only one prefix can be cached. Also, notice that the cache size is dynamic. When the memory is spacious, we can cache all the a, b, c, d, e. Otherwise, we do cache-aware scheduling.

[Ying1123]

@srush please see this comment (#14 (comment)). The cache-aware scheduling only works if the requests have been submitted to the server. If you are using colab, please set a larger num_threads to submit them to the server.