GPU Memory Cache not cleared.

[sonsai123]

I've been running it by installing on machine and running make commands. The model responses are lightning fast compared to transformers pipeline() method. But after few invocations using generate_stream, I observed that memory is full occupied and throwing CUDA OOM.

Model: OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5
Quantized: yes
GPU: A10G 24GB
OS: EC2 linux

Appreciate some solution or workaround.

Another issue I encountered is that, the input token length limit set to 1000 by default. So if I give some long text as input (some context and a question), getting error
inputs must have less than 1000 tokens. Given: 1013.
I've modified router rust files code to mitigate this validation as a workaround.

[OlivierDehaene]

But after few invocations using generate_stream, I observed that memory is full occupied and throwing CUDA OOM.

The GPU memory cache is cleared. You might oom because the tensor sizes are dynamic. The batch size, prompt length, decode length can all increase and therefore the memory consumption is not the same for every request.

Another issue I encountered is that, the input token length limit set to 1000 by default

You can use --max-input-length and --max-total-tokens to control the prompt and decode max length. You can also send the truncate parameter to automatically truncate to a fixed length.

[sonsai123]

Thanks @OlivierDehaene for very quick reply. Much appreciated.

You can use --max-input-length and --max-total-tokens

I'll revert the code and try these parameters

You might oom because the tensor sizes are dynamic. The batch size, prompt length, decode length can all increase and therefore the memory consumption is not the same for every request.

I've been running with pipeline() and face similar issue. So I manually added torch.cuda.empty_cache() and saw memory usage come down after every completion. I thought it is similar issue here.

However after I restart the server and tried same long input multiple times, and the server generated the response without going into OOM. I'll further look into it. This is happening only after few generate requests (like after 100 generate_stream requests).

[sonsai123]

I think I found a pattern @OlivierDehaene .

If I stop receiving two requests consecutively from generate_stream (the input is long and could generate long output as well), then the subsequent requests are not processing at all and throwing CUDA OOM.

[OlivierDehaene]

If I stop receiving two requests consecutively from generate_stream (the input is long and could generate long output as well), then the subsequent requests are not processing at all and throwing CUDA OOM.

Yes the requests you stop (by dropping the connection to the server) are not really stopped: they continue processing in the backend (see #202 that will fix this issue). Then when you send new requests, they get added to the batch of currently processing requests (that were dropped but not stopped). It is possible that you don't have enough VRAM to handle this new increased batch size.

You can control the maximum size of the batch size using the --max-batch-size parameter. Setting it to 1 disables the dynamic batching mechanism entirely.