The inference-manager manages inference runtimes (e.g., vLLM and Ollama) in containers, load models, and process requests.
Please see inference_request_flow.md.
-
Notifications
You must be signed in to change notification settings - Fork 3
Inference Manager
License
llmariner/inference-manager
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|