-
-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Open
Labels
enhancementNew feature or requestNew feature or requestup for grabsTickets that no-one is currently working onTickets that no-one is currently working on
Description
LocalAI version:
Environment, CPU architecture, OS, and Version:
Darwin N7L493PWK4 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64
Describe the bug
When you don't have a YAML config in models/
, you can get an obscure and large error response, not directly related to the actual underlying issue.
To Reproduce
Using llama-2-13b-ensemble-v5.Q4_K_M.gguf downloaded using huggingface_hub.hf_hub_download
:
> ls models
llama-2-13b-ensemble-v5.Q4_K_M.gguf
> curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "llama-2-13b-ensemble-v5.Q4_K_M",
"messages": [{"role": "user", "content": "What is an alpaca?"}],
"temperature": 0.1
}' | jq .
...
{
"error": {
"code": 500,
"message": "could not load model - all backends returned error: 25 errors occurred:\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unknown desc = failed loading model\n\t* could not load model: rpc error: code = Unavailable desc = error reading from server: EOF\n\t* could not load model: rpc error: code = Unknown desc = stat /models/llama-2-13b-ensemble-v5.Q4_K_M: no such file or directory\n\t* could not load model: rpc error: code = Unknown desc = stat /models/llama-2-13b-ensemble-v5.Q4_K_M: no such file or directory\n\t* could not load model: rpc error: code = Unknown desc = unsupported model type /models/llama-2-13b-ensemble-v5.Q4_K_M (should end with .onnx)\n\t* backend unsupported: /build/extra/grpc/huggingface/huggingface.py\n\t* backend unsupported: /build/extra/grpc/autogptq/autogptq.py\n\t* backend unsupported: /build/extra/grpc/bark/ttsbark.py\n\t* backend unsupported: /build/extra/grpc/diffusers/backend_diffusers.py\n\t* backend unsupported: /build/extra/grpc/exllama/exllama.py\n\t* backend unsupported: /build/extra/grpc/vall-e-x/ttsvalle.py\n\t* backend unsupported: /build/extra/grpc/vllm/backend_vllm.py\n\n",
"type": ""
}
}
Expected behavior
I expect a concise and simple error message stating the actual issue (which was not having a model YAML config made), not a ton of semi-related rpc error: code = Unknown desc
.
Logs
Here are logs in the docker compose
container (run without --detach
):
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:38781: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:39685: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:46393: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:37429: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:43713: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:38897: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:34381: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:35537: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:38983: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:36063: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:37487: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:36697: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:34451: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:35385: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:43293: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:42347: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:36319: connect: connection refused"
localai-api-1 | rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:42321: connect: connection refused"
Additional context
mudler
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestup for grabsTickets that no-one is currently working onTickets that no-one is currently working on